tobi/grammar-inference-engine
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
grammar-inference-engine/experiments/results/flask_gbnf.json

92 lines
2.3 KiB
JSON
Raw Normal View History

fix: validate SORE before returning, skip malformed grammars - Add validate_sore() to gbnf.py — checks parseability without converting - _infer_group now validates grammar and returns skip_reason='malformed_grammar' for SOREs containing raw code (e.g. w_body=, sult=, (+,+:N+...) - Results: 130 grammars, 130 GBNF OK, 0 GBNF FAIL - Flask: 5 OK, 0 FAIL, 3.4s - RAGSAK: 19 OK, 0 FAIL, 11 malformed, 12.9s - FastAPI: 106 OK, 0 FAIL, 6 malformed, 30.5s
2026-07-12 02:56:13 +02:00
[
{
"package": "examples/celery/src/task_app",
"ext": ".py",
"methods": 11,
feat: grammar_structure_score + min_structure filter Quantifies how structured a SORE is (0.0=flat bag, 1.0=fully structured). Flat bags are CRX over-approximations — they list symbols without ordering. With min_structure=0.2: Flask: 2 kept (was 5), 9 dropped RAGSAK: 10 kept (was 19), 114 dropped FastAPI: 47 kept (was 106), 95 dropped Total: 59 useful grammars, 218 noise removed CLI: --min-structure 0.2 (default: 0, keep all)
2026-07-12 03:02:50 +02:00
"skip": "too_diverse",
"structure_score": 0
fix: validate SORE before returning, skip malformed grammars - Add validate_sore() to gbnf.py — checks parseability without converting - _infer_group now validates grammar and returns skip_reason='malformed_grammar' for SOREs containing raw code (e.g. w_body=, sult=, (+,+:N+...) - Results: 130 grammars, 130 GBNF OK, 0 GBNF FAIL - Flask: 5 OK, 0 FAIL, 3.4s - RAGSAK: 19 OK, 0 FAIL, 11 malformed, 12.9s - FastAPI: 106 OK, 0 FAIL, 6 malformed, 30.5s
2026-07-12 02:56:13 +02:00
},
{
"package": "examples/javascript/tests",
"ext": ".py",
"methods": 5,
feat: grammar_structure_score + min_structure filter Quantifies how structured a SORE is (0.0=flat bag, 1.0=fully structured). Flat bags are CRX over-approximations — they list symbols without ordering. With min_structure=0.2: Flask: 2 kept (was 5), 9 dropped RAGSAK: 10 kept (was 19), 114 dropped FastAPI: 47 kept (was 106), 95 dropped Total: 59 useful grammars, 218 noise removed CLI: --min-structure 0.2 (default: 0, keep all)
2026-07-12 03:02:50 +02:00
"skip": "too_diverse",
"structure_score": 0
fix: validate SORE before returning, skip malformed grammars - Add validate_sore() to gbnf.py — checks parseability without converting - _infer_group now validates grammar and returns skip_reason='malformed_grammar' for SOREs containing raw code (e.g. w_body=, sult=, (+,+:N+...) - Results: 130 grammars, 130 GBNF OK, 0 GBNF FAIL - Flask: 5 OK, 0 FAIL, 3.4s - RAGSAK: 19 OK, 0 FAIL, 11 malformed, 12.9s - FastAPI: 106 OK, 0 FAIL, 6 malformed, 30.5s
2026-07-12 02:56:13 +02:00
},
{
"package": "examples/tutorial/flaskr",
"ext": ".py",
"methods": 18,
feat: grammar_structure_score + min_structure filter Quantifies how structured a SORE is (0.0=flat bag, 1.0=fully structured). Flat bags are CRX over-approximations — they list symbols without ordering. With min_structure=0.2: Flask: 2 kept (was 5), 9 dropped RAGSAK: 10 kept (was 19), 114 dropped FastAPI: 47 kept (was 106), 95 dropped Total: 59 useful grammars, 218 noise removed CLI: --min-structure 0.2 (default: 0, keep all)
2026-07-12 03:02:50 +02:00
"skip": "too_diverse",
"structure_score": 0
fix: validate SORE before returning, skip malformed grammars - Add validate_sore() to gbnf.py — checks parseability without converting - _infer_group now validates grammar and returns skip_reason='malformed_grammar' for SOREs containing raw code (e.g. w_body=, sult=, (+,+:N+...) - Results: 130 grammars, 130 GBNF OK, 0 GBNF FAIL - Flask: 5 OK, 0 FAIL, 3.4s - RAGSAK: 19 OK, 0 FAIL, 11 malformed, 12.9s - FastAPI: 106 OK, 0 FAIL, 6 malformed, 30.5s
2026-07-12 02:56:13 +02:00
},
{
"package": "examples/tutorial/tests",
"ext": ".py",
"methods": 25,
feat: grammar_structure_score + min_structure filter Quantifies how structured a SORE is (0.0=flat bag, 1.0=fully structured). Flat bags are CRX over-approximations — they list symbols without ordering. With min_structure=0.2: Flask: 2 kept (was 5), 9 dropped RAGSAK: 10 kept (was 19), 114 dropped FastAPI: 47 kept (was 106), 95 dropped Total: 59 useful grammars, 218 noise removed CLI: --min-structure 0.2 (default: 0, keep all)
2026-07-12 03:02:50 +02:00
"skip": "too_diverse",
"structure_score": 0
fix: validate SORE before returning, skip malformed grammars - Add validate_sore() to gbnf.py — checks parseability without converting - _infer_group now validates grammar and returns skip_reason='malformed_grammar' for SOREs containing raw code (e.g. w_body=, sult=, (+,+:N+...) - Results: 130 grammars, 130 GBNF OK, 0 GBNF FAIL - Flask: 5 OK, 0 FAIL, 3.4s - RAGSAK: 19 OK, 0 FAIL, 11 malformed, 12.9s - FastAPI: 106 OK, 0 FAIL, 6 malformed, 30.5s
2026-07-12 02:56:13 +02:00
},
{
"package": "src/flask",
"ext": ".py",
"methods": 216,
feat: grammar_structure_score + min_structure filter Quantifies how structured a SORE is (0.0=flat bag, 1.0=fully structured). Flat bags are CRX over-approximations — they list symbols without ordering. With min_structure=0.2: Flask: 2 kept (was 5), 9 dropped RAGSAK: 10 kept (was 19), 114 dropped FastAPI: 47 kept (was 106), 95 dropped Total: 59 useful grammars, 218 noise removed CLI: --min-structure 0.2 (default: 0, keep all)
2026-07-12 03:02:50 +02:00
"skip": "too_diverse",
"structure_score": 0
fix: validate SORE before returning, skip malformed grammars - Add validate_sore() to gbnf.py — checks parseability without converting - _infer_group now validates grammar and returns skip_reason='malformed_grammar' for SOREs containing raw code (e.g. w_body=, sult=, (+,+:N+...) - Results: 130 grammars, 130 GBNF OK, 0 GBNF FAIL - Flask: 5 OK, 0 FAIL, 3.4s - RAGSAK: 19 OK, 0 FAIL, 11 malformed, 12.9s - FastAPI: 106 OK, 0 FAIL, 6 malformed, 30.5s
2026-07-12 02:56:13 +02:00
},
{
"package": "src/flask/json",
"ext": ".py",
"methods": 50,
feat: grammar_structure_score + min_structure filter Quantifies how structured a SORE is (0.0=flat bag, 1.0=fully structured). Flat bags are CRX over-approximations — they list symbols without ordering. With min_structure=0.2: Flask: 2 kept (was 5), 9 dropped RAGSAK: 10 kept (was 19), 114 dropped FastAPI: 47 kept (was 106), 95 dropped Total: 59 useful grammars, 218 noise removed CLI: --min-structure 0.2 (default: 0, keep all)
2026-07-12 03:02:50 +02:00
"skip": "low_structure",
"structure_score": 0.042105263157894736
fix: validate SORE before returning, skip malformed grammars - Add validate_sore() to gbnf.py — checks parseability without converting - _infer_group now validates grammar and returns skip_reason='malformed_grammar' for SOREs containing raw code (e.g. w_body=, sult=, (+,+:N+...) - Results: 130 grammars, 130 GBNF OK, 0 GBNF FAIL - Flask: 5 OK, 0 FAIL, 3.4s - RAGSAK: 19 OK, 0 FAIL, 11 malformed, 12.9s - FastAPI: 106 OK, 0 FAIL, 6 malformed, 30.5s
2026-07-12 02:56:13 +02:00
},
{
"package": "src/flask/sansio",
"ext": ".py",
"methods": 102,
feat: grammar_structure_score + min_structure filter Quantifies how structured a SORE is (0.0=flat bag, 1.0=fully structured). Flat bags are CRX over-approximations — they list symbols without ordering. With min_structure=0.2: Flask: 2 kept (was 5), 9 dropped RAGSAK: 10 kept (was 19), 114 dropped FastAPI: 47 kept (was 106), 95 dropped Total: 59 useful grammars, 218 noise removed CLI: --min-structure 0.2 (default: 0, keep all)
2026-07-12 03:02:50 +02:00
"skip": "low_structure",
"structure_score": 0.009523809523809525
fix: validate SORE before returning, skip malformed grammars - Add validate_sore() to gbnf.py — checks parseability without converting - _infer_group now validates grammar and returns skip_reason='malformed_grammar' for SOREs containing raw code (e.g. w_body=, sult=, (+,+:N+...) - Results: 130 grammars, 130 GBNF OK, 0 GBNF FAIL - Flask: 5 OK, 0 FAIL, 3.4s - RAGSAK: 19 OK, 0 FAIL, 11 malformed, 12.9s - FastAPI: 106 OK, 0 FAIL, 6 malformed, 30.5s
2026-07-12 02:56:13 +02:00
},
{
"package": "tests",
"ext": ".py",
"methods": 962,
feat: grammar_structure_score + min_structure filter Quantifies how structured a SORE is (0.0=flat bag, 1.0=fully structured). Flat bags are CRX over-approximations — they list symbols without ordering. With min_structure=0.2: Flask: 2 kept (was 5), 9 dropped RAGSAK: 10 kept (was 19), 114 dropped FastAPI: 47 kept (was 106), 95 dropped Total: 59 useful grammars, 218 noise removed CLI: --min-structure 0.2 (default: 0, keep all)
2026-07-12 03:02:50 +02:00
"skip": "low_structure",
"structure_score": 0.021897810218978103
fix: validate SORE before returning, skip malformed grammars - Add validate_sore() to gbnf.py — checks parseability without converting - _infer_group now validates grammar and returns skip_reason='malformed_grammar' for SOREs containing raw code (e.g. w_body=, sult=, (+,+:N+...) - Results: 130 grammars, 130 GBNF OK, 0 GBNF FAIL - Flask: 5 OK, 0 FAIL, 3.4s - RAGSAK: 19 OK, 0 FAIL, 11 malformed, 12.9s - FastAPI: 106 OK, 0 FAIL, 6 malformed, 30.5s
2026-07-12 02:56:13 +02:00
},
{
"package": "tests/test_apps",
"ext": ".py",
"methods": 3,
feat: grammar_structure_score + min_structure filter Quantifies how structured a SORE is (0.0=flat bag, 1.0=fully structured). Flat bags are CRX over-approximations — they list symbols without ordering. With min_structure=0.2: Flask: 2 kept (was 5), 9 dropped RAGSAK: 10 kept (was 19), 114 dropped FastAPI: 47 kept (was 106), 95 dropped Total: 59 useful grammars, 218 noise removed CLI: --min-structure 0.2 (default: 0, keep all)
2026-07-12 03:02:50 +02:00
"skip": "too_diverse",
"structure_score": 0
fix: validate SORE before returning, skip malformed grammars - Add validate_sore() to gbnf.py — checks parseability without converting - _infer_group now validates grammar and returns skip_reason='malformed_grammar' for SOREs containing raw code (e.g. w_body=, sult=, (+,+:N+...) - Results: 130 grammars, 130 GBNF OK, 0 GBNF FAIL - Flask: 5 OK, 0 FAIL, 3.4s - RAGSAK: 19 OK, 0 FAIL, 11 malformed, 12.9s - FastAPI: 106 OK, 0 FAIL, 6 malformed, 30.5s
2026-07-12 02:56:13 +02:00
},
{
"package": "tests/test_apps/blueprintapp/apps",
"ext": ".py",
"methods": 4,
feat: grammar_structure_score + min_structure filter Quantifies how structured a SORE is (0.0=flat bag, 1.0=fully structured). Flat bags are CRX over-approximations — they list symbols without ordering. With min_structure=0.2: Flask: 2 kept (was 5), 9 dropped RAGSAK: 10 kept (was 19), 114 dropped FastAPI: 47 kept (was 106), 95 dropped Total: 59 useful grammars, 218 noise removed CLI: --min-structure 0.2 (default: 0, keep all)
2026-07-12 03:02:50 +02:00
"skip": "",
"structure_score": 0.2857142857142857,
fix: validate SORE before returning, skip malformed grammars - Add validate_sore() to gbnf.py — checks parseability without converting - _infer_group now validates grammar and returns skip_reason='malformed_grammar' for SOREs containing raw code (e.g. w_body=, sult=, (+,+:N+...) - Results: 130 grammars, 130 GBNF OK, 0 GBNF FAIL - Flask: 5 OK, 0 FAIL, 3.4s - RAGSAK: 19 OK, 0 FAIL, 11 malformed, 12.9s - FastAPI: 106 OK, 0 FAIL, 6 malformed, 30.5s
2026-07-12 02:56:13 +02:00
"sore": "return.render_template+",
"gbnf": "root ::= \"return\" \"render_template\"+",
"gbnf_ok": true
},
{
"package": "tests/type_check",
"ext": ".py",
"methods": 24,
feat: grammar_structure_score + min_structure filter Quantifies how structured a SORE is (0.0=flat bag, 1.0=fully structured). Flat bags are CRX over-approximations — they list symbols without ordering. With min_structure=0.2: Flask: 2 kept (was 5), 9 dropped RAGSAK: 10 kept (was 19), 114 dropped FastAPI: 47 kept (was 106), 95 dropped Total: 59 useful grammars, 218 noise removed CLI: --min-structure 0.2 (default: 0, keep all)
2026-07-12 03:02:50 +02:00
"skip": "",
"structure_score": 0.5597014925373135,
"sore": "def?.(Generator+encode+for+in+iter+range+return+show+str+t+x+yield)+?.stream_template+?.(Response+code+jsonify)+?.render_template+?.HTTPStatus?.name+?.self?.OK?.template_name+?",
"gbnf": "root ::= \"def\"? (\"Generator\" | \"encode\" | \"for\" | \"in\" | \"iter\" | \"range\" | \"return\" | \"show\" | \"str\" | \"t\" | \"x\" | \"yield\")* \"stream_template\"* (\"Response\" | \"code\" | \"jsonify\")* \"render_template\"* \"HTTPStatus\"? \"name\"* \"self\"? \"OK\"? \"template_name\"*",
fix: validate SORE before returning, skip malformed grammars - Add validate_sore() to gbnf.py — checks parseability without converting - _infer_group now validates grammar and returns skip_reason='malformed_grammar' for SOREs containing raw code (e.g. w_body=, sult=, (+,+:N+...) - Results: 130 grammars, 130 GBNF OK, 0 GBNF FAIL - Flask: 5 OK, 0 FAIL, 3.4s - RAGSAK: 19 OK, 0 FAIL, 11 malformed, 12.9s - FastAPI: 106 OK, 0 FAIL, 6 malformed, 30.5s
2026-07-12 02:56:13 +02:00
"gbnf_ok": true
},
{
"package": "(other)",
"ext": ".py",
feat: grammar_structure_score + min_structure filter Quantifies how structured a SORE is (0.0=flat bag, 1.0=fully structured). Flat bags are CRX over-approximations — they list symbols without ordering. With min_structure=0.2: Flask: 2 kept (was 5), 9 dropped RAGSAK: 10 kept (was 19), 114 dropped FastAPI: 47 kept (was 106), 95 dropped Total: 59 useful grammars, 218 noise removed CLI: --min-structure 0.2 (default: 0, keep all)
2026-07-12 03:02:50 +02:00
"methods": 4,
"skip": "",
"structure_score": 0
fix: validate SORE before returning, skip malformed grammars - Add validate_sore() to gbnf.py — checks parseability without converting - _infer_group now validates grammar and returns skip_reason='malformed_grammar' for SOREs containing raw code (e.g. w_body=, sult=, (+,+:N+...) - Results: 130 grammars, 130 GBNF OK, 0 GBNF FAIL - Flask: 5 OK, 0 FAIL, 3.4s - RAGSAK: 19 OK, 0 FAIL, 11 malformed, 12.9s - FastAPI: 106 OK, 0 FAIL, 6 malformed, 30.5s
2026-07-12 02:56:13 +02:00
}
]
Reference in a new issue Copy permalink