tobi/grammar-inference-engine
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
grammar-inference-engine/experiments/results
History
tobjend 3468813ec8 fix: validate SORE before returning, skip malformed grammars 
- Add validate_sore() to gbnf.py — checks parseability without converting
- _infer_group now validates grammar and returns skip_reason='malformed_grammar'
  for SOREs containing raw code (e.g. w_body=, sult=, (+,+:N+...)
- Results: 130 grammars, 130 GBNF OK, 0 GBNF FAIL
  - Flask: 5 OK, 0 FAIL, 3.4s
  - RAGSAK: 19 OK, 0 FAIL, 11 malformed, 12.9s
  - FastAPI: 106 OK, 0 FAIL, 6 malformed, 30.5s
2026-07-12 02:56:13 +02:00
..
baseline_package.json experiments: test all context strategies on RAGSAK — behavioral grouping wins 2026-07-11 23:40:21 +02:00
coarsen_coroutines.json feat: 4-codebase evaluation — conventions vs completions tradeoff 2026-07-12 01:14:09 +02:00
coarsen_fastapi.json feat: 4-codebase evaluation — conventions vs completions tradeoff 2026-07-12 01:14:09 +02:00
coarsen_flask.json feat: 4-codebase evaluation — conventions vs completions tradeoff 2026-07-12 01:14:09 +02:00
coarsen_ragsak.json feat: 4-codebase evaluation — conventions vs completions tradeoff 2026-07-12 01:14:09 +02:00
fastapi_gbnf.json fix: validate SORE before returning, skip malformed grammars 2026-07-12 02:56:13 +02:00
file_path_k1.json experiments: test all context strategies on RAGSAK — behavioral grouping wins 2026-07-11 23:40:21 +02:00
file_path_k2.json experiments: test all context strategies on RAGSAK — behavioral grouping wins 2026-07-11 23:40:21 +02:00
file_path_k3.json experiments: test all context strategies on RAGSAK — behavioral grouping wins 2026-07-11 23:40:21 +02:00
first_k_sym_1.json experiments: test all context strategies on RAGSAK — behavioral grouping wins 2026-07-11 23:40:21 +02:00
first_k_sym_2.json experiments: test all context strategies on RAGSAK — behavioral grouping wins 2026-07-11 23:40:21 +02:00
first_k_sym_3.json experiments: test all context strategies on RAGSAK — behavioral grouping wins 2026-07-11 23:40:21 +02:00
flask_baseline_package.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
flask_file_path_k1.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
flask_file_path_k2.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
flask_file_path_k3.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
flask_first_k_sym_1.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
flask_first_k_sym_2.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
flask_first_k_sym_3.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
flask_gbnf.json fix: validate SORE before returning, skip malformed grammars 2026-07-12 02:56:13 +02:00
flask_reduce_k1_e01.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
flask_reduce_k1_e02.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
flask_reduce_k1_e03.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
flask_reduce_k1_e04.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
flask_reduce_k1_e005.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
flask_reduce_k1_e015.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
flask_reduce_k2_e01.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
flask_reduce_k2_e02.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
flask_reduce_k2_e03.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
flask_reduce_k2_e04.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
flask_reduce_k2_e005.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
flask_reduce_k2_e015.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
flask_reduce_k3_e01.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
flask_reduce_k3_e02.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
flask_reduce_k3_e03.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
flask_reduce_k3_e04.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
flask_reduce_k3_e005.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
flask_reduce_k3_e015.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
flask_return_type_heuristic.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
flask_summary.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
flask_two_d_p1_s1.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
flask_two_d_p1_s2.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
flask_two_d_p2_s1.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
flask_two_d_p2_s2.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
freq_coroutines.json feat: frequency threshold sweep — 0.01-0.20 across 4 codebases 2026-07-12 01:26:45 +02:00
freq_fastapi.json feat: frequency threshold sweep — 0.01-0.20 across 4 codebases 2026-07-12 01:26:45 +02:00
freq_flask.json feat: frequency threshold sweep — 0.01-0.20 across 4 codebases 2026-07-12 01:26:45 +02:00
freq_ragsak.json feat: frequency threshold sweep — 0.01-0.20 across 4 codebases 2026-07-12 01:26:45 +02:00
ragsak_baseline_package.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
ragsak_file_path_k1.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
ragsak_file_path_k2.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
ragsak_file_path_k3.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
ragsak_first_k_sym_1.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
ragsak_first_k_sym_2.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
ragsak_first_k_sym_3.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
ragsak_gbnf.json fix: validate SORE before returning, skip malformed grammars 2026-07-12 02:56:13 +02:00
ragsak_reduce_k1_e01.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
ragsak_reduce_k1_e02.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
ragsak_reduce_k1_e03.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
ragsak_reduce_k1_e04.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
ragsak_reduce_k1_e005.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
ragsak_reduce_k1_e015.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
ragsak_reduce_k2_e01.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
ragsak_reduce_k2_e02.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
ragsak_reduce_k2_e03.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
ragsak_reduce_k2_e04.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
ragsak_reduce_k2_e005.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
ragsak_reduce_k2_e015.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
ragsak_reduce_k3_e01.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
ragsak_reduce_k3_e02.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
ragsak_reduce_k3_e03.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
ragsak_reduce_k3_e04.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
ragsak_reduce_k3_e005.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
ragsak_reduce_k3_e015.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
ragsak_return_type_heuristic.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
ragsak_summary.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
ragsak_two_d_p1_s1.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
ragsak_two_d_p1_s2.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
ragsak_two_d_p2_s1.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
ragsak_two_d_p2_s2.json feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
return_type_heuristic.json experiments: test all context strategies on RAGSAK — behavioral grouping wins 2026-07-11 23:40:21 +02:00
summary.json experiments: test all context strategies on RAGSAK — behavioral grouping wins 2026-07-11 23:40:21 +02:00
two_d_p1_s1.json experiments: test all context strategies on RAGSAK — behavioral grouping wins 2026-07-11 23:40:21 +02:00
two_d_p1_s2.json experiments: test all context strategies on RAGSAK — behavioral grouping wins 2026-07-11 23:40:21 +02:00
two_d_p2_s1.json experiments: test all context strategies on RAGSAK — behavioral grouping wins 2026-07-11 23:40:21 +02:00
two_d_p2_s2.json experiments: test all context strategies on RAGSAK — behavioral grouping wins 2026-07-11 23:40:21 +02:00