grammar-inference-engine/experiments/RESULTS.md

Experiment Results — Context Strategies + Reduce

Date: 2026-07-12

Codebases

Codebase	Files	Methods	Language
RAGSAK	462	1594	Kotlin
Flask	24	1391	Python

RAGSAK Results

Strategy	Contexts	SOREs	Coverage	Notes
Baseline (package)	116	2	0.6%	Structural — useless
File path k=1	43	2	1.1%	Structural — no better
File path k=2	54	2	1.1%	Same
File path k=3	54	2	1.1%	Same
First 1 symbol	134	20	4.6%	Behavioral — 10× better
First 2 symbols	141	39	9.8%	Behavioral — 20× better
First 3 symbols	117	47	12.0%	Winner
Two-dim (p1,s1)	151	24	6.3%	Hybrid — worse
Two-dim (p1,s2)	119	33	8.7%	Hybrid — worse
Two-dim (p2,s1)	146	24	6.3%	Hybrid — worse
Two-dim (p2,s2)	117	33	8.7%	Hybrid — worse
Return type heuristic	5	0	0.0%	Useless

Reduce Results (RAGSAK)

Base	Threshold	Merges	Coverage	Notes
k=1	0.05-0.4	0	4.6%	No merges — contexts too distinct
k=2	0.05-0.2	0	9.8%	No merges
k=2	0.3	1	9.6%	Merged (JobStatus, every) with (JobStatus, now)
k=2	0.4	2	9.5%	Merged 2 pairs
k=3	0.05-0.2	0	12.0%	No merges
k=3	0.3	1	11.7%	Merged (JobStatus, every, getJobStatus) with (JobStatus, now, minusMinutes)
k=3	0.4	1	11.7%	Same merge

Flask Results

Strategy	Contexts	SOREs	Coverage	Notes
Baseline (package)	9	0	0.0%	Structural — useless
File path k=1	7	0	0.0%	Same
File path k=2	9	0	0.0%	Same
File path k=3	9	0	0.0%	Same
First 1 symbol	53	2	1.4%	Behavioral — only 2 SOREs
First 2 symbols	96	18	7.3%	Behavioral — 15× better
First 3 symbols	85	21	10.7%	Winner
Two-dim (p1,s1)	68	5	2.2%	Hybrid — worse
Two-dim (p1,s2)	96	22	10.1%	Close to behavioral
Two-dim (p2,s1)	69	5	2.2%	Worse
Two-dim (p2,s2)	93	19	9.5%	Close to behavioral
Return type heuristic	3	0	0.0%	Useless

Reduce Results (Flask)

Base	Threshold	Merges	Coverage	Notes
k=1	0.05-0.4	0	1.4%	No merges
k=2	0.05-0.2	0	7.3%	No merges
k=2	0.3-0.4	1	7.3%	Merged (def, boolean) with (def, is_boolean)
k=3	0.05-0.2	0	10.7%	No merges
k=3	0.3-0.4	1	10.7%	Merged (def, boolean, return) with (def, is_boolean, return)

Cross-Codebase Comparison

Metric	RAGSAK	Flask
Best strategy	First 3 symbols	First 3 symbols
Best coverage	12.0%	10.7%
SOREs (best)	47	21
Reduce merges (ε=0.3)	1	1
Reduce impact on coverage	-0.3%	0%

Key Findings

1. Behavioral grouping wins consistently

• Both codebases: First-k-symbols beats all other strategies
• Structural (file path) is useless — doesn't predict behavioral similarity
• Hybrid (2D) is worse than pure behavioral

2. Coverage ceiling is ~10-12%

• RAGSAK: 12.0% with 47 SOREs
• Flask: 10.7% with 21 SOREs
• Most methods have unique call patterns — they don't share prefixes

3. Reduce has minimal impact

• Very few merges at any threshold (0-2 per codebase)
• Support-weighted distance is conservative — requires very similar SOAs
• Merges that do happen don't improve coverage
• Reason: contexts created by first-k-symbols are already quite distinct

4. The few Reduce merges are meaningful

• RAGSAK: (JobStatus, every, getJobStatus) ↔ (JobStatus, now, minusMinutes) — same polling pattern, different initial call
• Flask: (def, boolean) ↔ (def, is_boolean) — same type-checking pattern, different method name

5. Flask is harder than RAGSAK

• Flask has fewer files (24 vs 462) but similar methods (1391 vs 1594)
• Flask has more unique methods per package — less repetition
• Flask SOREs are shorter/simpler — less compressible patterns

Next Steps

1. Accept the ceiling: ~10-12% coverage may be the maximum for prefix-based grouping
2. Symbol coarsening: Collapse specific symbols into categories (GETTER, SETTER, etc.)
3. Hierarchical grouping: Group by k=1 first, then sub-group by k=2 within each group
4. GBNF output: Convert SOREs to GBNF format for llama.cpp constrained decoding
5. Test on kotlinx.coroutines: Third codebase for cross-validation

Files Generated

• experiments/results/ragsak_summary.json — RAGSAK metrics
• experiments/results/flask_summary.json — Flask metrics
• experiments/context_eval.py — experiment runner (supports multiple codebases)
• bex/reduce.py — Algorithm 4 (TODS 2010) implementation
• tests/test_reduce.py — 24 tests for Reduce