grammar-inference-engine/experiments/RESULTS.md

Experiment Results — Context Strategies + Reduce

Date: 2026-07-12

Codebases

Codebase	Files	Methods	Language
RAGSAK	462	1594	Kotlin
Flask	24	1391	Python

RAGSAK Results

Strategy	Contexts	SOREs	Coverage	Notes
Baseline (package)	116	2	0.6%	Structural — useless
File path k=1	43	2	1.1%	Structural — no better
File path k=2	54	2	1.1%	Same
File path k=3	54	2	1.1%	Same
First 1 symbol	134	20	4.6%	Behavioral — 10× better
First 2 symbols	141	39	9.8%	Behavioral — 20× better
First 3 symbols	117	47	12.0%	Winner
Two-dim (p1,s1)	151	24	6.3%	Hybrid — worse
Two-dim (p1,s2)	119	33	8.7%	Hybrid — worse
Two-dim (p2,s1)	146	24	6.3%	Hybrid — worse
Two-dim (p2,s2)	117	33	8.7%	Hybrid — worse
Return type heuristic	5	0	0.0%	Useless

Reduce Results (RAGSAK)

Base	Threshold	Merges	Coverage	Notes
k=1	0.05-0.4	0	4.6%	No merges — contexts too distinct
k=2	0.05-0.2	0	9.8%	No merges
k=2	0.3	1	9.6%	Merged (JobStatus, every) with (JobStatus, now)
k=2	0.4	2	9.5%	Merged 2 pairs
k=3	0.05-0.2	0	12.0%	No merges
k=3	0.3	1	11.7%	Merged (JobStatus, every, getJobStatus) with (JobStatus, now, minusMinutes)
k=3	0.4	1	11.7%	Same merge

Flask Results

Strategy	Contexts	SOREs	Coverage	Notes
Baseline (package)	9	0	0.0%	Structural — useless
File path k=1	7	0	0.0%	Same
File path k=2	9	0	0.0%	Same
File path k=3	9	0	0.0%	Same
First 1 symbol	53	2	1.4%	Behavioral — only 2 SOREs
First 2 symbols	96	18	7.3%	Behavioral — 15× better
First 3 symbols	85	21	10.7%	Winner
Two-dim (p1,s1)	68	5	2.2%	Hybrid — worse
Two-dim (p1,s2)	96	22	10.1%	Close to behavioral
Two-dim (p2,s1)	69	5	2.2%	Worse
Two-dim (p2,s2)	93	19	9.5%	Close to behavioral
Return type heuristic	3	0	0.0%	Useless

Reduce Results (Flask)

Base	Threshold	Merges	Coverage	Notes
k=1	0.05-0.4	0	1.4%	No merges
k=2	0.05-0.2	0	7.3%	No merges
k=2	0.3-0.4	1	7.3%	Merged (def, boolean) with (def, is_boolean)
k=3	0.05-0.2	0	10.7%	No merges
k=3	0.3-0.4	1	10.7%	Merged (def, boolean, return) with (def, is_boolean, return)

Cross-Codebase Comparison

Metric	RAGSAK	Flask
Best strategy	First 3 symbols	First 3 symbols
Best coverage	12.0%	10.7%
SOREs (best)	47	21
Reduce merges (ε=0.3)	1	1
Reduce impact on coverage	-0.3%	0%

Key Findings

1. Behavioral grouping wins consistently

• Both codebases: First-k-symbols beats all other strategies
• Structural (file path) is useless — doesn't predict behavioral similarity
• Hybrid (2D) is worse than pure behavioral

2. Coverage ceiling is ~10-12%

• RAGSAK: 12.0% with 47 SOREs
• Flask: 10.7% with 21 SOREs
• Most methods have unique call patterns — they don't share prefixes

3. Reduce has minimal impact

• Very few merges at any threshold (0-2 per codebase)
• Support-weighted distance is conservative — requires very similar SOAs
• Merges that do happen don't improve coverage
• Reason: contexts created by first-k-symbols are already quite distinct

4. The few Reduce merges are meaningful

• RAGSAK: (JobStatus, every, getJobStatus) ↔ (JobStatus, now, minusMinutes) — same polling pattern, different initial call
• Flask: (def, boolean) ↔ (def, is_boolean) — same type-checking pattern, different method name

5. Flask is harder than RAGSAK

• Flask has fewer files (24 vs 462) but similar methods (1391 vs 1594)
• Flask has more unique methods per package — less repetition
• Flask SOREs are shorter/simpler — less compressible patterns

Round 5: Cross-Package Exact Matches

Hypothesis: Some call sequences appear verbatim in multiple packages. These are the real cross-package conventions.

Method: Group all sequences by exact tuple match, count packages per sequence.

Result: 38 exact cross-package sequences in RAGSAK. Most trivial:

• ('clearAllMocks',) — 4 packages (test teardown)
• ('Builder',) — 4 packages (builder pattern)
• ('Any',) — 4 packages (Kotlin type)

Interesting ones:

• ('assumeTrue', 'isDockerAvailable', 'start', 'pullAndWarmup') — 4 pkgs (Docker setup)
• ('isNullOrBlank', 'error', 'error') — 3 pkgs (null check → error)
• ('sortedBy', 'map', 'toDescriptor') — 3 pkgs (data pipeline)
• ('ObjectMapper', 'findAndRegisterModules') — 2 pkgs (Jackson config)

Verdict: Exact matches too rare and mostly trivial. The real cross-package patterns are structural, not textual — "null check → error" appears with different method names in different packages.

Summary of Failed/Dismissed Approaches

Approach	Why it failed
Per-package inference	Too sparse (1-3 seqs/package)
Reduce algorithm	Wrong abstraction level — merges states within one automaton, not across packages
Cross-package grouping by first symbol	4.6% / 1.4% coverage — most groups skipped
Exact cross-package matches	38 sequences, mostly trivial single-call
MDL vs Language Size scoring	Scoring isn't the bottleneck — pattern extraction is

What Actually Works

• Behavioral grouping (first 3 symbols) — 12% / 10.7% coverage, consistent across codebases
• Calling context as prefix — the right signal, but package-specific
• GBNF conversion — correct implementation, but input patterns too specific

Next: Structural Coarsening + Cross-Package Detection

See EXPERIMENT_LOG.md for full reasoning and experiment design.

Core idea: collapse method names → categories using tree-sitter capture names. Converts textual sequences into structural shapes that repeat across packages.

('isNullOrBlank', 'error', 'error')     →  (CALL, ERROR, ERROR)
('raise', 'ValueError', 'ValueError')   →  (CALL, ERROR, ERROR)

Files Generated

• experiments/results/ragsak_summary.json — RAGSAK metrics
• experiments/results/flask_summary.json — Flask metrics
• experiments/context_eval.py — experiment runner (supports multiple codebases)
• experiments/EXPERIMENT_LOG.md — full experiment history and next steps
• bex/reduce.py — Algorithm 4 (TODS 2010) implementation
• bex/gbnf.py — SORE → GBNF converter
• tests/test_reduce.py — 24 tests for Reduce
• tests/test_gbnf.py — 15 tests for GBNF converter

---

Round 20: AST Pipeline + Scoring Fixes (2026-07-13)

Commit range: ea6cac5 → 8b3a454

Codebases

Codebase	Files	Methods	Language
RAGSAK	462	1609	Kotlin
FastAPI	—	—	Python
Zod	—	—	TypeScript

Scoring: Language Size over MDL (ADR-13)

Abandoned MDL scoring — it rewards short expressions, so generic info+ beat specific a.b.c.d.e+ (21% vs 98% success in Bex paper). Language Size (lang_size_score) chosen.

Metric	Bag grammar	Structured grammar
lang_size_score	9975	20
mdl_score	10^12 (clamped)	10^12 (clamped)

Final Defaults

Parameter	Before	After
decompose	False	True
max_seq_length	5	4
idregex_refine	False	True
_COUNT_CAP	10^12	10^30

Results

Codebase	Grammars	Pure Bags	Structured	Bag %
RAGSAK (v4)	126	6	120	4.8%
FastAPI (v3)	143	26	117	18.2%
Zod (v3)	23	5	18	21.7%

Quality breakdown: ~85% of grammars across codebases remain orderless bags (A|B|C)+. The ~15% that are structured represent real sequential flows:

• Web controller tests: post→jsonPath→isEqualTo→exchange→expectStatus
• API client patterns: request→header→send→statusCode→jsonPath
• Builder chains: builder→field→value→build→validate

iDRegEx Findings

iDRegEx does NOT help at small scale. On 3-method groups, iDRegEx achieves only 3.8× tighter (below the 10× gate threshold). The gate correctly rejects it.

Group size	CRX lang_size	iDRegEx lang_size	Ratio
3 methods	15	4	3.8×
4 methods (storage)	—	—	91× (outlier)

Bags survive because:

1. CRX emits one grammar deterministically (no alternative to compare)
2. lang_size_score only ranks between algorithms, not within CRX's own output
3. iDRegEx is too slow for large groups (200s+ timeout on 2036m FastAPI tests)

Key Insight

The grammar inference pipeline is fundamentally limited by the input: if methods in a package don't share a sequential calling pattern, no algorithm can find one. The ~15% structured grammars represent genuinely reusable patterns; the ~85% bags represent packages with diverse, unrelated methods grouped only by directory proximity.

Files

• bex/grammar.py: AST nodes, _count_concat memoization, _COUNT_CAP = 10^30
• bex/crx.py: CRX algorithm (AST-based)
• bex/mdl.py: lang_size_score, model_cost, data_cost
• bex/idregex.py: iDRegEx algorithm
• bex/decompose.py: sequence decomposition
• bex/tag_preprocessor/analyze.py: pipeline orchestration, all defaults
• experiments/results/round20_ast_verify/: full experiment data (v2/v3/v4)