grammar-inference-engine/experiments/ACHIEVEMENT_SUMMARY.md

Achievement Summary — Grammar Inference Pipeline

What We Built

A source code analysis pipeline that discovers per-package calling conventions from any codebase using tree-sitter AST → behavioral sequences → BEX grammar inference. Zero per-language code, one pipeline for all 10 supported languages.

Pipeline Components

1. Preprocessing (code.py)

   • tree-sitter AST parsing with correct byte/char offset handling
   • Behavioral prefix extraction (BEHAVIORAL_PREFIXES)
   • Token coarsening (RETURN, IF, EXCEPTION, LOOP)
   • Method-level call sequence extraction

2. Grouping (analyze.py)

   • Package slicing by directory
   • Split-by-first-symbol (_recursive_split())
   • Frequency filtering (remove rare symbols)

3. Inference (crx.py, crx_refined.py)

   • CRX: deterministic, fast (2ms), always produces output
   • Refined CRX: cluster-then-infer, tighter grammars on flat bags
   • Optional: iDRegEx (--idregex-refine), kORE (--kore)

4. Validation (gbnf.py)

   • SORE validation (validate_sore())
   • Structure scoring (grammar_structure_score())
   • GBNF conversion for constrained generation

5. Scoring (mdl.py)

   • Language Size (Bex et al.) — primary metric
   • MDL — fallback metric
   • Model cost — grammar complexity

6. Runtime (grammar_index.py, mcp_server.py)

   • GrammarIndex for lookup by (package, context_symbol)
   • MCP tools: get_grammar, get_package_grammars, analyze_directory

Key Fixes

• Byte/char offset mismatch — tree-sitter returns byte offsets, we indexed into strings with byte offsets. Fixed by encoding to bytes first. 653 truncated symbols → 0.
• GBNF parser — + inside () = alternation, outside = repetition. Fixed.
• Symbol sanitization — removed entirely (dead code after byte/char fix).

Experiments Conducted

Round	What	Finding
1	Context strategies	Package grouping is best
2	Reduce algorithm	Doesn't help (contexts too specific)
3-4	kORE, frequency filtering	kORE too slow, sweet spot 0.01-0.05
5-6	SORE/GBNF conversion	Fixed parser, 28 tests
7-8	Token coarsening	Coarsened tokens → fewer flat bags
9-10	CRX over-approximation	24% over-approximated, refined helps
11	Pipeline speed	19s → 2.8s (6.8x) via parallelism
12	Structure scoring	min_structure=0.2 drops flat bags
13	Recursive split	+67% more grammars, avg score 0.45→0.71
14	Byte/char fix	653→0 truncated symbols, 3.4x more grammars
15	kORE/iDRegEx vs CRX	iDRegEx/kORE return None on flat bags
16	iDRegEx refinement	Heuristic: n<=10, opt>50%, 477x improvement
17	CRX vs refined CRX	Refined wins 78% when useful, trivial 36%

What We Achieved

• Pipeline works: 462 Kotlin files → 27 structured grammars (RAGSAK)
• Multi-language: Tested on Kotlin, Python, TypeScript
• Fast: 13s on RAGSAK (with --idregex-refine), 74s without
• Clean symbols: 0 truncated (was 653)
• 28 GBNF tests: Parser handles real-world grammars
• 234 total tests: Full test suite passes

What We Learned

1. CRX is the right default — fast, reliable, always produces something
2. Refined CRX helps on flat bags — cluster-then-infer finds tighter groupings
3. iDRegEx is too situational — returns None on most real data
4. kORE adds nothing — same as iDRegEx but slower
5. Flat bags are grouping problems — no algorithm can find structure where there is none
6. lang_size is the right metric — counts words at each length, prefers tighter grammars

Current State

Pipeline: Fully functional, tested on 4 codebases Default: CRX only, refined opt-in via --crx-method refined Optional: iDRegEx refinement (--idregex-refine), kORE (--kore) Output: YAML by module, GBNF for constrained generation, MCP tools for runtime

What's Next

1. Integrate with LLM — test if grammars actually help code completion
2. Tune min_structure — find the sweet spot for each use case
3. Add more languages — currently 10 supported via tree-sitter
4. Package as skill — reusable agent skill for wiki/MCP integration

Git History

b2c1263 docs: ASCII diagrams for pipeline, parameters, and decision matrix
1d94c09 experiment: CRX vs refined CRX across 3 codebases (Round 17)
b92b765 feat: iDRegEx refinement for CRX flat bags (Round 16)
e9f672c experiment: kORE/iDRegEx vs CRX on flat bags (Round 15)
93d53f1 fix: byte/char offset mismatch in tree-sitter text extraction
...

Key Files

• bex/tag_preprocessor/analyze.py — main pipeline orchestrator
• bex/tag_preprocessor/code.py — tree-sitter preprocessing
• bex/crx.py — CRX algorithm
• bex/crx_refined.py — refined CRX (cluster-then-infer)
• bex/gbnf.py — SORE→GBNF converter
• bex/mdl.py — scoring (lang_size, mdl)
• bex/grammar_index.py — runtime grammar lookup
• bex/mcp_server.py — MCP server with tools
• experiments/DECISION_MATRIX.md — when to use what
• experiments/DIAGRAMS.md — ASCII pipeline diagrams
• experiments/EXPERIMENT_LOG.md — full experiment history