tobi/grammar-inference-engine
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
grammar-inference-engine/experiments
History
tobjend 8b2899d16e Phase 2: decomposition forest for complex sequences 
Decomposition breaks long sequences into shorter fragments before
inference. This helps when sequences are too long for CRX to handle
(>5 symbols → flat bags).

Results:
- RAGSAK: 21 → 80 grammars (3.8× increase)
- FastAPI: 111 → 118 grammars (small increase)

Changes:
- bex/decompose.py: decompose_sequence(), decompose_all(), decompose_with_coverage()
- bex/tag_preprocessor/analyze.py: --decompose, --max-seq-length flags
- Skip diversity check when decomposing (decomposition creates diverse fragments)
- 12 new tests in tests/test_decompose.py

Co-authored-by: OpenCode <opencode@corentic.eu>
2026-07-12 17:49:35 +02:00
..
results feat: grammar_structure_score + min_structure filter 2026-07-12 03:02:50 +02:00
round14 docs: achievement summary and experiment outputs (Round 14-15) 2026-07-12 17:25:15 +02:00
round15 docs: achievement summary and experiment outputs (Round 14-15) 2026-07-12 17:25:15 +02:00
ACHIEVEMENT_SUMMARY.md docs: achievement summary and experiment outputs (Round 14-15) 2026-07-12 17:25:15 +02:00
coarsen_eval.py feat: 4-codebase evaluation — conventions vs completions tradeoff 2026-07-12 01:14:09 +02:00
context_eval.py feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 2026-07-12 00:11:38 +02:00
DECISION_MATRIX.md docs: research positioning and decision matrix 2026-07-12 17:14:56 +02:00
DIAGRAMS.md docs: ASCII diagrams for pipeline, parameters, and decision matrix 2026-07-12 17:09:40 +02:00
EXPERIMENT_LOG.md experiment: CRX vs refined CRX across 3 codebases (Round 17) 2026-07-12 17:08:15 +02:00
freq_eval.py feat: frequency threshold sweep — 0.01-0.20 across 4 codebases 2026-07-12 01:26:45 +02:00
gbnf_eval.py perf: iDRegEx opt-in, GBNF newline fix, OverflowError fix 2026-07-12 02:52:05 +02:00
PHASE1_PLAN.md feat: distributional clustering for better grouping (Phase 1) 2026-07-12 17:37:36 +02:00
PHASE2_PLAN.md Phase 2: decomposition forest for complex sequences 2026-07-12 17:49:35 +02:00
RESEARCH_POSITIONING.md docs: research positioning and decision matrix 2026-07-12 17:14:56 +02:00
RESULTS.md docs: experiment log + cross-package analysis 2026-07-12 00:45:21 +02:00