tobi/grammar-inference-engine
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
grammar-inference-engine/bex
History
tobjend 8b2899d16e Phase 2: decomposition forest for complex sequences 
Decomposition breaks long sequences into shorter fragments before
inference. This helps when sequences are too long for CRX to handle
(>5 symbols → flat bags).

Results:
- RAGSAK: 21 → 80 grammars (3.8× increase)
- FastAPI: 111 → 118 grammars (small increase)

Changes:
- bex/decompose.py: decompose_sequence(), decompose_all(), decompose_with_coverage()
- bex/tag_preprocessor/analyze.py: --decompose, --max-seq-length flags
- Skip diversity check when decomposing (decomposition creates diverse fragments)
- 12 new tests in tests/test_decompose.py

Co-authored-by: OpenCode <opencode@corentic.eu>
2026-07-12 17:49:35 +02:00
..
tag_preprocessor Phase 2: decomposition forest for complex sequences 2026-07-12 17:49:35 +02:00
__init__.py feat: CRX refined — cluster-then-infer for tighter grammars 2026-07-12 01:50:40 +02:00
__main__.py Initial commit: BEX-based grammar inference engine 2026-07-01 08:01:16 +02:00
ast_to_gbnf.py Phase 2: decomposition forest for complex sequences 2026-07-12 17:49:35 +02:00
automaton.py Initial commit: BEX-based grammar inference engine 2026-07-01 08:01:16 +02:00
baum_welch.py Initial commit: BEX-based grammar inference engine 2026-07-01 08:01:16 +02:00
cli.py purge make_charts.py, examples/, full-text papers, blog_post (moved to ~/Desktop/kesai/); translate German CLI to English 2026-07-01 11:28:42 +02:00
crx.py Initial commit: BEX-based grammar inference engine 2026-07-01 08:01:16 +02:00
crx_refined.py feat: CRX refined — cluster-then-infer for tighter grammars 2026-07-12 01:50:40 +02:00
decompose.py Phase 2: decomposition forest for complex sequences 2026-07-12 17:49:35 +02:00
distributional.py feat: distributional clustering for better grouping (Phase 1) 2026-07-12 17:37:36 +02:00
ensemble.py perf: iDRegEx opt-in, GBNF newline fix, OverflowError fix 2026-07-12 02:52:05 +02:00
expr.py Initial commit: BEX-based grammar inference engine 2026-07-01 08:01:16 +02:00
gbnf.py feat: grammar_structure_score + min_structure filter 2026-07-12 03:02:50 +02:00
grammar_index.py feat: grammar index for runtime lookup 2026-07-12 14:08:05 +02:00
idregex.py Initial commit: BEX-based grammar inference engine 2026-07-01 08:01:16 +02:00
ikoa.py Initial commit: BEX-based grammar inference engine 2026-07-01 08:01:16 +02:00
ilocal.py fix badge position; purge remaining German user-reference comments 2026-07-01 13:28:55 +02:00
koa.py Initial commit: BEX-based grammar inference engine 2026-07-01 08:01:16 +02:00
kore.py Revert "parallelize kORE outer (k, n) trials via ProcessPoolExecutor" 2026-07-11 20:36:48 +02:00
marking.py Initial commit: BEX-based grammar inference engine 2026-07-01 08:01:16 +02:00
mcp_server.py feat: runtime grammar lookup via MCP + fix YAML output 2026-07-12 14:08:12 +02:00
mdl.py WIP: language size scoring + diversity threshold (step 1 pending) 2026-07-11 22:56:42 +02:00
pta.py Initial commit: BEX-based grammar inference engine 2026-07-01 08:01:16 +02:00
reduce.py feat: add reduce slicing, fix rwr0 timeout, fix sequence format 2026-07-12 13:13:16 +02:00
repair.py Initial commit: BEX-based grammar inference engine 2026-07-01 08:01:16 +02:00
rwr0.py feat: add reduce slicing, fix rwr0 timeout, fix sequence format 2026-07-12 13:13:16 +02:00
rwrsq.py Initial commit: BEX-based grammar inference engine 2026-07-01 08:01:16 +02:00
shrink.py Initial commit: BEX-based grammar inference engine 2026-07-01 08:01:16 +02:00
soa.py Initial commit: BEX-based grammar inference engine 2026-07-01 08:01:16 +02:00
template.py fix badge position; purge remaining German user-reference comments 2026-07-01 13:28:55 +02:00
tokenizer.py fix badge position; purge remaining German user-reference comments 2026-07-01 13:28:55 +02:00
twotinf.py Initial commit: BEX-based grammar inference engine 2026-07-01 08:01:16 +02:00