8b2899d16e
Decomposition breaks long sequences into shorter fragments before inference. This helps when sequences are too long for CRX to handle (>5 symbols → flat bags). Results: - RAGSAK: 21 → 80 grammars (3.8× increase) - FastAPI: 111 → 118 grammars (small increase) Changes: - bex/decompose.py: decompose_sequence(), decompose_all(), decompose_with_coverage() - bex/tag_preprocessor/analyze.py: --decompose, --max-seq-length flags - Skip diversity check when decomposing (decomposition creates diverse fragments) - 12 new tests in tests/test_decompose.py Co-authored-by: OpenCode <opencode@corentic.eu> |
||
|---|---|---|
| .. | ||
| test_analyze.py | ||
| test_bex.py | ||
| test_crx_refined.py | ||
| test_decompose.py | ||
| test_distributional.py | ||
| test_ensemble.py | ||
| test_gbnf.py | ||
| test_grammar_index.py | ||
| test_kore.py | ||
| test_reduce.py | ||
| test_scoring.py | ||