tobjend
e62fffc6e0
feat: grammar_structure_score + min_structure filter
...
Quantifies how structured a SORE is (0.0=flat bag, 1.0=fully structured).
Flat bags are CRX over-approximations — they list symbols without ordering.
With min_structure=0.2:
Flask: 2 kept (was 5), 9 dropped
RAGSAK: 10 kept (was 19), 114 dropped
FastAPI: 47 kept (was 106), 95 dropped
Total: 59 useful grammars, 218 noise removed
CLI: --min-structure 0.2 (default: 0, keep all)
2026-07-12 03:02:50 +02:00
tobjend
3468813ec8
fix: validate SORE before returning, skip malformed grammars
...
- Add validate_sore() to gbnf.py — checks parseability without converting
- _infer_group now validates grammar and returns skip_reason='malformed_grammar'
for SOREs containing raw code (e.g. w_body=, sult=, (+,+:N+...)
- Results: 130 grammars, 130 GBNF OK, 0 GBNF FAIL
- Flask: 5 OK, 0 FAIL, 3.4s
- RAGSAK: 19 OK, 0 FAIL, 11 malformed, 12.9s
- FastAPI: 106 OK, 0 FAIL, 6 malformed, 30.5s
2026-07-12 02:56:13 +02:00
tobjend
bc7d3b6ca1
perf: iDRegEx opt-in, GBNF newline fix, OverflowError fix
...
- iDRegEx now opt-in via --idregex flag (was running on every group,
causing 55s+ on Flask alone — src/flask/json took 55s in iDRegEx)
- GBNF tokenizer strips newlines from literals (multi-line symbols)
- Fix OverflowError: lang_size_score produces huge ints for large
disjunctions, format as string not float
- Flask: 2.7s (was 55s+), RAGSAK: 13s (was 74s)
2026-07-12 02:52:05 +02:00
