tobjend
9be44c2964
feat: add quality gate for grammar usefulness scoring
...
ci/woodpecker/push/woodpecker Pipeline failed
ci/woodpecker/pr/woodpecker Pipeline failed
- Add grammar_quality_score() — scores 0.0-1.0 based on structure
- Add is_useful_grammar() — filters grammars with quality >= 0.3
- Quality criteria: ordering (+0.3), alternation groups (+0.2 each),
symbol count (+0.2), concat depth (+0.1)
- Results: 89/102 RAGSAK, 116/121 FastAPI, 9/10 Zod pass quality gate
- Top 15-20 RAGSAK grammars have genuine domain patterns
2026-07-13 02:32:45 +02:00
tobjend
becbd82c56
feat: add post-hoc noise filtering for grammars
...
ci/woodpecker/push/woodpecker Pipeline failed
ci/woodpecker/pr/woodpecker Pipeline failed
- Add filter_noise() to remove test/stdlib tokens from AST
- Add grammar_noise_ratio() to calculate noise fraction
- Integrate into _build_json_output() and _build_yaml_output()
- Precision improvement: 28-55% → 84-94% across all 3 codebases
- Grammar count unchanged (233) — filtering is post-hoc, preserves recall
2026-07-13 02:23:37 +02:00
tobjend
ea6cac53e3
wip: AST foundation — grammar.py, expr.py→AST, soa.py→AST labels
2026-07-13 01:13:48 +02:00
tobjend
e62fffc6e0
feat: grammar_structure_score + min_structure filter
...
Quantifies how structured a SORE is (0.0=flat bag, 1.0=fully structured).
Flat bags are CRX over-approximations — they list symbols without ordering.
With min_structure=0.2:
Flask: 2 kept (was 5), 9 dropped
RAGSAK: 10 kept (was 19), 114 dropped
FastAPI: 47 kept (was 106), 95 dropped
Total: 59 useful grammars, 218 noise removed
CLI: --min-structure 0.2 (default: 0, keep all)
2026-07-12 03:02:50 +02:00
tobjend
3468813ec8
fix: validate SORE before returning, skip malformed grammars
...
- Add validate_sore() to gbnf.py — checks parseability without converting
- _infer_group now validates grammar and returns skip_reason='malformed_grammar'
for SOREs containing raw code (e.g. w_body=, sult=, (+,+:N+...)
- Results: 130 grammars, 130 GBNF OK, 0 GBNF FAIL
- Flask: 5 OK, 0 FAIL, 3.4s
- RAGSAK: 19 OK, 0 FAIL, 11 malformed, 12.9s
- FastAPI: 106 OK, 0 FAIL, 6 malformed, 30.5s
2026-07-12 02:56:13 +02:00
tobjend
bc7d3b6ca1
perf: iDRegEx opt-in, GBNF newline fix, OverflowError fix
...
- iDRegEx now opt-in via --idregex flag (was running on every group,
causing 55s+ on Flask alone — src/flask/json took 55s in iDRegEx)
- GBNF tokenizer strips newlines from literals (multi-line symbols)
- Fix OverflowError: lang_size_score produces huge ints for large
disjunctions, format as string not float
- Flask: 2.7s (was 55s+), RAGSAK: 13s (was 74s)
2026-07-12 02:52:05 +02:00
tobjend
f57c302c91
fix(gbnf): handle disjunction inside parens and compound repetition
...
The GBNF parser now correctly handles SORE's overloaded + operator:
- + inside (a+b+c) → alternation (not repetition)
- + outside parens → repetition
- +? and +* compound operators → normalized to Star
Also adds implicit concatenation when LPAREN follows a repetition,
so a+(b+c) parses as a+ followed by (b|c).
28 tests pass (13 new disjunction/compound tests). Full suite: 212 passed.
2026-07-12 02:20:58 +02:00
tobjend
011df391c2
feat: implement SORE → GBNF converter
...
- Recursive descent parser for SORE syntax (+, ?, *, |, ., parens)
- AST intermediate representation (_Literal, _Concat, _Alt, _Plus, _Optional, _Star)
- to_gbnf(sore) → full GBNF rule string
- to_gbnf_with_rules(sore, name) → named rule for composition
- 15 tests covering all SORE operators and nesting patterns
2026-07-12 00:31:47 +02:00
