b516b2985d
- bex/reduce.py: Faithful implementation of Reduce with support-weighted SOA edit distance, adjunction, iterative merging, and Minimize - experiments/context_eval.py: Multi-codebase support (RAGSAK + Flask), Reduce experiments with thresholds 0.05-0.4 - tests/test_reduce.py: 24 tests covering all Reduce components - Flask cloned to external_refs/flask for cross-validation Results: - RAGSAK: 12.0% coverage (First 3 symbols) - Flask: 10.7% coverage (First 3 symbols) - Reduce has minimal impact (1-2 merges per codebase at ε=0.3) - Coverage ceiling appears to be ~10-12% for prefix-based grouping
28 lines
924 B
Markdown
28 lines
924 B
Markdown
# TODO: GBNF Output Format
|
|
|
|
When grammars are finalized, output them in GBNF (GGML BNF) format for compatibility with llama.cpp and other inference engines.
|
|
|
|
GBNF is a BNF-like grammar format used by llama.cpp for constrained decoding. It supports:
|
|
- Sequences: `rule ::= token1 token2`
|
|
- Alternatives: `rule ::= alt1 | alt2`
|
|
- Optional: `rule ::= (token)?`
|
|
- Repetition: `rule ::= (token)*`
|
|
- Character classes: `[a-z]`, `[^abc]`
|
|
|
|
Example GBNF:
|
|
```
|
|
root ::= ws? item ws?
|
|
item ::= identifier ws? "(" ws? args? ws? ")"
|
|
args ::= identifier (ws? "," ws? identifier)*
|
|
identifier ::= [a-zA-Z_][a-zA-Z0-9_]*
|
|
ws ::= [ \t\n]*
|
|
```
|
|
|
|
Map SORE operators to GBNF:
|
|
- `r·s` (concatenation) → `rule ::= r s`
|
|
- `r|s` (disjunction) → `rule ::= r | s`
|
|
- `r*` (star) → `rule ::= (r)*`
|
|
- `r?` (optional) → `rule ::= (r)?`
|
|
- `r+` (plus) → `rule ::= r (r)*`
|
|
|
|
Implementation: Add `to_gbnf(sore)` function to `bex/` when ready.
|