b516b2985d
- bex/reduce.py: Faithful implementation of Reduce with support-weighted SOA edit distance, adjunction, iterative merging, and Minimize - experiments/context_eval.py: Multi-codebase support (RAGSAK + Flask), Reduce experiments with thresholds 0.05-0.4 - tests/test_reduce.py: 24 tests covering all Reduce components - Flask cloned to external_refs/flask for cross-validation Results: - RAGSAK: 12.0% coverage (First 3 symbols) - Flask: 10.7% coverage (First 3 symbols) - Reduce has minimal impact (1-2 merges per codebase at ε=0.3) - Coverage ceiling appears to be ~10-12% for prefix-based grouping
924 B
924 B
TODO: GBNF Output Format
When grammars are finalized, output them in GBNF (GGML BNF) format for compatibility with llama.cpp and other inference engines.
GBNF is a BNF-like grammar format used by llama.cpp for constrained decoding. It supports:
- Sequences:
rule ::= token1 token2 - Alternatives:
rule ::= alt1 | alt2 - Optional:
rule ::= (token)? - Repetition:
rule ::= (token)* - Character classes:
[a-z],[^abc]
Example GBNF:
root ::= ws? item ws?
item ::= identifier ws? "(" ws? args? ws? ")"
args ::= identifier (ws? "," ws? identifier)*
identifier ::= [a-zA-Z_][a-zA-Z0-9_]*
ws ::= [ \t\n]*
Map SORE operators to GBNF:
r·s(concatenation) →rule ::= r sr|s(disjunction) →rule ::= r | sr*(star) →rule ::= (r)*r?(optional) →rule ::= (r)?r+(plus) →rule ::= r (r)*
Implementation: Add to_gbnf(sore) function to bex/ when ready.