B (decompose default):
- analyze_directory + CLI now default decompose=True (--no-decompose to
disable); default max_seq_length 5 -> 4 (matches golden_config).
- Decomposing long sequences into <=4-symbol fragments yields tighter
grammars and disables the too_diverse skip, so far more packages get a
grammar. RAGSAK: 29 -> 95 grammars, pure bags 9 -> 5; fastapi 74 -> 26
pure bags; zod 15 -> 5. All runs exit 0, no stalls.
A (repair Language Size scorer, not a new one):
- _COUNT_CAP was applied to count_words' RETURN value, silently clamping
lang_size_score / model_cost / data_cost at 10^12 for every real codebase
grammar. That broke ADR-13 (Language Size picks most-specific grammar):
bags and tight grammars tied at 10^12, so the scorer could not prefer
specific grammars over generic ones.
- _COUNT_CAP raised to 10**30. Memoization (_count_concat, earlier commit)
already prevents the recursion hang the cap was guarding against, so the
cap no longer needs to clamp scores. lang_size_score now discriminates
(verified: tight=20 vs bag=9975).
Run RAGSAK (Kotlin), fastapi (Python), zod (TypeScript) with --slice
package --min-structure 0.5, verbose+timeout, results saved for later.
After _count_concat memoization fix: RAGSAK full run 54.9s -> 6.7s,
no tail stalls. RAGSAK: 121 packages (28 with grammar). fastapi/zod
complete in ~25s (dominated by arg-extraction on 2000+ method test
groups, not grammar explosion). Includes SUMMARY.md with before/after.
During the AST migration, _count_concat lost its @lru_cache (it was
replaced by an uncached recursive version). Distributing a length L
across concat parts then revisited the same (remaining_parts, length)
states exponentially -> 16M function calls for a single 4-method group,
and several RAGSAK groups took 18-52s (looked hung at the tail).
Restore memoization: _count_concat is now @lru_cache-keyed on
(tuple(parts), length). Also fix a stray blank line between the
@lru_cache decorator and count_words.
Impact (RAGSAK, --slice package --min-structure 0.5):
straggler groups 18-52s -> <0.5s; full run 54.9s -> 6.7s.
Adds a regression test asserting a length-20 (a|b)* string counts in <2s.
_vprint now honors the BEX_VERBOSE env var in addition to the CLI
--verbose flag, so non-CLI callers (MCP, dervish integration) and
worker processes (which start with a fresh module) emit logs too.
--verbose exports BEX_VERBOSE=1 so ProcessPoolExecutor workers inherit it.
analyze_by_package now logs real-time per-group completion with
[i/N] label (count methods) done (Xs), making it obvious the parallel
pipeline is progressing and not hung.
- Plan: match rate audit, decomposition A/B, distributional A/B, fragment quality
- Golden config: best-known values for all heuristic parameters
- Single source of truth in bex/golden_config.py
- Removed all_grammars expansion in _build_yaml_output (226→54 RAGSAK, 384→87 FastAPI)
- Added gbnf field to each YAML entry via to_gbnf()
- Updated test_roundtrip_persisted_ragsak to use correct file path
Closes#66
- ACHIEVEMENT_SUMMARY.md: what we built and learned
- Round 14: Full pipeline run on 4 codebases (clean symbols)
- Round 15: kORE/iDRegEx vs CRX comparison
Next: Crucio-inspired clustering for better grouping
- RESEARCH_POSITIONING.md: where we fit in the science
- DECISION_MATRIX.md: when to use CRX vs refined CRX
- Updated AGENTS.md with research context and updated CLI flags
Refined wins 7/9 when producing useful grammars (model_cost >= 2).
Trivial output (single symbol) in 5/13 cases on large groups.
CRX wins only once (FastAPI tests, 3618 methods).
Key finding: refined CRX is better ~78% of the time but needs
triviality check (model_cost >= 2) to avoid single-symbol grammars.
Heuristic: only run iDRegEx when n_methods ≤ 10 AND CRX grammar has
>50% top-level optional parts (flat chain signal). If iDRegEx grammar
is >10x tighter by lang_size, use it. Otherwise keep CRX.
RAGSAK result: agents/capability (5 methods) refined from
slot?.(defaultCapabilityId+summarize)?... (lang_size=1432) to
(defaultCapabilityId|summarize) (lang_size=3) — 477x tighter.
Speed cost: ~0.7s per candidate, negligible on 74s pipeline.
CLI: --idregex-refine flag (default off).
Also adds _count_optionals() and _should_try_idregex() helpers
with 8 pytest tests. 234 tests pass.
Key finding: iDRegEx/kORE return None on genuinely diverse groups (flat bags).
CRX is the only algorithm that produces anything for these groups.
For structured groups, CRX already captures ordering well.
Conclusion: kORE/iDRegEx are not better fallbacks for flat bags.
The pipeline's existing filtering (min_structure, split_mixed) is the right approach.
MCP server:
- analyze_directory: updated signature to match CLI (split_mixed,
min_structure, min_methods, min_coverage=0.05)
- get_grammar(directory, file_path, context_symbol): returns the right
GBNF for constrained generation at code generation time
- get_package_grammars(directory, file_path): lists all grammars for
a file's package, ranked by quality
YAML output:
- _build_yaml_output now includes leaf grammars from recursive split
(all_grammars in meta), so each calling context gets its own entry
Workflow for agents:
1. analyze_directory → grammars.yml persisted
2. get_grammar(file, 'return') → GBNF for constrained generation
3. LLM generates code following the package's convention
Add GrammarIndex class that loads grammars.yml and provides fast
lookup by (package, context_symbol). Enables agents to get the right
GBNF grammar at code generation time.
API:
load_grammar_index(project_root) → GrammarIndex
idx.get(file_path, context_symbol=None) → gbnf_string
idx.get_package(file_path) → [(symbol, grammar, score, methods)]
14 tests, all passing.
Replace single-level split with recursive splitting that drills deeper
into mixed-pattern groups. Each leaf group produces its own grammar,
and the best leaf is returned per parent group.
Results:
FastAPI: 22 → 32 grammars (+45%), 8 → 17 high-quality (>=0.5) (+113%)
RAGSAK: 5 → 8 grammars (+60%), 2 → 6 high-quality (+200%)
Flask: 0 → 2 grammars (was zero, now produces output)
The improvement comes from capturing grammars in groups that previously
couldn't produce one at all — sub-groups of 3-5 methods that are too
small for single-split but contain clear patterns (e.g., all 'return'
or all 'if' sequences).
Add --split-mixed flag that detects groups with diverse first symbols
and splits them before CRX. This separates distinct calling patterns
that CRX would otherwise merge into flat bags.
Results:
- FastAPI: 22→33 grammars, avg score 0.47→0.57
- RAGSAK: 5→8 grammars, avg score 0.45→0.71
- dependency_testing: bag→clean 'return.q?.commons?.skip?.limit?' (1.00)
212 tests pass
min_methods=3 gains 9 extra FastAPI grammars (3-4 method groups with
clear sequential structure) without quality loss. No change for Flask
or RAGSAK (their small groups fail other gates anyway).
Multi-codebase validation (Round 13): tested Flask, FastAPI, httpx,
Pydantic, SQLAlchemy. Key finding: CRX works on codebases with small
focused sequential modules (FastAPI docs_src=22 grammars) but not on
large utility libraries (Flask=0, Pydantic=0, SQLAlchemy=1).
The default min_coverage of 0.8 was filtering out almost all symbols
before CRX could see them, making every other parameter tuning moot.
At 0.8, only symbols present in 80%+ of methods survive — which means
0-2 symbols per group. At 0.05, 46 symbols survive for fastapi/src
and CRX can actually find structure.
This was the root cause of why max_unique_symbols gate never triggered
and why CRX kept producing flat bags.
- iDRegEx now opt-in via --idregex flag (was running on every group,
causing 55s+ on Flask alone — src/flask/json took 55s in iDRegEx)
- GBNF tokenizer strips newlines from literals (multi-line symbols)
- Fix OverflowError: lang_size_score produces huge ints for large
disjunctions, format as string not float
- Flask: 2.7s (was 55s+), RAGSAK: 13s (was 74s)
The GBNF parser now correctly handles SORE's overloaded + operator:
- + inside (a+b+c) → alternation (not repetition)
- + outside parens → repetition
- +? and +* compound operators → normalized to Star
Also adds implicit concatenation when LPAREN follows a repetition,
so a+(b+c) parses as a+ followed by (b|c).
28 tests pass (13 new disjunction/compound tests). Full suite: 212 passed.
- Add --crx-method (standard|refined) to CLI and analyze_directory
- Pass crx_method through analyze_by_package → _infer_group
- Fix hardcoded min_coverage=0.2 in 3 places (was ignoring parameter)
- Refined CRX wraps crx_with_confidence into standard result format
Standard CRX over-approximates when Hasse diagram is non-linear (24% of
RAGSAK packages). Cluster-then-infer groups sequences by (first, last,
length), infers per-cluster, picks largest cluster's grammar.
Results on RAGSAK:
Avg max disjunction: 2.8 → 1.7 (39% tighter)
Packages improved: 6/10
Tradeoff: cluster granularity (too coarse = over-approximation,
too fine = no generalization). Current: (first, last, length_bucket).
Exports crx_refined() and crx_with_confidence() from bex package.
20 new tests. All 199 tests pass.
Sweet spot: 0.01-0.05. At 0.01 RAGSAK gets 61 SOREs (26.1% cov) with
real conventions like warn.status.body.ErrorResponse. At 0.05 coverage
jumps to 45.4% but that pattern disappears. Flask dies at 0.15+.
Current default min_coverage=0.2 is too aggressive for most codebases.
Adds --min-methods CLI flag (default 5). Groups that are too diverse or
too small are skipped with skip_reason in meta. Prevents noisy/meaningless
grammars from diverse packages.
