- min_methods: 3→2 (keep groups with 2+ methods)
- unique_ratio: 0.9→0.95 (keep groups with up to 95% unique sequences)
- max_mdl: 200→500 (keep higher-MDL grammars)
This prevents throwing away good grammars that have genuine structure
but happen to have many unique sequences or moderate MDL scores.
- idregex_refine=True default across all entry points + CLI
- Added --no-idregex-refine flag to CLI
- Rewrote _count_optionals from SORE string parser to AST walker
- Added _is_pure_bag() helper for bag detection
- Updated _should_try_idregex() to trigger on pure Plus(Alt) bags
- Updated tests for AST-based _count_optionals
- Added RAGSAK v4 experiment results
B (decompose default):
- analyze_directory + CLI now default decompose=True (--no-decompose to
disable); default max_seq_length 5 -> 4 (matches golden_config).
- Decomposing long sequences into <=4-symbol fragments yields tighter
grammars and disables the too_diverse skip, so far more packages get a
grammar. RAGSAK: 29 -> 95 grammars, pure bags 9 -> 5; fastapi 74 -> 26
pure bags; zod 15 -> 5. All runs exit 0, no stalls.
A (repair Language Size scorer, not a new one):
- _COUNT_CAP was applied to count_words' RETURN value, silently clamping
lang_size_score / model_cost / data_cost at 10^12 for every real codebase
grammar. That broke ADR-13 (Language Size picks most-specific grammar):
bags and tight grammars tied at 10^12, so the scorer could not prefer
specific grammars over generic ones.
- _COUNT_CAP raised to 10**30. Memoization (_count_concat, earlier commit)
already prevents the recursion hang the cap was guarding against, so the
cap no longer needs to clamp scores. lang_size_score now discriminates
(verified: tight=20 vs bag=9975).
During the AST migration, _count_concat lost its @lru_cache (it was
replaced by an uncached recursive version). Distributing a length L
across concat parts then revisited the same (remaining_parts, length)
states exponentially -> 16M function calls for a single 4-method group,
and several RAGSAK groups took 18-52s (looked hung at the tail).
Restore memoization: _count_concat is now @lru_cache-keyed on
(tuple(parts), length). Also fix a stray blank line between the
@lru_cache decorator and count_words.
Impact (RAGSAK, --slice package --min-structure 0.5):
straggler groups 18-52s -> <0.5s; full run 54.9s -> 6.7s.
Adds a regression test asserting a length-20 (a|b)* string counts in <2s.
_vprint now honors the BEX_VERBOSE env var in addition to the CLI
--verbose flag, so non-CLI callers (MCP, dervish integration) and
worker processes (which start with a fresh module) emit logs too.
--verbose exports BEX_VERBOSE=1 so ProcessPoolExecutor workers inherit it.
analyze_by_package now logs real-time per-group completion with
[i/N] label (count methods) done (Xs), making it obvious the parallel
pipeline is progressing and not hung.
- Plan: match rate audit, decomposition A/B, distributional A/B, fragment quality
- Golden config: best-known values for all heuristic parameters
- Single source of truth in bex/golden_config.py
- Removed all_grammars expansion in _build_yaml_output (226→54 RAGSAK, 384→87 FastAPI)
- Added gbnf field to each YAML entry via to_gbnf()
- Updated test_roundtrip_persisted_ragsak to use correct file path
Closes#66
Heuristic: only run iDRegEx when n_methods ≤ 10 AND CRX grammar has
>50% top-level optional parts (flat chain signal). If iDRegEx grammar
is >10x tighter by lang_size, use it. Otherwise keep CRX.
RAGSAK result: agents/capability (5 methods) refined from
slot?.(defaultCapabilityId+summarize)?... (lang_size=1432) to
(defaultCapabilityId|summarize) (lang_size=3) — 477x tighter.
Speed cost: ~0.7s per candidate, negligible on 74s pipeline.
CLI: --idregex-refine flag (default off).
Also adds _count_optionals() and _should_try_idregex() helpers
with 8 pytest tests. 234 tests pass.
MCP server:
- analyze_directory: updated signature to match CLI (split_mixed,
min_structure, min_methods, min_coverage=0.05)
- get_grammar(directory, file_path, context_symbol): returns the right
GBNF for constrained generation at code generation time
- get_package_grammars(directory, file_path): lists all grammars for
a file's package, ranked by quality
YAML output:
- _build_yaml_output now includes leaf grammars from recursive split
(all_grammars in meta), so each calling context gets its own entry
Workflow for agents:
1. analyze_directory → grammars.yml persisted
2. get_grammar(file, 'return') → GBNF for constrained generation
3. LLM generates code following the package's convention
Add GrammarIndex class that loads grammars.yml and provides fast
lookup by (package, context_symbol). Enables agents to get the right
GBNF grammar at code generation time.
API:
load_grammar_index(project_root) → GrammarIndex
idx.get(file_path, context_symbol=None) → gbnf_string
idx.get_package(file_path) → [(symbol, grammar, score, methods)]
14 tests, all passing.
Replace single-level split with recursive splitting that drills deeper
into mixed-pattern groups. Each leaf group produces its own grammar,
and the best leaf is returned per parent group.
Results:
FastAPI: 22 → 32 grammars (+45%), 8 → 17 high-quality (>=0.5) (+113%)
RAGSAK: 5 → 8 grammars (+60%), 2 → 6 high-quality (+200%)
Flask: 0 → 2 grammars (was zero, now produces output)
The improvement comes from capturing grammars in groups that previously
couldn't produce one at all — sub-groups of 3-5 methods that are too
small for single-split but contain clear patterns (e.g., all 'return'
or all 'if' sequences).
Add --split-mixed flag that detects groups with diverse first symbols
and splits them before CRX. This separates distinct calling patterns
that CRX would otherwise merge into flat bags.
Results:
- FastAPI: 22→33 grammars, avg score 0.47→0.57
- RAGSAK: 5→8 grammars, avg score 0.45→0.71
- dependency_testing: bag→clean 'return.q?.commons?.skip?.limit?' (1.00)
212 tests pass
min_methods=3 gains 9 extra FastAPI grammars (3-4 method groups with
clear sequential structure) without quality loss. No change for Flask
or RAGSAK (their small groups fail other gates anyway).
Multi-codebase validation (Round 13): tested Flask, FastAPI, httpx,
Pydantic, SQLAlchemy. Key finding: CRX works on codebases with small
focused sequential modules (FastAPI docs_src=22 grammars) but not on
large utility libraries (Flask=0, Pydantic=0, SQLAlchemy=1).
The default min_coverage of 0.8 was filtering out almost all symbols
before CRX could see them, making every other parameter tuning moot.
At 0.8, only symbols present in 80%+ of methods survive — which means
0-2 symbols per group. At 0.05, 46 symbols survive for fastapi/src
and CRX can actually find structure.
This was the root cause of why max_unique_symbols gate never triggered
and why CRX kept producing flat bags.
- iDRegEx now opt-in via --idregex flag (was running on every group,
causing 55s+ on Flask alone — src/flask/json took 55s in iDRegEx)
- GBNF tokenizer strips newlines from literals (multi-line symbols)
- Fix OverflowError: lang_size_score produces huge ints for large
disjunctions, format as string not float
- Flask: 2.7s (was 55s+), RAGSAK: 13s (was 74s)
The GBNF parser now correctly handles SORE's overloaded + operator:
- + inside (a+b+c) → alternation (not repetition)
- + outside parens → repetition
- +? and +* compound operators → normalized to Star
Also adds implicit concatenation when LPAREN follows a repetition,
so a+(b+c) parses as a+ followed by (b|c).
28 tests pass (13 new disjunction/compound tests). Full suite: 212 passed.
- Add --crx-method (standard|refined) to CLI and analyze_directory
- Pass crx_method through analyze_by_package → _infer_group
- Fix hardcoded min_coverage=0.2 in 3 places (was ignoring parameter)
- Refined CRX wraps crx_with_confidence into standard result format
Standard CRX over-approximates when Hasse diagram is non-linear (24% of
RAGSAK packages). Cluster-then-infer groups sequences by (first, last,
length), infers per-cluster, picks largest cluster's grammar.
Results on RAGSAK:
Avg max disjunction: 2.8 → 1.7 (39% tighter)
Packages improved: 6/10
Tradeoff: cluster granularity (too coarse = over-approximation,
too fine = no generalization). Current: (first, last, length_bucket).
Exports crx_refined() and crx_with_confidence() from bex package.
20 new tests. All 199 tests pass.
Adds --min-methods CLI flag (default 5). Groups that are too diverse or
too small are skipped with skip_reason in meta. Prevents noisy/meaningless
grammars from diverse packages.
Add _kore_trial() module-level worker for pickling. infer() accepts
n_workers param — >1 runs all (k x N) trials concurrently instead
of serial. Default 1 preserves existing behavior.
Thread n_workers through _run_kore() and infer_ensemble() to support
--kore flag with parallel kORE inference.
Replace serial file-read + tree-sitter parse loop with _preprocess_files()
using ProcessPoolExecutor. Module-level _preprocess_file() for pickling.
Covers all three callers: analyze_clusters, analyze_by_package, infer.
- Remove kORE from top-level imports, _ALGORITHMS, and infer_ensemble body
- Add include_kore=False parameter, runs kORE only when opted in
- Add --kore CLI flag threaded through all pipeline layers
- Remove koreinference from --prefer choices
- Keep kore.py module in repo for reference/future use
- Update tests: remove test_prefer_koreinference, update algorithm assertions
- CALL_PREFIXES: add bare "function" as last fallback — Kotlin uses
@function for both calls and definitions (no function.call capture)
- _find_arglist_node: remove template_string from arglist detection
- IMPORT_PATTERNS: require_relative before require (Ruby fix)
- pipeline-overview.txt: remove .gitignore, add min_coverage=0.8 + core/outlier
- DEFAULT_COVERAGE=0.8, frequency_filter at fixed 0.2
- frequency_filter(min_coverage=0.2) strips rare symbols before clustering/inference
- DEFAULT_COVERAGE=0.8 passed to infer_ensemble for sequence-level core/outlier detection
- Both filters active: symbol-level (0.2) then BEX sequence-level (0.8)
- Fix seq_of_file tracking broken by filter creating new list objects
- _extract_call_tokens filters to call-like captures only
- cluster_methods groups sequences by shared 3-gram call patterns
- analyze_clusters runs per-cluster CRX/iDRegEx/kORE inference
- Lists 17 convention clusters for RAGSAK test code
- iDRegEx and kOREInference now produce ordered grammars per cluster
- (other) cluster captures diverse conventions as CRX vocabulary
- min-cluster-size (default 3) and ngram-size (default 3) CLI flags
- _find_method_bodies uses tree-sitter's universal body field (9/10 grammars)
- Kotlin fallback: scan children for body-like types
- preprocess_by_method groups highlight captures by enclosing method body
- Returns per-method sequences for k-ORE ordering analysis
- analyze.py infer() now uses preprocess_by_method
- Document findings in ANALYSIS.md
- 97 tests pass
