Heuristic: only run iDRegEx when n_methods ≤ 10 AND CRX grammar has
>50% top-level optional parts (flat chain signal). If iDRegEx grammar
is >10x tighter by lang_size, use it. Otherwise keep CRX.
RAGSAK result: agents/capability (5 methods) refined from
slot?.(defaultCapabilityId+summarize)?... (lang_size=1432) to
(defaultCapabilityId|summarize) (lang_size=3) — 477x tighter.
Speed cost: ~0.7s per candidate, negligible on 74s pipeline.
CLI: --idregex-refine flag (default off).
Also adds _count_optionals() and _should_try_idregex() helpers
with 8 pytest tests. 234 tests pass.
MCP server:
- analyze_directory: updated signature to match CLI (split_mixed,
min_structure, min_methods, min_coverage=0.05)
- get_grammar(directory, file_path, context_symbol): returns the right
GBNF for constrained generation at code generation time
- get_package_grammars(directory, file_path): lists all grammars for
a file's package, ranked by quality
YAML output:
- _build_yaml_output now includes leaf grammars from recursive split
(all_grammars in meta), so each calling context gets its own entry
Workflow for agents:
1. analyze_directory → grammars.yml persisted
2. get_grammar(file, 'return') → GBNF for constrained generation
3. LLM generates code following the package's convention
Add GrammarIndex class that loads grammars.yml and provides fast
lookup by (package, context_symbol). Enables agents to get the right
GBNF grammar at code generation time.
API:
load_grammar_index(project_root) → GrammarIndex
idx.get(file_path, context_symbol=None) → gbnf_string
idx.get_package(file_path) → [(symbol, grammar, score, methods)]
14 tests, all passing.
Replace single-level split with recursive splitting that drills deeper
into mixed-pattern groups. Each leaf group produces its own grammar,
and the best leaf is returned per parent group.
Results:
FastAPI: 22 → 32 grammars (+45%), 8 → 17 high-quality (>=0.5) (+113%)
RAGSAK: 5 → 8 grammars (+60%), 2 → 6 high-quality (+200%)
Flask: 0 → 2 grammars (was zero, now produces output)
The improvement comes from capturing grammars in groups that previously
couldn't produce one at all — sub-groups of 3-5 methods that are too
small for single-split but contain clear patterns (e.g., all 'return'
or all 'if' sequences).
Add --split-mixed flag that detects groups with diverse first symbols
and splits them before CRX. This separates distinct calling patterns
that CRX would otherwise merge into flat bags.
Results:
- FastAPI: 22→33 grammars, avg score 0.47→0.57
- RAGSAK: 5→8 grammars, avg score 0.45→0.71
- dependency_testing: bag→clean 'return.q?.commons?.skip?.limit?' (1.00)
212 tests pass
min_methods=3 gains 9 extra FastAPI grammars (3-4 method groups with
clear sequential structure) without quality loss. No change for Flask
or RAGSAK (their small groups fail other gates anyway).
Multi-codebase validation (Round 13): tested Flask, FastAPI, httpx,
Pydantic, SQLAlchemy. Key finding: CRX works on codebases with small
focused sequential modules (FastAPI docs_src=22 grammars) but not on
large utility libraries (Flask=0, Pydantic=0, SQLAlchemy=1).
The default min_coverage of 0.8 was filtering out almost all symbols
before CRX could see them, making every other parameter tuning moot.
At 0.8, only symbols present in 80%+ of methods survive — which means
0-2 symbols per group. At 0.05, 46 symbols survive for fastapi/src
and CRX can actually find structure.
This was the root cause of why max_unique_symbols gate never triggered
and why CRX kept producing flat bags.
- iDRegEx now opt-in via --idregex flag (was running on every group,
causing 55s+ on Flask alone — src/flask/json took 55s in iDRegEx)
- GBNF tokenizer strips newlines from literals (multi-line symbols)
- Fix OverflowError: lang_size_score produces huge ints for large
disjunctions, format as string not float
- Flask: 2.7s (was 55s+), RAGSAK: 13s (was 74s)
The GBNF parser now correctly handles SORE's overloaded + operator:
- + inside (a+b+c) → alternation (not repetition)
- + outside parens → repetition
- +? and +* compound operators → normalized to Star
Also adds implicit concatenation when LPAREN follows a repetition,
so a+(b+c) parses as a+ followed by (b|c).
28 tests pass (13 new disjunction/compound tests). Full suite: 212 passed.
- Add --crx-method (standard|refined) to CLI and analyze_directory
- Pass crx_method through analyze_by_package → _infer_group
- Fix hardcoded min_coverage=0.2 in 3 places (was ignoring parameter)
- Refined CRX wraps crx_with_confidence into standard result format
Standard CRX over-approximates when Hasse diagram is non-linear (24% of
RAGSAK packages). Cluster-then-infer groups sequences by (first, last,
length), infers per-cluster, picks largest cluster's grammar.
Results on RAGSAK:
Avg max disjunction: 2.8 → 1.7 (39% tighter)
Packages improved: 6/10
Tradeoff: cluster granularity (too coarse = over-approximation,
too fine = no generalization). Current: (first, last, length_bucket).
Exports crx_refined() and crx_with_confidence() from bex package.
20 new tests. All 199 tests pass.
Adds --min-methods CLI flag (default 5). Groups that are too diverse or
too small are skipped with skip_reason in meta. Prevents noisy/meaningless
grammars from diverse packages.
Add _kore_trial() module-level worker for pickling. infer() accepts
n_workers param — >1 runs all (k x N) trials concurrently instead
of serial. Default 1 preserves existing behavior.
Thread n_workers through _run_kore() and infer_ensemble() to support
--kore flag with parallel kORE inference.
Replace serial file-read + tree-sitter parse loop with _preprocess_files()
using ProcessPoolExecutor. Module-level _preprocess_file() for pickling.
Covers all three callers: analyze_clusters, analyze_by_package, infer.
- Remove kORE from top-level imports, _ALGORITHMS, and infer_ensemble body
- Add include_kore=False parameter, runs kORE only when opted in
- Add --kore CLI flag threaded through all pipeline layers
- Remove koreinference from --prefer choices
- Keep kore.py module in repo for reference/future use
- Update tests: remove test_prefer_koreinference, update algorithm assertions
- CALL_PREFIXES: add bare "function" as last fallback — Kotlin uses
@function for both calls and definitions (no function.call capture)
- _find_arglist_node: remove template_string from arglist detection
- IMPORT_PATTERNS: require_relative before require (Ruby fix)
- pipeline-overview.txt: remove .gitignore, add min_coverage=0.8 + core/outlier
- DEFAULT_COVERAGE=0.8, frequency_filter at fixed 0.2
- frequency_filter(min_coverage=0.2) strips rare symbols before clustering/inference
- DEFAULT_COVERAGE=0.8 passed to infer_ensemble for sequence-level core/outlier detection
- Both filters active: symbol-level (0.2) then BEX sequence-level (0.8)
- Fix seq_of_file tracking broken by filter creating new list objects
- _extract_call_tokens filters to call-like captures only
- cluster_methods groups sequences by shared 3-gram call patterns
- analyze_clusters runs per-cluster CRX/iDRegEx/kORE inference
- Lists 17 convention clusters for RAGSAK test code
- iDRegEx and kOREInference now produce ordered grammars per cluster
- (other) cluster captures diverse conventions as CRX vocabulary
- min-cluster-size (default 3) and ngram-size (default 3) CLI flags
- _find_method_bodies uses tree-sitter's universal body field (9/10 grammars)
- Kotlin fallback: scan children for body-like types
- preprocess_by_method groups highlight captures by enclosing method body
- Returns per-method sequences for k-ORE ordering analysis
- analyze.py infer() now uses preprocess_by_method
- Document findings in ANALYSIS.md
- 97 tests pass
