grammar-inference-engine/experiments/EXPERIMENT_LOG.md

# Experiment Log — Grammar Inference Pipeline

Track what we tried, what worked, what failed, and what's next. Each experiment
includes: hypothesis, method, result, verdict.

---

## Round 1: Context Strategies (commit `bbdfe93`)

**Hypothesis:** The calling context (prefix before the method body) determines
which methods share a convention. Better context grouping → better grammars.

**Method:** Tested 4 strategies on RAGSAK (Kotlin) and Flask (Python):
- **Baseline:** Group by package directory
- **Option A:** Group by last k components of file path (`file_path_k{k}`)
- **Option B:** Group by first k symbols of call sequence (`first_k_sym_{k}`)
- **Option C:** Two-dimensional: (path_k, first_k_symbols)

**Result:**
| Strategy | RAGSAK patterns | RAGSAK coverage | Flask patterns | Flask coverage |
|----------|----------------|-----------------|----------------|---------------|
| Package baseline | 73 | 12.0% | 21 | 10.7% |
| File path k=1 | 73 | 12.0% | 21 | 10.7% |
| First k=1 | 20 | 4.6% | 2 | 1.4% |
| First k=3 | 47 | 12.0% | 21 | 10.7% |

**Verdict:** Package grouping and first_k_sym_3 produce similar results.
Cross-package grouping by first symbol is too sparse — most groups are either
too large (skipped) or too diverse (skipped). The useful patterns are
package-specific, not cross-package.

---

## Round 2: Reduce Algorithm (commit `b516b29`)

**Hypothesis:** Reduce (Algorithm 4, TODS 2010) merges structurally similar
contexts, revealing cross-package patterns by unifying equivalent states.

**Method:** Implemented faithful Reduce with support-weighted SOA edit distance,
adjunction, iterative merging, and minimize. Tested at ε=0.05 to 0.4.

**Result:**
- RAGSAK at ε=0.3: 1 merge (`JobStatus.every.getJobStatus` ↔ `JobStatus.now.minusMinutes`)
- Flask at ε=0.3: 1 merge (`def.boolean` ↔ `def.is_boolean`)
- Coverage improvement: negligible (< 1%)

**Verdict:** Reduce doesn't help. The contexts we produce are already too
specific (unique per package) for the distance metric to find meaningful merges.
Reduce works when you have a large SOA with many equivalent states — we have
one SOA per package with few states. Wrong abstraction level.

**Why it failed:** Reduce merges states in a single automaton. We're producing
one automaton per package group. There's nothing to merge across packages
because each package gets its own inference run. Reduce would need to operate
on a cross-package SOA, which we don't build.

---

## Round 3: Language Size Scoring (commit `dfb56a0`)

**Hypothesis:** Bex et al.'s Language Size measure (arXiv:1004.2372, Section
4.3.1) is a better scoring function than MDL for our use case.

**Method:** Implemented `lang_size_score()` as default scoring method. Added
diversity threshold: skip groups with unique_ratio > 0.9 or methods < 5.

**Result:** 39 new tests. Pipeline runs correctly with new scoring. Coverage
numbers similar to before — scoring method doesn't change which patterns are
found, just which grammar is selected per group.

**Verdict:** Scoring method is not the bottleneck. The problem is upstream
(pattern extraction), not downstream (pattern selection).

---

## Round 4: GBNF Output (commit `011df39`)

**Hypothesis:** SORE → GBNF conversion enables constrained LLM generation.
SORE operators map directly to GBNF syntax.

**Method:** Implemented recursive descent parser for SORE, AST intermediate
representation, and GBNF renderer. 15 tests.

**Result:** All tests pass. `to_gbnf('raise.(ValueError)+')` →
`"raise" "ValueError"+`. Correct mapping of +, ?, *, |, ., parens.

**Verdict:** Implementation works. But the input SOREs are too specific
to individual packages to be useful for constrained generation. The converter
is correct; the patterns it converts are the problem.

---

## Round 5: Cross-Package Exact Matches

**Hypothesis:** Some call sequences appear verbatim in multiple packages.
These are the real cross-package conventions.

**Method:** Grouped all sequences by exact tuple match across packages.

**Result:** 38 exact cross-package sequences in RAGSAK. Most are trivial:
- `('clearAllMocks',)` — 4 packages (test teardown)
- `('Builder',)` — 4 packages (builder pattern)
- `('Any',)` — 4 packages (Kotlin type)
- `('get',)` — 4 packages (getter)

Interesting ones:
- `('assumeTrue', 'isDockerAvailable', 'start', 'pullAndWarmup')` — 4 packages (Docker test setup)
- `('isNullOrBlank', 'error', 'error')` — 3 packages (null check → error)
- `('sortedBy', 'map', 'toDescriptor')` — 3 packages (data pipeline)
- `('ObjectMapper', 'findAndRegisterModules')` — 2 packages (Jackson config)

**Verdict:** Exact matches are too rare and mostly trivial. The real
cross-package patterns are structural, not textual — "null check → error"
appears with different method names in different packages.

---

## Round 6: Structural Coarsening (Experiment `coarsen_eval.py`)

**Hypothesis:** Coarsening structural tokens (keywords, types) while keeping
function names raw reveals cross-package patterns that pure-text misses.

**Method:** `coarsen_token()` maps tree-sitter capture names to categories.
Function calls kept as raw text (they ARE the content). Only structural tokens
coarsened: RETURN, RAISE, IF, LOOP, EXCEPTION, KW, TYPE, FUN, ATTR.

**Result:**

| Codebase | Structural % | Raw coverage | Coarsened coverage | Δ |
|----------|-------------|--------------|-------------------|---|
| RAGSAK | 4.7% | 18.6% | 18.3% | -0.3% |
| Flask | 36.1% | 17.5% | **28.8%** | **+11.3%** |

Key findings:
- RAGSAK: 95.3% function calls → coarsening has nothing to work with
- Flask: 36.1% structural → coarsening significantly improves grouping
- Flask k=3: cross-package contexts increase 40 → 47
- Coarsened cross-package shapes: `('IF', 'KW', 'RETURN')` in 4 packages,
  `('RETURN', 'render_template', 'render_template')` in 6 packages

**Verdict:** Coarsening helps codebases with rich structural tokens (Python:
IF, LOOP, EXCEPTION, KW). Doesn't help codebases dominated by function calls
(Kotlin: 95% calls). The approach is sound but language-dependent in practice —
depends on how rich the highlights.scm is.

**What we learned:**
- The 5% structural tokens DO carry signal when they exist
- Python highlights.scm is much richer than Kotlin's
- Coarsening is not dead — it's a tool for languages with rich captures
- The real question is whether the coarsened patterns are USEFUL, not just
  whether they exist

---

## Round 6b: Kotlin Capture Fix + Minimal Coarsening

**Hypothesis:** Kotlin's highlights.scm uses bare captures (`conditional`,
`exception`, `repeat`) while `BEHAVIORAL_PREFIXES` expected dotted forms
(`keyword.conditional`, etc.). All Kotlin structural context was being dropped.

**Method:** Added bare captures to BEHAVIORAL_PREFIXES. Dropped ATTR, TYPE,
VAR from coarsening (too noisy). Only coarsen RETURN, IF, EXCEPTION, LOOP.

**Result:**

| Codebase | Raw coverage | Coarsened coverage | Cross-pkg contexts |
|----------|-------------|-------------------|-------------------|
| RAGSAK k=3 | 18.5% | 6.0% | 87 (was 69) |
| Flask k=2 | 4.5% | 15.8% | 35 (was 54) |
| Flask k=3 | 17.1% | 17.3% | 28 (was 40) |

Key cross-package patterns (coarsened):
- `('IF', 'isEmpty', 'isEmpty')` — 8 RAGSAK packages (null-check convention)
- `('IF', 'isNullOrBlank', 'isNullOrBlank')` — 6 RAGSAK packages
- `('RETURN', 'render_template', 'render_template')` — 5 Flask packages
- `('IF', 'RETURN')` — 8 RAGSAK packages (guard clause pattern)

**Verdict:** The 4 high-signal categories (RETURN, IF, EXCEPTION, LOOP) DO
reveal cross-package conventions. Coarsening trades per-package coverage for
cross-package reach. Whether this is useful depends on the use case:
- For code completion: raw is better (specific method names)
- For documentation: coarsened is better (structural conventions)

---

## Round 7: Four-Codebase Evaluation

**Method:** Run raw vs coarsened on RAGSAK (Kotlin), Flask (Python),
kotlinx.coroutines (Kotlin), FastAPI (Python).

**Results (k=3):**

| Codebase | Raw cov | Coarse cov | Raw xpkg | Coarse xpkg | Δ |
|----------|---------|------------|----------|-------------|---|
| RAGSAK | 18.5% | 6.0% | 69 | 87 | +18 |
| Flask | 17.1% | 17.3% | 40 | 27 | -13 |
| Coroutines | 30.7% | 19.1% | 524 | 539 | +15 |
| FastAPI | 12.4% | 20.7% | 82 | 97 | +15 |

Cross-package patterns discovered:
- FastAPI: `('response', 'client', 'get')` — 72 packages (HTTP request pattern)
- Coroutines: `('RETURN', 'EXCEPTION', 'UnsupportedOperationException')` — 13 packages
- RAGSAK: `('IF', 'isEmpty', 'isEmpty')` — 8 packages (null-check)
- Flask: `('RETURN', 'render_template', 'render_template')` — 5 packages

**Verdict:** The conventions vs completions tradeoff is real and measurable.
Coarsening consistently trades per-package coverage for cross-package reach.
FastAPI is the exception: coverage improves (12.4% → 20.7%) because its
structural tokens (response/client patterns) are highly repetitive.

---

## Round 8: Frequency Threshold Sweep

**Hypothesis:** The fixed `min_coverage=0.2` is too aggressive. Lower thresholds
reveal more patterns while still filtering noise.

**Method:** Test thresholds 0.00–0.20 on all 4 codebases. Measure symbol count,
surviving sequences, SORE success, coverage.

**Results:**

| Codebase | Thresh | Syms | Seqs | SOREs | Coverage |
|----------|--------|------|------|-------|----------|
| RAGSAK | 0.01 | 130 | 1270 | 61 | 26.1% |
| RAGSAK | 0.05 | 16 | 919 | 40 | 45.4% |
| RAGSAK | 0.10 | 5 | 657 | 19 | 69.1% |
| Flask | 0.01 | 47 | 784 | 28 | 35.2% |
| Flask | 0.05 | 5 | 414 | 7 | 44.4% |
| Flask | 0.10 | 2 | 237 | 5 | 100.0% |
| Coroutines | 0.01 | 76 | 4802 | 175 | 40.4% |
| FastAPI | 0.01 | 23 | 2678 | 14 | 17.0% |

Key findings:
- Coverage increases with threshold (trivial: 1 symbol = 100% coverage)
- Sweet spot: 0.01–0.05. Enough symbols for meaningful patterns, enough
  filtering to remove noise.
- At 0.01: RAGSAK gets `warn.status.body.ErrorResponse` (real convention)
- At 0.05: that pattern disappears (too aggressive)
- Flask dies at 0.15+ (0 symbols survive)

---

## What We Learned (Summary)

1. **Per-package grouping is too sparse.** 1-3 sequences per package isn't
   enough for any inference method to produce general patterns.

2. **Cross-package exact matches are rare.** Only 38 in RAGSAK, mostly trivial
   single-call sequences.

3. **Reduce doesn't help at our abstraction level.** It merges states within
   one automaton; we need to merge patterns across packages.

4. **Scoring/selection isn't the bottleneck.** MDL vs Language Size doesn't
   change what patterns are found.

5. **The calling context prefix is the right signal** but grouping by it
   produces groups that are either too large, too diverse, or trivial.

6. **GBNF converter works correctly** but the input patterns are too specific.

---

## Next: Structural Coarsening + Cross-Package Detection

### Idea

Collapse method names → categories using tree-sitter capture names. This
converts textual sequences into structural shapes:

```
('isNullOrBlank', 'error', 'error')     →  (CALL, ERROR, ERROR)
('raise', 'ValueError', 'ValueError')   →  (CALL, ERROR, ERROR)
```

Same structural shape, different packages → cross-package convention.

### Why This Might Work

- We already extract tree-sitter capture names in `code.py:56-69`
- We already classify nodes into categories in `code.py:72-100`
  (`lit`, `call`, `var`, `lambda`, `kwarg`, `expr`, `template`, `other`)
- The behavioral prefix filter (`CALL_PREFIXES`) keeps raw text; we need a
  parallel path that keeps the category instead
- Coarsened sequences have smaller alphabets → more methods per group →
  better inference
- Patterns like `(CALL, ERROR, ERROR)` are meaningful conventions that
  repeat across packages

### What We Need

1. **Coarsening map:** `capture_name → category` using the existing
   `CALL_PREFIXES`, `ARG_LITERAL_TYPES`, `LAMBDA_TYPES` classifications
   plus a new `ERROR_TYPES` set

2. **Coarsened sequence extraction:** Same pipeline as now, but output
   category labels instead of method names

3. **Cross-package grouping:** Group by coarsened shape (first k categories),
   find shapes that appear in ≥2 packages

4. **SORE inference on coarsened sequences:** Smaller alphabet, more examples
   per group → better patterns

5. **Evaluation:** Compare coarsened patterns vs raw patterns on:
   - Coverage (% of methods in learned groups)
   - Cross-package reach (# of packages per pattern)
   - Usefulness for constrained generation (GBNF quality)

### Open Questions

- Does coarsening lose too much specificity? `(CALL, ERROR, ERROR)` is less
  informative than `(raise, ValueError, ValueError)` — is the tradeoff worth it?
- What categories to use? The existing classifications in `code.py` are a
  starting point but may need refinement (e.g., separating ERROR from CALL)
- How to handle the long tail? Most sequences are 1-2 symbols — coarsening
  doesn't help much for those
- Is the GBNF output useful at all? Maybe the output should be a conditional
  frequency table instead of a grammar

### Experiment Design

**Phase 1: Coarsening Proof of Concept**
- Implement coarsening map in `code.py`
- Add `--coarsen` flag to CLI
- Run on RAGSAK + Flask, compare raw vs coarsened patterns
- Measure: alphabet size reduction, group size increase, pattern count

**Phase 2: Cross-Package Detection**
- Group coarsened sequences by shape (first k categories)
- Find shapes appearing in ≥2 packages
- For each shape, infer SORE on coarsened sequences
- Measure: cross-package patterns found, coverage improvement

**Phase 3: Output Quality**
- Convert coarsened SOREs to GBNF
- Evaluate: are the GBNF rules more general/useful than raw SOREs?
- Compare: coarsened GBNF vs raw GBNF vs conditional frequency table

---

## The "Redacted" Concept

When collapsing method names to categories, we lose the specific method name
but gain the structural pattern. This is a form of **abstraction** — moving
from concrete examples to general rules.

The question is whether the abstraction is at the right level:
- Too specific: `(raise, ValueError, ValueError)` — package-specific noise
- Right level: `(CALL, ERROR, ERROR)` — cross-package convention
- Too abstract: `(X, Y, Y)` — trivial, tells the LLM nothing

The middle ground depends on the category vocabulary. We need enough categories
to be informative (CALL, ERROR, LIT, VAR, TYPE, etc.) but not so many that
every sequence is unique.

---

## Round 9: CRX Over-Approximation Analysis

**Hypothesis:** Standard CRX produces over-approximated grammars when the
Hasse diagram is non-linear (branching). This results in flat disjunctions
like `(a+b+c+d)+?` that accept almost any combination — useless as conventions.

**Method:**
1. Measured tightness across RAGSAK packages: fraction of symbol pairs in data
   vs total possible pairs. Average: 0.300 (only 30% of pairs actually occur).
2. Measured over-approximation rate: 24% of packages have `+?` factors with
   4+ symbols — massive over-approximation.
3. Analyzed CRX algorithm (Algorithm 3, Bex et al. VLDB 2006):
   - CRX computes equivalence classes ≈_S (mutual reachability)
   - Merges singletons with identical (Pred, Succ) in Hasse diagram
   - Key limitation: **only merges singletons**, not multi-symbol classes
   - Theorem 5: CRX is optimal **only when** Γ_W is linearly ordered
   - Non-linear → suboptimal (paper counterexample: `{abc, ade, abe}` →
     `a.b?.d?.c?.e?` instead of better `a.(b+d).(c+e)`)

**Findings:**
- CRX was designed for XML DTDs with hierarchical structure. Code call
  sequences have branching patterns that CHAREs can't represent.
- The `+?` (zero-or-more) factor is the over-approximation signal: it means
  "any subset of these symbols in any order" — which is trivially true.
- Standard CRX CAN'T fix this — the CHARE representation is inherently linear.
- Two possible improvements: (a) detect over-approximation, (b) cluster before
  inferring.

**Verdict:** CRX has fundamental limitations for code sequences. Proceed to
cluster-then-infer experiment.

---

## Round 10: Cluster-Then-Infer (CRX Improvement)

**Hypothesis:** Grouping similar sequences before CRX inference produces tighter
grammars, because each cluster's Hasse diagram is more likely to be linear.

**Method:**
1. Cluster sequences by (first_symbol, last_symbol) — a simple structural hash
2. Infer CRX per cluster
3. Pick the most common cluster's grammar
4. Compare: avg max disjunction size (standard CRX vs clustered)

**Result:**
| Metric | Standard CRX | Cluster-Then-Infer | Improvement |
|--------|-------------|-------------------|-------------|
| Avg max disjunction size | 2.8 | 1.7 | 39% tighter |
| Packages improved | — | 6/10 | 60% |

Example improvements:
- `service/job` (14 seqs): `(any+asJobId+assertEquals+assertTrue+build+exchange...)+?`
  → `get` (single symbol — much tighter)
- `agent/rag/embabel` (7 seqs): `(any+assertEquals+assertTrue+contains+emptyList+every+verify)+?`
  → `(any+every)+.emptyList+.assertEquals+` (structured)
- `batch/listener` (5 seqs): `(any+asJobId+assertEquals+assertTrue)+?.uri?.exchange?.expectStatus?`
  → `uri.build+.exchange.expectStatus.get` (structured)

**Decision:** Add `crx_refined` module with cluster-then-infer as the default
CRX method. Keep standard CRX available for comparison.

**Tradeoff parameter identified:** Cluster granularity.
- Too coarse (no clustering): over-approximation (current CRX)
- Too fine (1 seq per cluster): every sequence gets its own grammar, no generalization
- Sweet spot: cluster by structural features (first/last symbols, length, etc.)

**Next steps:**
- Test clustering on Flask, Coroutines, FastAPI
- Try better clustering features (k-mer, edit distance, prefix sharing)
- Evaluate: does tighter grammar → better code completion / convention docs?

---

## Round 11: Pipeline Speed + GBNF Conversion (commits `bc7d3b6`, `3468813`)

**Hypothesis:** iDRegEx in the ensemble is the bottleneck. GBNF conversion needs error handling.

**Method:**
- Made iDRegEx opt-in via `--idregex` flag (was running on every group)
- Added `validate_sore()` — skip malformed SOREs gracefully
- Fixed OverflowError: `lang_size_score` produces huge ints for large disjunctions
- Fixed GBNF tokenizer: strip newlines from literals

**Results — Pipeline Speed:**
| Codebase | Before (with iDRegEx) | After (CRX only) | Speedup |
|----------|----------------------|-------------------|---------|
| Flask    | 55s+                 | 2.8s              | 20×     |
| RAGSAK   | 74s                  | 13s               | 5.7×    |
| FastAPI  | hung at 300s         | 30s               | >10×    |

Root cause: `src/flask/json` (50 methods) alone took 55s in iDRegEx. Flask's `tests` group (962 methods) would have been worse.

**Results — GBNF Conversion:**
| Codebase | Grammars | GBNF OK | GBNF FAIL | Malformed (skipped) |
|----------|----------|---------|-----------|---------------------|
| Flask    | 5        | 5       | 0         | 0                   |
| RAGSAK   | 19       | 19      | 0         | 11                  |
| FastAPI  | 106      | 106     | 0         | 6                   |
| **Total**| **130**  | **130** | **0**     | **17**              |

17 malformed SOREs contain raw code (e.g. `w_body=`, `(+,+:N+...)`) — preprocessing bug, not parser issue.

**Grammar Quality Analysis:**
- **Structured** (has ordering via `.`, `?`): RAGSAK 18, FastAPI 94, Flask 3
- **Flat disjunction** (bag of symbols): RAGSAK 1, FastAPI 10, Flask 2
- **Trivial** (single symbol): RAGSAK 0, FastAPI 2, Flask 0

Best structured examples:
- `return.render_template+` — clear: return, then render_template one or more times
- `assertNull?.parseS3Location.error+?.(assertEquals+bucket)+?.key?` — test flow
- `buildObservationContext?.shouldRetrieve?.(ASK+ChatResponse+...)?...` — agent flow
- `if.(img+item_id).(FileResponse+else+media_type+return)+.JSONResponse+?.status_code?.content?` — if/else structure

**Decision:** iDRegEx stays opt-in. GBNF validation catches malformed SOREs early.
The structured grammars (43% of all grammars) capture real calling conventions.

**Open questions:**
- 1129-file FastAPI has 30 diverse groups — need better grouping for large codebases
- Malformed SOREs from raw code in symbol names — need upstream fix in code.py
- Flat disjunctions are noisy — should we filter by grammar complexity?

---

## Round 12: Grammar Structure Scoring (commit `e62fffc`)

**Hypothesis:** Flat disjunctions `(a+b+c+...+z)+` are CRX over-approximations — they list symbols without ordering and aren't useful for code completion or convention docs. We can filter them.

**Method:** `grammar_structure_score(sore)` — measures structural richness:
- Count dots (`.`), optional (`?`), repetition outside parens (`+`, `*`) = ordering ops
- Count disjunction parts inside parens = noise
- Score = `min(1.0, struct_ratio * 3)`, penalized if high disjunction ratio with no concatenation

**Thresholds:**
- **Structured** (≥0.5): Has ordering — tells you the SEQUENCE things happen
- **Semi** (0.2-0.5): Partial structure
- **Flat** (0.05-0.2): Bag of symbols — CRX over-approximation
- **Trivial** (<0.05): Single symbol or empty

**Results with `min_structure=0.2`:**
| Codebase | Before | After | Kept | Dropped |
|----------|--------|-------|------|---------|
| Flask    | 5      | 2     | 2    | 3 flat + 6 diverse |
| RAGSAK   | 19     | 10    | 10   | 9 flat + 114 diverse/malformed |
| FastAPI  | 106    | 47    | 47   | 59 flat + 36 diverse/malformed |
| **Total**| **130**| **59**| **59**| **218** |

**Example kept grammars (score ≥ 0.2):**
- `return.render_template+` (score=0.29) — Flask test convention
- `assertNull?.parseS3Location.error+?.(assertEquals+bucket)+?.key?` (0.59) — RAGSAK test flow
- `if?.return?.(token+x_token)?.raise?.HTTPException+?.status_code?.detail?` (0.76) — FastAPI auth pattern
- `return.request?.scope?.get+?` (1.00) — maximally structured

**Example dropped grammars (score < 0.2):**
- `(@+Blueprint+__name__+app+append+class+client+...)+` (0.02) — Flask test bag-of-words
- `(Any+BlueprintSetupState+ValueError+...)+` (0.01) — Flask sansio bag-of-words
- `(@+FastAPI+TestClient+app+client+data+...)+` (0.10) — FastAPI test bag-of-words

**Decision:** `min_structure=0.2` is the right default. Flat bags are noise — they tell you what symbols exist but not how they're used. The 59 structured grammars capture real calling conventions with ordering information.

---

## Round 13: Recursive Split-by-First-Symbol (commit `ca8a13b`)

**Hypothesis:** Single split by first symbol misses patterns in sub-groups.
Recursive splitting (max_depth=3) produces more uniform leaf groups,
enabling CRX to infer tighter grammars.

**Method:**
- Added `_recursive_split()` that drills deeper into each first-symbol sub-group
- Each leaf group gets its own grammar; the best leaf is returned per parent group
- Tested on FastAPI (1129 .py), RAGSAK (462 .kt), Flask (83 .py)

**Results:**

| Codebase | No split | Recursive split | Δ grammars | Δ high (≥0.5) | Δ avg score |
|----------|----------|-----------------|------------|----------------|-------------|
| FastAPI  | 22       | 32              | +45%       | +113% (8→17)  | 0.47→0.58   |
| RAGSAK   | 5        | 8               | +60%       | +200% (2→6)   | 0.45→0.71   |
| Flask    | 0        | 2               | —          | — (was zero)   | 0.00→0.68   |

**Key examples:**
- `return.commons?.q?.skip?.limit?` (1.00) — FastAPI dependency testing
- `value+.map+?.let+?.toDomain+?.storageUri?.imageType?.pageNo?` (1.00) — RAGSAK search adapter
- `state.app.code?.f.name+?` (1.00) — Flask sansio state management

**Mechanism:** The improvement comes from capturing grammars in groups that previously
couldn't produce one at all — sub-groups of 3-5 methods that are too small for
single-split but contain clear patterns (e.g., all `return` or all `if` sequences).

**SOA distance after split:** Cross-package distances dropped from 2.0 (completely
disjoint) to 1.5-1.7, but still above Reduce threshold (0.15). Reduce step not
useful here — recursive split already does the separation work.

**Decision:** Recursive split is the right default. `_recursive_split()` replaces
single-level `_split_by_first_symbol()` when `split_mixed=True`.

---

## Round 14: Byte/Char Offset Fix — Tree-sitter Text Extraction

**Hypothesis:** Tree-sitter returns byte offsets, but we indexed into Python strings
(character offsets). Non-ASCII characters create cumulative drift, truncating symbols.

**Root cause:** `parser.parse(code.encode())` feeds bytes → tree-sitter returns
byte offsets. `code[node.start_byte:node.end_byte]` indexes a **string** with
**byte** offsets. Every multi-byte char shifts the index further.

**Example (Kotlin):**
```python
# Before fix: byte offset 498 in string "override fun ddCheck(..." 
# "ddCheck" at chars 498–505, but byte 498 lands at 'd' only after accounting
# for emoji/non-ASCII in earlier strings. Result: truncated to "ddCheck" (OK here)
# but in practice: "ddCheck" → "ddCheck" ( lucky), "ddCheck" → "ddChe" ( unlucky)
```

**Evidence:**
- Zod (TypeScript): 653 truncated symbols → **0 after fix**
- RAGSAK (Kotlin): 653 truncated symbols → **0 after fix**

**Method:** Store `code_bytes = code.encode()`, index into that, decode only final
text. Applied to `preprocess_by_method()`, `preprocess()`, and `extract_arg_info()`.

**Results (all codebases, `--slice package --min-coverage 0.05 --min-methods 3 --split-mixed`):**

| Codebase | Language | Files | Before (kept) | After (kept) | Malformed | Too diverse |
|----------|----------|-------|---------------|--------------|-----------|-------------|
| RAGSAK   | Kotlin   | 462   | 8             | **27**       | 2         | 95          |
| Zod      | TypeScript| 372  | —             | **10**       | 1         | 1           |
| Flask    | Python   | 83    | 2             | **5**        | 0         | —           |
| FastAPI  | Python   | 1129  | 33            | **111**      | 0         | —           |

**RAGSAK (27 kept grammars):**
- Clean symbol names: `ToolInvocationRequest`, `ToolingRequest`, `ToolInvocationResult`
- Previous: `ToolingReq`, `ToolInvocationRe` (truncated)
- Score range: 3–3.97e+100 (wide spread, some very large from CRX over-approximation)

**Zod (10 kept grammars):**
- `safeParse.(expect+toEqual)+?.as+?.z?` (score 41)
- `return.i+?.new?.test?.Date+?.DATA?` (score 48)
- `(Object+typeof)+.map?.(Error+Promise+Set+any+as+def+for+if+key+new+of+return+then+throw+util)+?` (score 282393216)
- Truncation gone: previous `ddCheck`, `arse`, `rty` → now full names

**FastAPI (111 kept grammars):**
- Largest jump: 33 → 111 (3.4× more grammars)
- Many test files now produce grammars: `(test_create_existing_item+test_create_item+...)` patterns

**Malformed grammars:**
- RAGSAK: 2 (down from 6 in Round 12)
- Zod: 1 (`src/v4/locales` — 214 methods, too diverse)
- FastAPI/Flask: 0

**Chain of thought:**
1. Symbols were truncated → CRX saw incomplete names → merged wrong things or produced flat bags
2. With clean symbols, CRX can distinguish `ddCheck` from `ddChecker` → tighter grammars
3. More methods survive the `min_coverage` filter → more groups produce grammars
4. The 3.4× jump in FastAPI confirms: truncation was the primary bottleneck, not the algorithm

**Verdict:** This was the single highest-impact fix in the pipeline. Tree-sitter
byte/char mismatch was silently corrupting every symbol extraction. The
`sanitize_symbol()` function (removed in this round) was a band-aid that hid
the real problem. Clean symbols → clean grammars.

**Decision:** Keep `code_bytes = code.encode()` pattern permanently. Remove
`sanitize_symbol()` (dead code). Remove `call_only` parameter (no longer needed).

---

## Round 15: kORE/iDRegEx vs CRX on Flat Bags

**Question:** Would kORE or iDRegEx produce tighter grammars for the groups where
CRX over-approximates (flat bags with structure < 0.3)?

**Method:** Hand-crafted sequences mimicking real RAGSAK patterns. Tested CRX,
iDRegEx (k=2, N=3), and kORE (k=2, N=3) on two example types.

**Example 1: Structured sequences (clear branching)**
```
Input: 6 methods with session.use.{run/execute}.{parameters/query}.{single/list}.{get/map/filter}
CRX:     session.use.execute?.run?.query?.parameters?.list?.single?.filter?.map?.get?.count?.(toLong+toString)?.(asLong+asString)?
         → Flat optional chain. All symbols listed, no real structure.
         
iDRegEx: session.use.(run.parameters|execute.query).(single.get.(asLong|asString)|list.(filter.count|map.(toLong|toString)))
         → Nested disjunctions. Shows actual branching: run vs execute, single vs list.
         
kORE:    Same as iDRegEx.
```
**Verdict:** iDRegEx/kORE produce MORE informative grammars. Nested disjunctions
show the actual code paths. CRX flattens everything into optional chains.

**Example 2: Flat bag (diverse patterns)**
```
Input: 6 methods with different call patterns (request/response/error paths)
CRX:     return.error?.(request+response)?.message?.json?.data?.status?.ok?
         → Partial structure, some optional paths.
         
iDRegEx: None
kORE:    None
```
**Verdict:** When there's genuinely no structure, iDRegEx/kORE return None. CRX
is the only one that produces anything.

**Real-world RAGSAK test (6 methods, health check package):**
```
CRX:     (HealthCheckReply+`when`+collectionExistsAsync+...)+ → flat bag, score 0.126
iDRegEx: None (at k=2,3)
kORE:    None
```
Both iDRegEx and kORE return None on real flat bags because the sequences are
too diverse.

**Key insight:** CRX and iDRegEx/kORE operate on different principles:
- **CRX**: Deterministic, always produces something, but over-approximates on diverse groups
- **iDRegEx/kORE**: Probabilistic, need repeating patterns to infer, return None when patterns are too diverse

**Recommendation:** The flat bags (structure < 0.2) are genuinely diverse groups —
no algorithm can find meaningful structure. The fix is:
1. **Split further** (recursive split already does this)
2. **Filter by structure** (min_structure ≥ 0.3 drops flat bags)
3. **Accept that some groups are noise** and skip them

Using kORE/iDRegEx as fallback for low-structure groups would just return None
more often. CRX is the right default — it's fast and always produces something.
For the structured groups, CRX already captures the ordering well (score ≥ 0.5).

**Decision:** Keep CRX as default. kORE/iDRegEx are not better for flat bags
(they return None) and not needed for structured groups (CRX already works).
The pipeline's existing filtering (min_structure, split_mixed) is the right
approach to handle diversity.

---

## Round 16: iDRegEx Refinement for CRX Flat Bags (commit pending)

**Hypothesis:** CRX over-approximates on small groups with many optional parts
(flat chains like `a?.b?.c?.d?.e?`). iDRegEx produces tighter nested
disjunctions on these groups. We can detect the flat bags with a heuristic
and refine them with iDRegEx, getting >10x tighter grammars at minimal cost.

**Method:**
1. After CRX produces a grammar, count top-level optional parts
2. If `n_methods ≤ 10` AND `optionals/total_parts > 0.5` → CRX produced a flat bag
3. Run iDRegEx on the same sequences
4. Compare by `lang_size_score` — if >10x improvement, use iDRegEx

**Heuristic (`_count_optionals`):** Splits grammar on top-level dots, counts
parts ending with `?`. `a?.b?.c?.d?` → 4/4 optionals. `a.(b|c).(d|e)` → 0/3.

**Key insight:** `lang_size_score` (Bex et al.) is the right metric for comparing
grammars — it counts how many words the grammar accepts at each input length.
- CRX flat chains accept exponentially many words (e.g., 9432)
- iDRegEx nested disjunctions accept only the actual sequences (e.g., 60)
- `lang_size_score` naturally prefers iDRegEx when it produces something

**RAGSAK results:**
| Package | Methods | CRX optionals | iDRegEx result | lang_size improvement |
|---------|---------|---------------|----------------|----------------------|
| agents/capability | 5 | 75% | `(defaultCapabilityId\|summarize)` | 477x tighter |

**Speed cost:** 1 candidate × ~700ms = negligible (0.7s on 74s pipeline).

**Why kORE is dropped:** kORE produces the same or worse output as iDRegEx,
is sometimes slower, and returns None more often. iDRegEx supersedes kORE.

**Decision:** `--idregex-refine` flag enables this. Default: off.
When enabled, ~1 candidate per RAGSAK run gets refined. Cost is negligible.

---

## Round 17: CRX vs Refined CRX — When Does Clustering Help?

**Hypothesis:** Refined CRX (cluster-then-infer) produces tighter grammars than
standard CRX by grouping sequences by first symbol before inference. But it might
be too tight on already-structured groups, or produce trivial single-symbol grammars.

**Method:** Compared CRX vs refined CRX on all packages across 3 codebases:
RAGSAK (Kotlin), FastAPI (Python), Flask (Python). Metrics: structure score,
model_cost, and whether refined output is trivial (model_cost < 2).

**Results:**

| Codebase | CRX wins | Refined wins | Tie | Trivial (refined) |
|----------|----------|--------------|-----|-------------------|
| RAGSAK   | 0        | 4            | 0   | 3                 |
| FastAPI  | 1        | 2            | 0   | 1                 |
| Flask    | 0        | 1            | 1   | 1                 |
| **Total**| **1**    | **7**        | **1**| **5**            |

**When refined CRX wins (7 cases):**
- Low structure (CRX struct < 0.05), large groups (50-700 methods)
- Refined clusters by first symbol, finds tighter groupings
- Example: RAGSAK `agents` (519 methods): CRX struct=0.037 → refined struct=0.281
- Example: FastAPI `fastapi` (375 methods): CRX model_cost=78 → refined model_cost=52

**When refined CRX is trivial (5 cases):**
- Large groups (368-973 methods) where all sequences share one common first symbol
- Refined clusters everything into one group → CRX on that group → single symbol
- Example: RAGSAK `app` (383 methods): refined to `return+` (model_cost=1)
- Example: Flask `tests` (993 methods): refined to single symbol (model_cost=1)

**When CRX wins (1 case):**
- FastAPI `tests` (3618 methods): CRX struct=0.117, refined struct=0.043
- Refined split too aggressively, lost the overall pattern

**Key insight:** Refined CRX is better ~78% of the time when it produces something
useful (model_cost ≥ 2), but produces trivial output ~36% of the time on large
groups. The triviality check (model_cost ≥ 2) is essential.

**Decision:** Refined CRX should be the default when `--split-mixed` is enabled.
The triviality check ensures we don't replace good CRX grammars with single symbols.
The pipeline should: (1) run refined CRX, (2) if trivial, fall back to CRX.

**Recommendation:** Make refined CRX the default for `--split-mixed` mode.
Keep standard CRX as fallback. No need for iDRegEx or kORE in the pipeline.

---

## Round 18: Decomposition Forest (Phase 2)

**Goal:** Break down long sequences into shorter fragments before inference (Crucio Phase 2).

**Method:** Implemented `bex/decompose.py` with prefix/suffix/window extraction.

**Results:**
- RAGSAK: 21 → 80 grammars (3.8× increase)
- FastAPI: 111 → 118 grammars (small increase)

**Key insight:** Decomposition creates diverse fragments, so skip diversity check when enabled.

**Files:**
- `bex/decompose.py`: decompose_sequence(), decompose_all(), decompose_with_coverage()
- `bex/tag_preprocessor/analyze.py`: --decompose, --max-seq-length flags
- `tests/test_decompose.py`: 12 new tests

---

## Round 19: AST Migration

**Goal:** Replace SORE string representations with proper AST nodes throughout the pipeline.

**Method:** Migrated CRX, iDRegEx, ensemble, mdl, and SORE parser from string-based grammars
to `bex.grammar` AST nodes (Symbol, Concat, Alt, Optional, Plus, Star, Empty). Purged all
SORE string operations from the inference pipeline. Added `_count_concat` memoization.

**Commits:** `ea6cac5` → `52f286a` (6 commits)

**Results:**
- RAGSAK: CRX inference dropped from 54.9s → 6.7s after restoring memoization on `_count_concat`
- Fixed `_count_concat` losing its `@lru_cache` decorator during AST migration — was the real performance bug, not ProcessPoolExecutor
- All 313 tests pass

**Key insight:** The memoization loss was invisible because `_count_concat` is called recursively
on every grammar node. Without caching, identical subtrees were re-evaluated exponentially.

**Files changed:**
- `bex/crx.py`: CRX algorithm returns AST nodes
- `bex/ensemble.py`: ensemble matching uses AST comparison
- `bex/mdl.py`: scoring functions operate on AST
- `bex/grammar.py`: AST node definitions, `_count_concat` with `@lru_cache`

---

## Round 20: AST Pipeline Verification + Scoring Fixes

**Goal:** Verify the AST pipeline end-to-end across 3 codebases, fix scoring issues.

**Codebases:** RAGSAK (Kotlin, 462 files, 1609 methods), FastAPI (Python, 143 groups), Zod (TypeScript, 23 groups)

### Phase A: `_COUNT_CAP` fix

**Problem:** `_COUNT_CAP = 10^12` clamped all `lang_size_score` values to the same ceiling,
making bags and tight grammars indistinguishable (all scored 10^12).

**Fix:** Raised `_COUNT_CAP` from `10^12` to `10^30`. With memoization preventing the
recursion hang, the cap no longer needs to be low.

**Result:** `lang_size_score` now discriminates: tight grammar = 20, pure bag = 9975.
MDL score abandoned (ADR-13) — language size scoring chosen because MDL rewards short
expressions over specific patterns.

### Phase B: `decompose=True` default

**Change:** Made decomposition ON by default (CLI + `analyze_directory`). Reduced
`max_seq_length` from 5→4.

**Results:**
| Codebase | Before | After | Change |
|----------|--------|-------|--------|
| RAGSAK | 29 grammars | 95 grammars | 3.3× more patterns |
| RAGSAK pure bags | 9 | 5 | Fewer orderless bags |
| FastAPI | 109 grammars | 118 grammars | Small increase |
| Zod | 16 grammars | 10 grammars | More selective |

### Phase C: `idregex_refine=True` default

**Change:** Enabled iDRegEx refinement by default. Rewrote `_count_optionals` from SORE
string parser to AST walker. Added `_is_pure_bag()` helper.

**Results:**
- RAGSAK v4: 126 grammars total, 6 pure bags, 120 structured
- FastAPI v3: 143 grammars, 26 pure bags, 117 structured
- Zod v3: 23 grammars, 5 pure bags, 18 structured

**Key finding:** iDRegEx doesn't help small bags. On 3-method groups, iDRegEx achieves
only 3.8× tighter (below the 10× threshold gate). The gate correctly rejects it.
Earlier test on `storage` (4 methods) showed 91× — that was an outlier, not the norm.

**Decision:** idregex_refine stays ON but the gate effectively limits it to groups where
iDRegEx produces a genuinely tighter grammar. No further algorithmic changes planned for
orderless bags — they survive because CRX emits one grammar deterministically and
`lang_size_score` only ranks between algorithms, not within CRX's own output.

**Quality reality:** ~85% of grammars remain orderless bags `(A|B|C)+`. The ~15% that are
structured represent real sequential flows (e.g., `post→jsonPath→isEqualTo→exchange→expectStatus`).
Bags are concentrated in large groups (tests, v4/locales) where method diversity is too high
for any algorithm to find ordering.

**Files changed:**
- `bex/tag_preprocessor/analyze.py`: idregex_refine default, _count_optionals AST rewrite, _is_pure_bag helper
- `bex/grammar.py`: _COUNT_CAP raised to 10^30
- `tests/test_analyze.py`: updated tests for AST-based _count_optionals