# Round 19: Crucio Evaluation & Golden Config

**Goal:** Quantify whether Crucio-inspired changes (distributional clustering,
decomposition forest) actually improved grammar quality. Define the "golden config"
with best heuristic values.

**Open questions:**
1. Do the 21 RAGSAK / 22 FastAPI / 11 kotlinx patterns that survive `min_structure=0.5`
   actually match their input sequences?
2. Did decomposition improve pattern quality, or just create more patterns?
3. Did distributional clustering help when it differs from first-symbol?
4. What's the compression ratio (grammar size vs input size)?

---

## Experiment 1: Match Rate Audit

**What:** For each YAML entry, check what fraction of input sequences the grammar
actually accepts via `_matches()`.

**Method:**
1. Run pipeline with golden config on all 3 codebases
2. For each surviving pattern (min_structure=0.5), reproduce the input sequences
3. Check `_matches(grammar, seq)` for each input sequence
4. Report: match rate, grammar size, compression ratio

**Metrics:**
- `match_rate` = sequences accepted / total sequences (1.0 = perfect)
- `compression` = len(grammar) / total_input_chars (< 1.0 = good)
- `structure_score` = grammar_structure_score(grammar)

---

## Experiment 2: Decomposition A/B

**What:** Compare pipeline output WITH decomposition ON vs OFF.

**Method:**
1. Run with `--decompose --max-seq-length 4` (current config)
2. Run without `--decompose`
3. Compare: pattern count, avg match rate, avg structure score, avg compression

**Hypothesis:** Decomposition helps diverse codebases (RAGSAK, FastAPI) but
hurts already-structured ones (kotlinx.coroutines).

---

## Experiment 3: Distributional vs First-Symbol A/B

**What:** Compare `--cluster-method distributional` vs `--cluster-method first-symbol`.

**Method:**
1. Run with `--cluster-method distributional`
2. Run with `--cluster-method first-symbol` (current default)
3. Compare on the groups where they differ

**Hypothesis:** Distributional clustering doesn't help because the contexts
are already too specific per-package.

---

## Experiment 4: Decomposition Fragment Quality

**What:** Are decomposition fragments meaningful sub-patterns or noise?

**Method:**
1. Take decomposed fragments from a diverse package (e.g., RAGSAK agents/rag/embabel)
2. Check match rate of each fragment's grammar
3. Check if fragments capture real sub-patterns (e.g., "return path" vs "error path")

---

## Golden Config

The "golden config" captures our best-known heuristic values:

```python
GOLDEN_CONFIG = {
    # Core pipeline
    "min_coverage": 0.05,      # BEX outlier threshold (was 0.8, too aggressive)
    "min_methods": 3,           # Min methods per group (was 5, lost too many)
    "method": "langsize",       # Scoring: Language Size (Bex et al.)
    
    # Grouping
    "slice": "package",         # Per-directory (not flat, not reduce)
    "split_mixed": True,        # Recursive split by first symbol
    "max_depth": 3,             # Max recursion depth for split
    "cluster_method": "first-symbol",  # Split method (distributional = no improvement)
    
    # Quality filter
    "min_structure": 0.5,       # Drop flat bags (noise)
    "max_mdl": 200.0,           # Drop high-score grammars
    
    # Decomposition (Crucio Phase 2)
    "decompose": True,          # Break long sequences into fragments
    "max_seq_length": 4,        # Max fragment length (5 = too aggressive, 4 = sweet spot)
    
    # Algorithms
    "crx_method": "standard",   # Standard CRX (refined = trivial on large groups)
    "include_kore": False,      # kORE = slow, no improvement
    "include_idregex": False,   # iDRegEx = slow, rare benefit
    "idregex_refine": False,    # iDRegEx refinement = rare benefit
}
```

**Rationale for each value:**
- `min_coverage=0.05`: At 0.8, almost all symbols filtered out. 0.05 sees real vocabulary.
- `min_methods=3`: At 5, lost 9 FastAPI grammars. 3 is safe minimum.
- `min_structure=0.5`: Below this, patterns are flat bags (noise).
- `decompose=True`: Helps RAGSAK 7×, FastAPI 1.8×. Hurts kotlinx.coroutines.
- `max_seq_length=4`: At 5, fragments too short. 4 captures meaningful sub-patterns.
- `cluster_method="first-symbol"`: Distributional clustering showed no improvement.
- `crx_method="standard"`: Refined CRX produces trivial output 36% of the time.

---

## Execution Order

1. Write plan + golden config → commit
2. Experiment 1: Match rate audit (all 3 codebases)
3. Experiment 2: Decomposition A/B
4. Experiment 3: Distributional vs first-symbol A/B
5. Experiment 4: Decomposition fragment quality
6. Update golden config if needed
7. Summarize results and implications