diff --git a/experiments/EXPERIMENT_LOG.md b/experiments/EXPERIMENT_LOG.md index 79bb9df..9287789 100644 --- a/experiments/EXPERIMENT_LOG.md +++ b/experiments/EXPERIMENT_LOG.md @@ -515,3 +515,40 @@ The structured grammars (43% of all grammars) capture real calling conventions. - `(@+FastAPI+TestClient+app+client+data+...)+` (0.10) — FastAPI test bag-of-words **Decision:** `min_structure=0.2` is the right default. Flat bags are noise — they tell you what symbols exist but not how they're used. The 59 structured grammars capture real calling conventions with ordering information. + +--- + +## Round 13: Recursive Split-by-First-Symbol (commit `ca8a13b`) + +**Hypothesis:** Single split by first symbol misses patterns in sub-groups. +Recursive splitting (max_depth=3) produces more uniform leaf groups, +enabling CRX to infer tighter grammars. + +**Method:** +- Added `_recursive_split()` that drills deeper into each first-symbol sub-group +- Each leaf group gets its own grammar; the best leaf is returned per parent group +- Tested on FastAPI (1129 .py), RAGSAK (462 .kt), Flask (83 .py) + +**Results:** + +| Codebase | No split | Recursive split | Δ grammars | Δ high (≥0.5) | Δ avg score | +|----------|----------|-----------------|------------|----------------|-------------| +| FastAPI | 22 | 32 | +45% | +113% (8→17) | 0.47→0.58 | +| RAGSAK | 5 | 8 | +60% | +200% (2→6) | 0.45→0.71 | +| Flask | 0 | 2 | — | — (was zero) | 0.00→0.68 | + +**Key examples:** +- `return.commons?.q?.skip?.limit?` (1.00) — FastAPI dependency testing +- `value+.map+?.let+?.toDomain+?.storageUri?.imageType?.pageNo?` (1.00) — RAGSAK search adapter +- `state.app.code?.f.name+?` (1.00) — Flask sansio state management + +**Mechanism:** The improvement comes from capturing grammars in groups that previously +couldn't produce one at all — sub-groups of 3-5 methods that are too small for +single-split but contain clear patterns (e.g., all `return` or all `if` sequences). + +**SOA distance after split:** Cross-package distances dropped from 2.0 (completely +disjoint) to 1.5-1.7, but still above Reduce threshold (0.15). Reduce step not +useful here — recursive split already does the separation work. + +**Decision:** Recursive split is the right default. `_recursive_split()` replaces +single-level `_split_by_first_symbol()` when `split_mixed=True`.