diff --git a/experiments/EXPERIMENT_LOG.md b/experiments/EXPERIMENT_LOG.md index 82df3ad..8ecc4f8 100644 --- a/experiments/EXPERIMENT_LOG.md +++ b/experiments/EXPERIMENT_LOG.md @@ -722,3 +722,51 @@ is sometimes slower, and returns None more often. iDRegEx supersedes kORE. **Decision:** `--idregex-refine` flag enables this. Default: off. When enabled, ~1 candidate per RAGSAK run gets refined. Cost is negligible. + +--- + +## Round 17: CRX vs Refined CRX — When Does Clustering Help? + +**Hypothesis:** Refined CRX (cluster-then-infer) produces tighter grammars than +standard CRX by grouping sequences by first symbol before inference. But it might +be too tight on already-structured groups, or produce trivial single-symbol grammars. + +**Method:** Compared CRX vs refined CRX on all packages across 3 codebases: +RAGSAK (Kotlin), FastAPI (Python), Flask (Python). Metrics: structure score, +model_cost, and whether refined output is trivial (model_cost < 2). + +**Results:** + +| Codebase | CRX wins | Refined wins | Tie | Trivial (refined) | +|----------|----------|--------------|-----|-------------------| +| RAGSAK | 0 | 4 | 0 | 3 | +| FastAPI | 1 | 2 | 0 | 1 | +| Flask | 0 | 1 | 1 | 1 | +| **Total**| **1** | **7** | **1**| **5** | + +**When refined CRX wins (7 cases):** +- Low structure (CRX struct < 0.05), large groups (50-700 methods) +- Refined clusters by first symbol, finds tighter groupings +- Example: RAGSAK `agents` (519 methods): CRX struct=0.037 → refined struct=0.281 +- Example: FastAPI `fastapi` (375 methods): CRX model_cost=78 → refined model_cost=52 + +**When refined CRX is trivial (5 cases):** +- Large groups (368-973 methods) where all sequences share one common first symbol +- Refined clusters everything into one group → CRX on that group → single symbol +- Example: RAGSAK `app` (383 methods): refined to `return+` (model_cost=1) +- Example: Flask `tests` (993 methods): refined to single symbol (model_cost=1) + +**When CRX wins (1 case):** +- FastAPI `tests` (3618 methods): CRX struct=0.117, refined struct=0.043 +- Refined split too aggressively, lost the overall pattern + +**Key insight:** Refined CRX is better ~78% of the time when it produces something +useful (model_cost ≥ 2), but produces trivial output ~36% of the time on large +groups. The triviality check (model_cost ≥ 2) is essential. + +**Decision:** Refined CRX should be the default when `--split-mixed` is enabled. +The triviality check ensures we don't replace good CRX grammars with single symbols. +The pipeline should: (1) run refined CRX, (2) if trivial, fall back to CRX. + +**Recommendation:** Make refined CRX the default for `--split-mixed` mode. +Keep standard CRX as fallback. No need for iDRegEx or kORE in the pipeline.