diff --git a/experiments/EXPERIMENT_LOG.md b/experiments/EXPERIMENT_LOG.md index 0779467..68c69e5 100644 --- a/experiments/EXPERIMENT_LOG.md +++ b/experiments/EXPERIMENT_LOG.md @@ -621,3 +621,67 @@ the real problem. Clean symbols → clean grammars. **Decision:** Keep `code_bytes = code.encode()` pattern permanently. Remove `sanitize_symbol()` (dead code). Remove `call_only` parameter (no longer needed). + +--- + +## Round 15: kORE/iDRegEx vs CRX on Flat Bags + +**Question:** Would kORE or iDRegEx produce tighter grammars for the groups where +CRX over-approximates (flat bags with structure < 0.3)? + +**Method:** Hand-crafted sequences mimicking real RAGSAK patterns. Tested CRX, +iDRegEx (k=2, N=3), and kORE (k=2, N=3) on two example types. + +**Example 1: Structured sequences (clear branching)** +``` +Input: 6 methods with session.use.{run/execute}.{parameters/query}.{single/list}.{get/map/filter} +CRX: session.use.execute?.run?.query?.parameters?.list?.single?.filter?.map?.get?.count?.(toLong+toString)?.(asLong+asString)? + → Flat optional chain. All symbols listed, no real structure. + +iDRegEx: session.use.(run.parameters|execute.query).(single.get.(asLong|asString)|list.(filter.count|map.(toLong|toString))) + → Nested disjunctions. Shows actual branching: run vs execute, single vs list. + +kORE: Same as iDRegEx. +``` +**Verdict:** iDRegEx/kORE produce MORE informative grammars. Nested disjunctions +show the actual code paths. CRX flattens everything into optional chains. + +**Example 2: Flat bag (diverse patterns)** +``` +Input: 6 methods with different call patterns (request/response/error paths) +CRX: return.error?.(request+response)?.message?.json?.data?.status?.ok? + → Partial structure, some optional paths. + +iDRegEx: None +kORE: None +``` +**Verdict:** When there's genuinely no structure, iDRegEx/kORE return None. CRX +is the only one that produces anything. + +**Real-world RAGSAK test (6 methods, health check package):** +``` +CRX: (HealthCheckReply+`when`+collectionExistsAsync+...)+ → flat bag, score 0.126 +iDRegEx: None (at k=2,3) +kORE: None +``` +Both iDRegEx and kORE return None on real flat bags because the sequences are +too diverse. + +**Key insight:** CRX and iDRegEx/kORE operate on different principles: +- **CRX**: Deterministic, always produces something, but over-approximates on diverse groups +- **iDRegEx/kORE**: Probabilistic, need repeating patterns to infer, return None when patterns are too diverse + +**Recommendation:** The flat bags (structure < 0.2) are genuinely diverse groups — +no algorithm can find meaningful structure. The fix is: +1. **Split further** (recursive split already does this) +2. **Filter by structure** (min_structure ≥ 0.3 drops flat bags) +3. **Accept that some groups are noise** and skip them + +Using kORE/iDRegEx as fallback for low-structure groups would just return None +more often. CRX is the right default — it's fast and always produces something. +For the structured groups, CRX already captures the ordering well (score ≥ 0.5). + +**Decision:** Keep CRX as default. kORE/iDRegEx are not better for flat bags +(they return None) and not needed for structured groups (CRX already works). +The pipeline's existing filtering (min_structure, split_mixed) is the right +approach to handle diversity.