105 lines
4.8 KiB
Markdown
105 lines
4.8 KiB
Markdown
|
# Decision Matrix: CRX vs Refined CRX
|
||
|
|
|
||
|
|
## The Question
|
||
|
|
|
||
|
|
When should we use standard CRX (fast, always works) vs refined CRX (cluster-then-infer, tighter but sometimes trivial)?
|
||
|
|
|
||
|
|
## The Data
|
||
|
|
|
||
|
|
Tested on 3 codebases, 14 packages total:
|
||
|
|
|
||
|
|
| Package | N | CRX struct | Refined struct | Winner |
|
||
|
|
|---------|---|-----------|---------------|--------|
|
||
|
|
| RAGSAK agents | 519 | 0.037 | 0.281 | Refined |
|
||
|
|
| RAGSAK buildSrc | 13 | 0.146 | 0.361 | Refined |
|
||
|
|
| RAGSAK entrypoints | 417 | 0.027 | 0.214 | Refined |
|
||
|
|
| RAGSAK platform | 35 | 0.039 | 0.145 | Refined |
|
||
|
|
| FastAPI fastapi | 375 | 0.004 | 0.007 | Refined |
|
||
|
|
| FastAPI scripts | 168 | 0.005 | 0.030 | Refined |
|
||
|
|
| FastAPI tests | 3618 | 0.117 | 0.043 | CRX |
|
||
|
|
| Flask examples | 61 | 0.011 | 0.043 | Refined |
|
||
|
|
| Flask src | 368 | 0.014 | 0.016 | Tie |
|
||
|
|
| RAGSAK app | 383 | 0.028 | 0.600 | Trivial (return+) |
|
||
|
|
| RAGSAK infrastructure | 724 | 0.015 | 0.600 | Trivial (single sym) |
|
||
|
|
| RAGSAK modules | 973 | 0.015 | 0.600 | Trivial (single sym) |
|
||
|
|
| FastAPI docs_src | 650 | 0.027 | 0.500 | Trivial (single sym) |
|
||
|
|
| Flask tests | 993 | 0.016 | 0.500 | Trivial (single sym) |
|
||
|
|
|
||
|
|
## The Pattern
|
||
|
|
|
||
|
|
**Refined CRX wins** (7/14) when:
|
||
|
|
- CRX structure is low (< 0.05) — flat bags where CRX over-approximates
|
||
|
|
- Group size is small-to-medium (13-519 methods)
|
||
|
|
- First symbols are diverse enough to create meaningful clusters
|
||
|
|
|
||
|
|
**Refined CRX is trivial** (5/14) when:
|
||
|
|
- Group size is large (383-973 methods)
|
||
|
|
- Most sequences share the same first symbol (e.g., all start with `return`)
|
||
|
|
- Refined clusters everything into one group → CRX on that group → single symbol
|
||
|
|
|
||
|
|
**CRX wins** (1/14) when:
|
||
|
|
- CRX already has decent structure (> 0.1)
|
||
|
|
- Refined splits too aggressively, losing the overall pattern
|
||
|
|
|
||
|
|
## The Decision Matrix
|
||
|
|
|
||
|
|
```
|
||
|
|
┌─────────────────────────────────┐
|
||
|
|
│ Group size (N methods)? │
|
||
|
|
├────────────┬────────────────────┤
|
||
|
|
│ N <= 50 │ N > 50 │
|
||
|
|
┌───────────────────────┼────────────┼────────────────────┤
|
||
|
|
│ CRX structure < 0.05 │ REFINED │ REFINED │
|
||
|
|
│ (flat bag) │ (always) │ (check for trivial)│
|
||
|
|
├───────────────────────┼────────────┼────────────────────┤
|
||
|
|
│ CRX structure 0.05-0.2│ REFINED │ CRX │
|
||
|
|
│ (semi-structured) │ (usually) │ (safe default) │
|
||
|
|
├───────────────────────┼────────────┼────────────────────┤
|
||
|
|
│ CRX structure > 0.2 │ CRX │ CRX │
|
||
|
|
│ (already structured) │ (already │ (already good) │
|
||
|
|
│ │ good) │ │
|
||
|
|
└───────────────────────┴────────────┴────────────────────┘
|
||
|
|
```
|
||
|
|
|
||
|
|
## The Rule
|
||
|
|
|
||
|
|
```python
|
||
|
|
if crx_structure >= 0.2:
|
||
|
|
use CRX # already good enough
|
||
|
|
elif n_methods <= 50:
|
||
|
|
use refined # small group, safe to cluster
|
||
|
|
elif crx_structure < 0.05:
|
||
|
|
use refined with triviality check # flat bag, worth trying
|
||
|
|
else:
|
||
|
|
use CRX # medium group, semi-structured, CRX is safer
|
||
|
|
```
|
||
|
|
|
||
|
|
## Triviality Check
|
||
|
|
|
||
|
|
When using refined CRX, always check:
|
||
|
|
```python
|
||
|
|
if model_cost(refined_grammar) < 2:
|
||
|
|
use CRX instead # refined produced a single symbol, useless
|
||
|
|
```
|
||
|
|
|
||
|
|
This catches the 36% of cases where refined clusters everything into one group.
|
||
|
|
|
||
|
|
## Chain of Reasoning
|
||
|
|
|
||
|
|
1. **Started with CRX only** — fast, always works, but over-approximates on diverse groups
|
||
|
|
2. **Tried kORE** — slow (400ms), returns None on real data, no advantage over iDRegEx
|
||
|
|
3. **Tried iDRegEx** — slow (700ms), returns None on most data, occasionally useful (477x improvement on 1 package)
|
||
|
|
4. **Tried refined CRX** — cluster-then-infer, better structure on flat bags, but sometimes trivial
|
||
|
|
5. **Tested across 3 codebases** — refined wins 78% when useful, trivial 36% on large groups
|
||
|
|
6. **Conclusion**: CRX is the default, refined is opt-in for flat bags, iDRegEx is optional for rare cases
|
||
|
|
|
||
|
|
## Final Recommendation
|
||
|
|
|
||
|
|
| Scenario | Algorithm | Flag |
|
||
|
|
|----------|-----------|------|
|
||
|
|
| Default (most cases) | CRX | `--crx-method standard` |
|
||
|
|
| Flat bags (struct < 0.05) | Refined CRX | `--crx-method refined` |
|
||
|
|
| Need absolute best grammar | iDRegEx | `--idregex-refine` |
|
||
|
|
| Large groups (N > 500) | CRX | (avoid refined, likely trivial) |
|
||
|
|
| Small groups (N < 20) | Refined CRX | (safe to cluster) |
|