grammar-inference-engine/experiments/DECISION_MATRIX.md

Decision Matrix: CRX vs Refined CRX

The Question

When should we use standard CRX (fast, always works) vs refined CRX (cluster-then-infer, tighter but sometimes trivial)?

The Data

Tested on 3 codebases, 14 packages total:

Package	N	CRX struct	Refined struct	Winner
RAGSAK agents	519	0.037	0.281	Refined
RAGSAK buildSrc	13	0.146	0.361	Refined
RAGSAK entrypoints	417	0.027	0.214	Refined
RAGSAK platform	35	0.039	0.145	Refined
FastAPI fastapi	375	0.004	0.007	Refined
FastAPI scripts	168	0.005	0.030	Refined
FastAPI tests	3618	0.117	0.043	CRX
Flask examples	61	0.011	0.043	Refined
Flask src	368	0.014	0.016	Tie
RAGSAK app	383	0.028	0.600	Trivial (return+)
RAGSAK infrastructure	724	0.015	0.600	Trivial (single sym)
RAGSAK modules	973	0.015	0.600	Trivial (single sym)
FastAPI docs_src	650	0.027	0.500	Trivial (single sym)
Flask tests	993	0.016	0.500	Trivial (single sym)

The Pattern

Refined CRX wins (7/14) when:

• CRX structure is low (< 0.05) — flat bags where CRX over-approximates
• Group size is small-to-medium (13-519 methods)
• First symbols are diverse enough to create meaningful clusters

Refined CRX is trivial (5/14) when:

• Group size is large (383-973 methods)
• Most sequences share the same first symbol (e.g., all start with return)
• Refined clusters everything into one group → CRX on that group → single symbol

CRX wins (1/14) when:

• CRX already has decent structure (> 0.1)
• Refined splits too aggressively, losing the overall pattern

The Decision Matrix

                        ┌─────────────────────────────────┐
                        │     Group size (N methods)?      │
                        ├────────────┬────────────────────┤
                        │   N <= 50  │     N > 50         │
┌───────────────────────┼────────────┼────────────────────┤
│ CRX structure < 0.05  │  REFINED   │  REFINED           │
│ (flat bag)            │  (always)  │  (check for trivial)│
├───────────────────────┼────────────┼────────────────────┤
│ CRX structure 0.05-0.2│  REFINED   │  CRX               │
│ (semi-structured)     │  (usually) │  (safe default)     │
├───────────────────────┼────────────┼────────────────────┤
│ CRX structure > 0.2   │  CRX       │  CRX               │
│ (already structured)  │  (already  │  (already good)     │
│                       │   good)    │                     │
└───────────────────────┴────────────┴────────────────────┘

The Rule

if crx_structure >= 0.2:
    use CRX  # already good enough
elif n_methods <= 50:
    use refined  # small group, safe to cluster
elif crx_structure < 0.05:
    use refined with triviality check  # flat bag, worth trying
else:
    use CRX  # medium group, semi-structured, CRX is safer

Triviality Check

When using refined CRX, always check:

if model_cost(refined_grammar) < 2:
    use CRX instead  # refined produced a single symbol, useless

This catches the 36% of cases where refined clusters everything into one group.

Chain of Reasoning

1. Started with CRX only — fast, always works, but over-approximates on diverse groups
2. Tried kORE — slow (400ms), returns None on real data, no advantage over iDRegEx
3. Tried iDRegEx — slow (700ms), returns None on most data, occasionally useful (477x improvement on 1 package)
4. Tried refined CRX — cluster-then-infer, better structure on flat bags, but sometimes trivial
5. Tested across 3 codebases — refined wins 78% when useful, trivial 36% on large groups
6. Conclusion: CRX is the default, refined is opt-in for flat bags, iDRegEx is optional for rare cases

Final Recommendation

Scenario	Algorithm	Flag
Default (most cases)	CRX	--crx-method standard
Flat bags (struct < 0.05)	Refined CRX	--crx-method refined
Need absolute best grammar	iDRegEx	--idregex-refine
Large groups (N > 500)	CRX	(avoid refined, likely trivial)
Small groups (N < 20)	Refined CRX	(safe to cluster)