grammar-inference-engine/experiments/RESULTS.md

# Experiment Results — Context Strategies + Reduce

**Date**: 2026-07-12

## Codebases

| Codebase | Files | Methods | Language |
|----------|-------|---------|----------|
| RAGSAK | 462 | 1594 | Kotlin |
| Flask | 24 | 1391 | Python |

## RAGSAK Results

| Strategy | Contexts | SOREs | Coverage | Notes |
|----------|----------|-------|----------|-------|
| Baseline (package) | 116 | 2 | 0.6% | Structural — useless |
| File path k=1 | 43 | 2 | 1.1% | Structural — no better |
| File path k=2 | 54 | 2 | 1.1% | Same |
| File path k=3 | 54 | 2 | 1.1% | Same |
| **First 1 symbol** | 134 | **20** | **4.6%** | Behavioral — 10× better |
| **First 2 symbols** | 141 | **39** | **9.8%** | Behavioral — 20× better |
| **First 3 symbols** | 117 | **47** | **12.0%** | **Winner** |
| Two-dim (p1,s1) | 151 | 24 | 6.3% | Hybrid — worse |
| Two-dim (p1,s2) | 119 | 33 | 8.7% | Hybrid — worse |
| Two-dim (p2,s1) | 146 | 24 | 6.3% | Hybrid — worse |
| Two-dim (p2,s2) | 117 | 33 | 8.7% | Hybrid — worse |
| Return type heuristic | 5 | 0 | 0.0% | Useless |

### Reduce Results (RAGSAK)

| Base | Threshold | Merges | Coverage | Notes |
|------|-----------|--------|----------|-------|
| k=1 | 0.05-0.4 | 0 | 4.6% | No merges — contexts too distinct |
| k=2 | 0.05-0.2 | 0 | 9.8% | No merges |
| k=2 | 0.3 | 1 | 9.6% | Merged `(JobStatus, every)` with `(JobStatus, now)` |
| k=2 | 0.4 | 2 | 9.5% | Merged 2 pairs |
| k=3 | 0.05-0.2 | 0 | 12.0% | No merges |
| k=3 | 0.3 | 1 | 11.7% | Merged `(JobStatus, every, getJobStatus)` with `(JobStatus, now, minusMinutes)` |
| k=3 | 0.4 | 1 | 11.7% | Same merge |

## Flask Results

| Strategy | Contexts | SOREs | Coverage | Notes |
|----------|----------|-------|----------|-------|
| Baseline (package) | 9 | 0 | 0.0% | Structural — useless |
| File path k=1 | 7 | 0 | 0.0% | Same |
| File path k=2 | 9 | 0 | 0.0% | Same |
| File path k=3 | 9 | 0 | 0.0% | Same |
| **First 1 symbol** | 53 | **2** | **1.4%** | Behavioral — only 2 SOREs |
| **First 2 symbols** | 96 | **18** | **7.3%** | Behavioral — 15× better |
| **First 3 symbols** | 85 | **21** | **10.7%** | **Winner** |
| Two-dim (p1,s1) | 68 | 5 | 2.2% | Hybrid — worse |
| Two-dim (p1,s2) | 96 | 22 | 10.1% | Close to behavioral |
| Two-dim (p2,s1) | 69 | 5 | 2.2% | Worse |
| Two-dim (p2,s2) | 93 | 19 | 9.5% | Close to behavioral |
| Return type heuristic | 3 | 0 | 0.0% | Useless |

### Reduce Results (Flask)

| Base | Threshold | Merges | Coverage | Notes |
|------|-----------|--------|----------|-------|
| k=1 | 0.05-0.4 | 0 | 1.4% | No merges |
| k=2 | 0.05-0.2 | 0 | 7.3% | No merges |
| k=2 | 0.3-0.4 | 1 | 7.3% | Merged `(def, boolean)` with `(def, is_boolean)` |
| k=3 | 0.05-0.2 | 0 | 10.7% | No merges |
| k=3 | 0.3-0.4 | 1 | 10.7% | Merged `(def, boolean, return)` with `(def, is_boolean, return)` |

## Cross-Codebase Comparison

| Metric | RAGSAK | Flask |
|--------|--------|-------|
| Best strategy | First 3 symbols | First 3 symbols |
| Best coverage | 12.0% | 10.7% |
| SOREs (best) | 47 | 21 |
| Reduce merges (ε=0.3) | 1 | 1 |
| Reduce impact on coverage | -0.3% | 0% |

## Key Findings

### 1. Behavioral grouping wins consistently
- Both codebases: First-k-symbols beats all other strategies
- Structural (file path) is useless — doesn't predict behavioral similarity
- Hybrid (2D) is worse than pure behavioral

### 2. Coverage ceiling is ~10-12%
- RAGSAK: 12.0% with 47 SOREs
- Flask: 10.7% with 21 SOREs
- Most methods have unique call patterns — they don't share prefixes

### 3. Reduce has minimal impact
- Very few merges at any threshold (0-2 per codebase)
- Support-weighted distance is conservative — requires very similar SOAs
- Merges that do happen don't improve coverage
- Reason: contexts created by first-k-symbols are already quite distinct

### 4. The few Reduce merges are meaningful
- RAGSAK: `(JobStatus, every, getJobStatus)` ↔ `(JobStatus, now, minusMinutes)` — same polling pattern, different initial call
- Flask: `(def, boolean)` ↔ `(def, is_boolean)` — same type-checking pattern, different method name

### 5. Flask is harder than RAGSAK
- Flask has fewer files (24 vs 462) but similar methods (1391 vs 1594)
- Flask has more unique methods per package — less repetition
- Flask SOREs are shorter/simpler — less compressible patterns

## Next Steps

1. **Accept the ceiling**: ~10-12% coverage may be the maximum for prefix-based grouping
2. **Symbol coarsening**: Collapse specific symbols into categories (GETTER, SETTER, etc.)
3. **Hierarchical grouping**: Group by k=1 first, then sub-group by k=2 within each group
4. **GBNF output**: Convert SOREs to GBNF format for llama.cpp constrained decoding
5. **Test on kotlinx.coroutines**: Third codebase for cross-validation

## Files Generated

- `experiments/results/ragsak_summary.json` — RAGSAK metrics
- `experiments/results/flask_summary.json` — Flask metrics
- `experiments/context_eval.py` — experiment runner (supports multiple codebases)
- `bex/reduce.py` — Algorithm 4 (TODS 2010) implementation
- `tests/test_reduce.py` — 24 tests for Reduce