b516b2985d
- bex/reduce.py: Faithful implementation of Reduce with support-weighted SOA edit distance, adjunction, iterative merging, and Minimize - experiments/context_eval.py: Multi-codebase support (RAGSAK + Flask), Reduce experiments with thresholds 0.05-0.4 - tests/test_reduce.py: 24 tests covering all Reduce components - Flask cloned to external_refs/flask for cross-validation Results: - RAGSAK: 12.0% coverage (First 3 symbols) - Flask: 10.7% coverage (First 3 symbols) - Reduce has minimal impact (1-2 merges per codebase at ε=0.3) - Coverage ceiling appears to be ~10-12% for prefix-based grouping
119 lines
5 KiB
Markdown
119 lines
5 KiB
Markdown
# Experiment Results — Context Strategies + Reduce
|
||
|
||
**Date**: 2026-07-12
|
||
|
||
## Codebases
|
||
|
||
| Codebase | Files | Methods | Language |
|
||
|----------|-------|---------|----------|
|
||
| RAGSAK | 462 | 1594 | Kotlin |
|
||
| Flask | 24 | 1391 | Python |
|
||
|
||
## RAGSAK Results
|
||
|
||
| Strategy | Contexts | SOREs | Coverage | Notes |
|
||
|----------|----------|-------|----------|-------|
|
||
| Baseline (package) | 116 | 2 | 0.6% | Structural — useless |
|
||
| File path k=1 | 43 | 2 | 1.1% | Structural — no better |
|
||
| File path k=2 | 54 | 2 | 1.1% | Same |
|
||
| File path k=3 | 54 | 2 | 1.1% | Same |
|
||
| **First 1 symbol** | 134 | **20** | **4.6%** | Behavioral — 10× better |
|
||
| **First 2 symbols** | 141 | **39** | **9.8%** | Behavioral — 20× better |
|
||
| **First 3 symbols** | 117 | **47** | **12.0%** | **Winner** |
|
||
| Two-dim (p1,s1) | 151 | 24 | 6.3% | Hybrid — worse |
|
||
| Two-dim (p1,s2) | 119 | 33 | 8.7% | Hybrid — worse |
|
||
| Two-dim (p2,s1) | 146 | 24 | 6.3% | Hybrid — worse |
|
||
| Two-dim (p2,s2) | 117 | 33 | 8.7% | Hybrid — worse |
|
||
| Return type heuristic | 5 | 0 | 0.0% | Useless |
|
||
|
||
### Reduce Results (RAGSAK)
|
||
|
||
| Base | Threshold | Merges | Coverage | Notes |
|
||
|------|-----------|--------|----------|-------|
|
||
| k=1 | 0.05-0.4 | 0 | 4.6% | No merges — contexts too distinct |
|
||
| k=2 | 0.05-0.2 | 0 | 9.8% | No merges |
|
||
| k=2 | 0.3 | 1 | 9.6% | Merged `(JobStatus, every)` with `(JobStatus, now)` |
|
||
| k=2 | 0.4 | 2 | 9.5% | Merged 2 pairs |
|
||
| k=3 | 0.05-0.2 | 0 | 12.0% | No merges |
|
||
| k=3 | 0.3 | 1 | 11.7% | Merged `(JobStatus, every, getJobStatus)` with `(JobStatus, now, minusMinutes)` |
|
||
| k=3 | 0.4 | 1 | 11.7% | Same merge |
|
||
|
||
## Flask Results
|
||
|
||
| Strategy | Contexts | SOREs | Coverage | Notes |
|
||
|----------|----------|-------|----------|-------|
|
||
| Baseline (package) | 9 | 0 | 0.0% | Structural — useless |
|
||
| File path k=1 | 7 | 0 | 0.0% | Same |
|
||
| File path k=2 | 9 | 0 | 0.0% | Same |
|
||
| File path k=3 | 9 | 0 | 0.0% | Same |
|
||
| **First 1 symbol** | 53 | **2** | **1.4%** | Behavioral — only 2 SOREs |
|
||
| **First 2 symbols** | 96 | **18** | **7.3%** | Behavioral — 15× better |
|
||
| **First 3 symbols** | 85 | **21** | **10.7%** | **Winner** |
|
||
| Two-dim (p1,s1) | 68 | 5 | 2.2% | Hybrid — worse |
|
||
| Two-dim (p1,s2) | 96 | 22 | 10.1% | Close to behavioral |
|
||
| Two-dim (p2,s1) | 69 | 5 | 2.2% | Worse |
|
||
| Two-dim (p2,s2) | 93 | 19 | 9.5% | Close to behavioral |
|
||
| Return type heuristic | 3 | 0 | 0.0% | Useless |
|
||
|
||
### Reduce Results (Flask)
|
||
|
||
| Base | Threshold | Merges | Coverage | Notes |
|
||
|------|-----------|--------|----------|-------|
|
||
| k=1 | 0.05-0.4 | 0 | 1.4% | No merges |
|
||
| k=2 | 0.05-0.2 | 0 | 7.3% | No merges |
|
||
| k=2 | 0.3-0.4 | 1 | 7.3% | Merged `(def, boolean)` with `(def, is_boolean)` |
|
||
| k=3 | 0.05-0.2 | 0 | 10.7% | No merges |
|
||
| k=3 | 0.3-0.4 | 1 | 10.7% | Merged `(def, boolean, return)` with `(def, is_boolean, return)` |
|
||
|
||
## Cross-Codebase Comparison
|
||
|
||
| Metric | RAGSAK | Flask |
|
||
|--------|--------|-------|
|
||
| Best strategy | First 3 symbols | First 3 symbols |
|
||
| Best coverage | 12.0% | 10.7% |
|
||
| SOREs (best) | 47 | 21 |
|
||
| Reduce merges (ε=0.3) | 1 | 1 |
|
||
| Reduce impact on coverage | -0.3% | 0% |
|
||
|
||
## Key Findings
|
||
|
||
### 1. Behavioral grouping wins consistently
|
||
- Both codebases: First-k-symbols beats all other strategies
|
||
- Structural (file path) is useless — doesn't predict behavioral similarity
|
||
- Hybrid (2D) is worse than pure behavioral
|
||
|
||
### 2. Coverage ceiling is ~10-12%
|
||
- RAGSAK: 12.0% with 47 SOREs
|
||
- Flask: 10.7% with 21 SOREs
|
||
- Most methods have unique call patterns — they don't share prefixes
|
||
|
||
### 3. Reduce has minimal impact
|
||
- Very few merges at any threshold (0-2 per codebase)
|
||
- Support-weighted distance is conservative — requires very similar SOAs
|
||
- Merges that do happen don't improve coverage
|
||
- Reason: contexts created by first-k-symbols are already quite distinct
|
||
|
||
### 4. The few Reduce merges are meaningful
|
||
- RAGSAK: `(JobStatus, every, getJobStatus)` ↔ `(JobStatus, now, minusMinutes)` — same polling pattern, different initial call
|
||
- Flask: `(def, boolean)` ↔ `(def, is_boolean)` — same type-checking pattern, different method name
|
||
|
||
### 5. Flask is harder than RAGSAK
|
||
- Flask has fewer files (24 vs 462) but similar methods (1391 vs 1594)
|
||
- Flask has more unique methods per package — less repetition
|
||
- Flask SOREs are shorter/simpler — less compressible patterns
|
||
|
||
## Next Steps
|
||
|
||
1. **Accept the ceiling**: ~10-12% coverage may be the maximum for prefix-based grouping
|
||
2. **Symbol coarsening**: Collapse specific symbols into categories (GETTER, SETTER, etc.)
|
||
3. **Hierarchical grouping**: Group by k=1 first, then sub-group by k=2 within each group
|
||
4. **GBNF output**: Convert SOREs to GBNF format for llama.cpp constrained decoding
|
||
5. **Test on kotlinx.coroutines**: Third codebase for cross-validation
|
||
|
||
## Files Generated
|
||
|
||
- `experiments/results/ragsak_summary.json` — RAGSAK metrics
|
||
- `experiments/results/flask_summary.json` — Flask metrics
|
||
- `experiments/context_eval.py` — experiment runner (supports multiple codebases)
|
||
- `bex/reduce.py` — Algorithm 4 (TODS 2010) implementation
|
||
- `tests/test_reduce.py` — 24 tests for Reduce
|