8e1c7f3767
- Full experiment history: context strategies, Reduce, scoring, GBNF, cross-package - Documented failures: per-package sparsity, Reduce at wrong level, exact match rarity - Designed next experiment: structural coarsening via tree-sitter categories - Updated RESULTS.md with Round 5 findings and summary table
164 lines
6.9 KiB
Markdown
164 lines
6.9 KiB
Markdown
# Experiment Results — Context Strategies + Reduce
|
||
|
||
**Date**: 2026-07-12
|
||
|
||
## Codebases
|
||
|
||
| Codebase | Files | Methods | Language |
|
||
|----------|-------|---------|----------|
|
||
| RAGSAK | 462 | 1594 | Kotlin |
|
||
| Flask | 24 | 1391 | Python |
|
||
|
||
## RAGSAK Results
|
||
|
||
| Strategy | Contexts | SOREs | Coverage | Notes |
|
||
|----------|----------|-------|----------|-------|
|
||
| Baseline (package) | 116 | 2 | 0.6% | Structural — useless |
|
||
| File path k=1 | 43 | 2 | 1.1% | Structural — no better |
|
||
| File path k=2 | 54 | 2 | 1.1% | Same |
|
||
| File path k=3 | 54 | 2 | 1.1% | Same |
|
||
| **First 1 symbol** | 134 | **20** | **4.6%** | Behavioral — 10× better |
|
||
| **First 2 symbols** | 141 | **39** | **9.8%** | Behavioral — 20× better |
|
||
| **First 3 symbols** | 117 | **47** | **12.0%** | **Winner** |
|
||
| Two-dim (p1,s1) | 151 | 24 | 6.3% | Hybrid — worse |
|
||
| Two-dim (p1,s2) | 119 | 33 | 8.7% | Hybrid — worse |
|
||
| Two-dim (p2,s1) | 146 | 24 | 6.3% | Hybrid — worse |
|
||
| Two-dim (p2,s2) | 117 | 33 | 8.7% | Hybrid — worse |
|
||
| Return type heuristic | 5 | 0 | 0.0% | Useless |
|
||
|
||
### Reduce Results (RAGSAK)
|
||
|
||
| Base | Threshold | Merges | Coverage | Notes |
|
||
|------|-----------|--------|----------|-------|
|
||
| k=1 | 0.05-0.4 | 0 | 4.6% | No merges — contexts too distinct |
|
||
| k=2 | 0.05-0.2 | 0 | 9.8% | No merges |
|
||
| k=2 | 0.3 | 1 | 9.6% | Merged `(JobStatus, every)` with `(JobStatus, now)` |
|
||
| k=2 | 0.4 | 2 | 9.5% | Merged 2 pairs |
|
||
| k=3 | 0.05-0.2 | 0 | 12.0% | No merges |
|
||
| k=3 | 0.3 | 1 | 11.7% | Merged `(JobStatus, every, getJobStatus)` with `(JobStatus, now, minusMinutes)` |
|
||
| k=3 | 0.4 | 1 | 11.7% | Same merge |
|
||
|
||
## Flask Results
|
||
|
||
| Strategy | Contexts | SOREs | Coverage | Notes |
|
||
|----------|----------|-------|----------|-------|
|
||
| Baseline (package) | 9 | 0 | 0.0% | Structural — useless |
|
||
| File path k=1 | 7 | 0 | 0.0% | Same |
|
||
| File path k=2 | 9 | 0 | 0.0% | Same |
|
||
| File path k=3 | 9 | 0 | 0.0% | Same |
|
||
| **First 1 symbol** | 53 | **2** | **1.4%** | Behavioral — only 2 SOREs |
|
||
| **First 2 symbols** | 96 | **18** | **7.3%** | Behavioral — 15× better |
|
||
| **First 3 symbols** | 85 | **21** | **10.7%** | **Winner** |
|
||
| Two-dim (p1,s1) | 68 | 5 | 2.2% | Hybrid — worse |
|
||
| Two-dim (p1,s2) | 96 | 22 | 10.1% | Close to behavioral |
|
||
| Two-dim (p2,s1) | 69 | 5 | 2.2% | Worse |
|
||
| Two-dim (p2,s2) | 93 | 19 | 9.5% | Close to behavioral |
|
||
| Return type heuristic | 3 | 0 | 0.0% | Useless |
|
||
|
||
### Reduce Results (Flask)
|
||
|
||
| Base | Threshold | Merges | Coverage | Notes |
|
||
|------|-----------|--------|----------|-------|
|
||
| k=1 | 0.05-0.4 | 0 | 1.4% | No merges |
|
||
| k=2 | 0.05-0.2 | 0 | 7.3% | No merges |
|
||
| k=2 | 0.3-0.4 | 1 | 7.3% | Merged `(def, boolean)` with `(def, is_boolean)` |
|
||
| k=3 | 0.05-0.2 | 0 | 10.7% | No merges |
|
||
| k=3 | 0.3-0.4 | 1 | 10.7% | Merged `(def, boolean, return)` with `(def, is_boolean, return)` |
|
||
|
||
## Cross-Codebase Comparison
|
||
|
||
| Metric | RAGSAK | Flask |
|
||
|--------|--------|-------|
|
||
| Best strategy | First 3 symbols | First 3 symbols |
|
||
| Best coverage | 12.0% | 10.7% |
|
||
| SOREs (best) | 47 | 21 |
|
||
| Reduce merges (ε=0.3) | 1 | 1 |
|
||
| Reduce impact on coverage | -0.3% | 0% |
|
||
|
||
## Key Findings
|
||
|
||
### 1. Behavioral grouping wins consistently
|
||
- Both codebases: First-k-symbols beats all other strategies
|
||
- Structural (file path) is useless — doesn't predict behavioral similarity
|
||
- Hybrid (2D) is worse than pure behavioral
|
||
|
||
### 2. Coverage ceiling is ~10-12%
|
||
- RAGSAK: 12.0% with 47 SOREs
|
||
- Flask: 10.7% with 21 SOREs
|
||
- Most methods have unique call patterns — they don't share prefixes
|
||
|
||
### 3. Reduce has minimal impact
|
||
- Very few merges at any threshold (0-2 per codebase)
|
||
- Support-weighted distance is conservative — requires very similar SOAs
|
||
- Merges that do happen don't improve coverage
|
||
- Reason: contexts created by first-k-symbols are already quite distinct
|
||
|
||
### 4. The few Reduce merges are meaningful
|
||
- RAGSAK: `(JobStatus, every, getJobStatus)` ↔ `(JobStatus, now, minusMinutes)` — same polling pattern, different initial call
|
||
- Flask: `(def, boolean)` ↔ `(def, is_boolean)` — same type-checking pattern, different method name
|
||
|
||
### 5. Flask is harder than RAGSAK
|
||
- Flask has fewer files (24 vs 462) but similar methods (1391 vs 1594)
|
||
- Flask has more unique methods per package — less repetition
|
||
- Flask SOREs are shorter/simpler — less compressible patterns
|
||
|
||
## Round 5: Cross-Package Exact Matches
|
||
|
||
**Hypothesis:** Some call sequences appear verbatim in multiple packages.
|
||
These are the real cross-package conventions.
|
||
|
||
**Method:** Group all sequences by exact tuple match, count packages per sequence.
|
||
|
||
**Result:** 38 exact cross-package sequences in RAGSAK. Most trivial:
|
||
- `('clearAllMocks',)` — 4 packages (test teardown)
|
||
- `('Builder',)` — 4 packages (builder pattern)
|
||
- `('Any',)` — 4 packages (Kotlin type)
|
||
|
||
Interesting ones:
|
||
- `('assumeTrue', 'isDockerAvailable', 'start', 'pullAndWarmup')` — 4 pkgs (Docker setup)
|
||
- `('isNullOrBlank', 'error', 'error')` — 3 pkgs (null check → error)
|
||
- `('sortedBy', 'map', 'toDescriptor')` — 3 pkgs (data pipeline)
|
||
- `('ObjectMapper', 'findAndRegisterModules')` — 2 pkgs (Jackson config)
|
||
|
||
**Verdict:** Exact matches too rare and mostly trivial. The real cross-package
|
||
patterns are structural, not textual — "null check → error" appears with
|
||
different method names in different packages.
|
||
|
||
## Summary of Failed/Dismissed Approaches
|
||
|
||
| Approach | Why it failed |
|
||
|----------|--------------|
|
||
| Per-package inference | Too sparse (1-3 seqs/package) |
|
||
| Reduce algorithm | Wrong abstraction level — merges states within one automaton, not across packages |
|
||
| Cross-package grouping by first symbol | 4.6% / 1.4% coverage — most groups skipped |
|
||
| Exact cross-package matches | 38 sequences, mostly trivial single-call |
|
||
| MDL vs Language Size scoring | Scoring isn't the bottleneck — pattern extraction is |
|
||
|
||
## What Actually Works
|
||
|
||
- **Behavioral grouping (first 3 symbols)** — 12% / 10.7% coverage, consistent across codebases
|
||
- **Calling context as prefix** — the right signal, but package-specific
|
||
- **GBNF conversion** — correct implementation, but input patterns too specific
|
||
|
||
## Next: Structural Coarsening + Cross-Package Detection
|
||
|
||
See `EXPERIMENT_LOG.md` for full reasoning and experiment design.
|
||
|
||
Core idea: collapse method names → categories using tree-sitter capture names.
|
||
Converts textual sequences into structural shapes that repeat across packages.
|
||
|
||
```
|
||
('isNullOrBlank', 'error', 'error') → (CALL, ERROR, ERROR)
|
||
('raise', 'ValueError', 'ValueError') → (CALL, ERROR, ERROR)
|
||
```
|
||
|
||
## Files Generated
|
||
|
||
- `experiments/results/ragsak_summary.json` — RAGSAK metrics
|
||
- `experiments/results/flask_summary.json` — Flask metrics
|
||
- `experiments/context_eval.py` — experiment runner (supports multiple codebases)
|
||
- `experiments/EXPERIMENT_LOG.md` — full experiment history and next steps
|
||
- `bex/reduce.py` — Algorithm 4 (TODS 2010) implementation
|
||
- `bex/gbnf.py` — SORE → GBNF converter
|
||
- `tests/test_reduce.py` — 24 tests for Reduce
|
||
- `tests/test_gbnf.py` — 15 tests for GBNF converter
|