243 lines
9.8 KiB
Markdown
243 lines
9.8 KiB
Markdown
# Experiment Results — Context Strategies + Reduce
|
||
|
||
**Date**: 2026-07-12
|
||
|
||
## Codebases
|
||
|
||
| Codebase | Files | Methods | Language |
|
||
|----------|-------|---------|----------|
|
||
| RAGSAK | 462 | 1594 | Kotlin |
|
||
| Flask | 24 | 1391 | Python |
|
||
|
||
## RAGSAK Results
|
||
|
||
| Strategy | Contexts | SOREs | Coverage | Notes |
|
||
|----------|----------|-------|----------|-------|
|
||
| Baseline (package) | 116 | 2 | 0.6% | Structural — useless |
|
||
| File path k=1 | 43 | 2 | 1.1% | Structural — no better |
|
||
| File path k=2 | 54 | 2 | 1.1% | Same |
|
||
| File path k=3 | 54 | 2 | 1.1% | Same |
|
||
| **First 1 symbol** | 134 | **20** | **4.6%** | Behavioral — 10× better |
|
||
| **First 2 symbols** | 141 | **39** | **9.8%** | Behavioral — 20× better |
|
||
| **First 3 symbols** | 117 | **47** | **12.0%** | **Winner** |
|
||
| Two-dim (p1,s1) | 151 | 24 | 6.3% | Hybrid — worse |
|
||
| Two-dim (p1,s2) | 119 | 33 | 8.7% | Hybrid — worse |
|
||
| Two-dim (p2,s1) | 146 | 24 | 6.3% | Hybrid — worse |
|
||
| Two-dim (p2,s2) | 117 | 33 | 8.7% | Hybrid — worse |
|
||
| Return type heuristic | 5 | 0 | 0.0% | Useless |
|
||
|
||
### Reduce Results (RAGSAK)
|
||
|
||
| Base | Threshold | Merges | Coverage | Notes |
|
||
|------|-----------|--------|----------|-------|
|
||
| k=1 | 0.05-0.4 | 0 | 4.6% | No merges — contexts too distinct |
|
||
| k=2 | 0.05-0.2 | 0 | 9.8% | No merges |
|
||
| k=2 | 0.3 | 1 | 9.6% | Merged `(JobStatus, every)` with `(JobStatus, now)` |
|
||
| k=2 | 0.4 | 2 | 9.5% | Merged 2 pairs |
|
||
| k=3 | 0.05-0.2 | 0 | 12.0% | No merges |
|
||
| k=3 | 0.3 | 1 | 11.7% | Merged `(JobStatus, every, getJobStatus)` with `(JobStatus, now, minusMinutes)` |
|
||
| k=3 | 0.4 | 1 | 11.7% | Same merge |
|
||
|
||
## Flask Results
|
||
|
||
| Strategy | Contexts | SOREs | Coverage | Notes |
|
||
|----------|----------|-------|----------|-------|
|
||
| Baseline (package) | 9 | 0 | 0.0% | Structural — useless |
|
||
| File path k=1 | 7 | 0 | 0.0% | Same |
|
||
| File path k=2 | 9 | 0 | 0.0% | Same |
|
||
| File path k=3 | 9 | 0 | 0.0% | Same |
|
||
| **First 1 symbol** | 53 | **2** | **1.4%** | Behavioral — only 2 SOREs |
|
||
| **First 2 symbols** | 96 | **18** | **7.3%** | Behavioral — 15× better |
|
||
| **First 3 symbols** | 85 | **21** | **10.7%** | **Winner** |
|
||
| Two-dim (p1,s1) | 68 | 5 | 2.2% | Hybrid — worse |
|
||
| Two-dim (p1,s2) | 96 | 22 | 10.1% | Close to behavioral |
|
||
| Two-dim (p2,s1) | 69 | 5 | 2.2% | Worse |
|
||
| Two-dim (p2,s2) | 93 | 19 | 9.5% | Close to behavioral |
|
||
| Return type heuristic | 3 | 0 | 0.0% | Useless |
|
||
|
||
### Reduce Results (Flask)
|
||
|
||
| Base | Threshold | Merges | Coverage | Notes |
|
||
|------|-----------|--------|----------|-------|
|
||
| k=1 | 0.05-0.4 | 0 | 1.4% | No merges |
|
||
| k=2 | 0.05-0.2 | 0 | 7.3% | No merges |
|
||
| k=2 | 0.3-0.4 | 1 | 7.3% | Merged `(def, boolean)` with `(def, is_boolean)` |
|
||
| k=3 | 0.05-0.2 | 0 | 10.7% | No merges |
|
||
| k=3 | 0.3-0.4 | 1 | 10.7% | Merged `(def, boolean, return)` with `(def, is_boolean, return)` |
|
||
|
||
## Cross-Codebase Comparison
|
||
|
||
| Metric | RAGSAK | Flask |
|
||
|--------|--------|-------|
|
||
| Best strategy | First 3 symbols | First 3 symbols |
|
||
| Best coverage | 12.0% | 10.7% |
|
||
| SOREs (best) | 47 | 21 |
|
||
| Reduce merges (ε=0.3) | 1 | 1 |
|
||
| Reduce impact on coverage | -0.3% | 0% |
|
||
|
||
## Key Findings
|
||
|
||
### 1. Behavioral grouping wins consistently
|
||
- Both codebases: First-k-symbols beats all other strategies
|
||
- Structural (file path) is useless — doesn't predict behavioral similarity
|
||
- Hybrid (2D) is worse than pure behavioral
|
||
|
||
### 2. Coverage ceiling is ~10-12%
|
||
- RAGSAK: 12.0% with 47 SOREs
|
||
- Flask: 10.7% with 21 SOREs
|
||
- Most methods have unique call patterns — they don't share prefixes
|
||
|
||
### 3. Reduce has minimal impact
|
||
- Very few merges at any threshold (0-2 per codebase)
|
||
- Support-weighted distance is conservative — requires very similar SOAs
|
||
- Merges that do happen don't improve coverage
|
||
- Reason: contexts created by first-k-symbols are already quite distinct
|
||
|
||
### 4. The few Reduce merges are meaningful
|
||
- RAGSAK: `(JobStatus, every, getJobStatus)` ↔ `(JobStatus, now, minusMinutes)` — same polling pattern, different initial call
|
||
- Flask: `(def, boolean)` ↔ `(def, is_boolean)` — same type-checking pattern, different method name
|
||
|
||
### 5. Flask is harder than RAGSAK
|
||
- Flask has fewer files (24 vs 462) but similar methods (1391 vs 1594)
|
||
- Flask has more unique methods per package — less repetition
|
||
- Flask SOREs are shorter/simpler — less compressible patterns
|
||
|
||
## Round 5: Cross-Package Exact Matches
|
||
|
||
**Hypothesis:** Some call sequences appear verbatim in multiple packages.
|
||
These are the real cross-package conventions.
|
||
|
||
**Method:** Group all sequences by exact tuple match, count packages per sequence.
|
||
|
||
**Result:** 38 exact cross-package sequences in RAGSAK. Most trivial:
|
||
- `('clearAllMocks',)` — 4 packages (test teardown)
|
||
- `('Builder',)` — 4 packages (builder pattern)
|
||
- `('Any',)` — 4 packages (Kotlin type)
|
||
|
||
Interesting ones:
|
||
- `('assumeTrue', 'isDockerAvailable', 'start', 'pullAndWarmup')` — 4 pkgs (Docker setup)
|
||
- `('isNullOrBlank', 'error', 'error')` — 3 pkgs (null check → error)
|
||
- `('sortedBy', 'map', 'toDescriptor')` — 3 pkgs (data pipeline)
|
||
- `('ObjectMapper', 'findAndRegisterModules')` — 2 pkgs (Jackson config)
|
||
|
||
**Verdict:** Exact matches too rare and mostly trivial. The real cross-package
|
||
patterns are structural, not textual — "null check → error" appears with
|
||
different method names in different packages.
|
||
|
||
## Summary of Failed/Dismissed Approaches
|
||
|
||
| Approach | Why it failed |
|
||
|----------|--------------|
|
||
| Per-package inference | Too sparse (1-3 seqs/package) |
|
||
| Reduce algorithm | Wrong abstraction level — merges states within one automaton, not across packages |
|
||
| Cross-package grouping by first symbol | 4.6% / 1.4% coverage — most groups skipped |
|
||
| Exact cross-package matches | 38 sequences, mostly trivial single-call |
|
||
| MDL vs Language Size scoring | Scoring isn't the bottleneck — pattern extraction is |
|
||
|
||
## What Actually Works
|
||
|
||
- **Behavioral grouping (first 3 symbols)** — 12% / 10.7% coverage, consistent across codebases
|
||
- **Calling context as prefix** — the right signal, but package-specific
|
||
- **GBNF conversion** — correct implementation, but input patterns too specific
|
||
|
||
## Next: Structural Coarsening + Cross-Package Detection
|
||
|
||
See `EXPERIMENT_LOG.md` for full reasoning and experiment design.
|
||
|
||
Core idea: collapse method names → categories using tree-sitter capture names.
|
||
Converts textual sequences into structural shapes that repeat across packages.
|
||
|
||
```
|
||
('isNullOrBlank', 'error', 'error') → (CALL, ERROR, ERROR)
|
||
('raise', 'ValueError', 'ValueError') → (CALL, ERROR, ERROR)
|
||
```
|
||
|
||
## Files Generated
|
||
|
||
- `experiments/results/ragsak_summary.json` — RAGSAK metrics
|
||
- `experiments/results/flask_summary.json` — Flask metrics
|
||
- `experiments/context_eval.py` — experiment runner (supports multiple codebases)
|
||
- `experiments/EXPERIMENT_LOG.md` — full experiment history and next steps
|
||
- `bex/reduce.py` — Algorithm 4 (TODS 2010) implementation
|
||
- `bex/gbnf.py` — SORE → GBNF converter
|
||
- `tests/test_reduce.py` — 24 tests for Reduce
|
||
- `tests/test_gbnf.py` — 15 tests for GBNF converter
|
||
|
||
---
|
||
|
||
## Round 20: AST Pipeline + Scoring Fixes (2026-07-13)
|
||
|
||
**Commit range:** `ea6cac5` → `8b3a454`
|
||
|
||
### Codebases
|
||
|
||
| Codebase | Files | Methods | Language |
|
||
|----------|-------|---------|----------|
|
||
| RAGSAK | 462 | 1609 | Kotlin |
|
||
| FastAPI | — | — | Python |
|
||
| Zod | — | — | TypeScript |
|
||
|
||
### Scoring: Language Size over MDL (ADR-13)
|
||
|
||
Abandoned MDL scoring — it rewards short expressions, so generic `info+` beat specific
|
||
`a.b.c.d.e+` (21% vs 98% success in Bex paper). Language Size (`lang_size_score`) chosen.
|
||
|
||
| Metric | Bag grammar | Structured grammar |
|
||
|--------|-------------|-------------------|
|
||
| `lang_size_score` | 9975 | 20 |
|
||
| `mdl_score` | 10^12 (clamped) | 10^12 (clamped) |
|
||
|
||
### Final Defaults
|
||
|
||
| Parameter | Before | After |
|
||
|-----------|--------|-------|
|
||
| `decompose` | False | **True** |
|
||
| `max_seq_length` | 5 | **4** |
|
||
| `idregex_refine` | False | **True** |
|
||
| `_COUNT_CAP` | 10^12 | **10^30** |
|
||
|
||
### Results
|
||
|
||
| Codebase | Grammars | Pure Bags | Structured | Bag % |
|
||
|----------|----------|-----------|------------|-------|
|
||
| RAGSAK (v4) | 126 | 6 | 120 | 4.8% |
|
||
| FastAPI (v3) | 143 | 26 | 117 | 18.2% |
|
||
| Zod (v3) | 23 | 5 | 18 | 21.7% |
|
||
|
||
**Quality breakdown:** ~85% of grammars across codebases remain orderless bags `(A|B|C)+`.
|
||
The ~15% that are structured represent real sequential flows:
|
||
- Web controller tests: `post→jsonPath→isEqualTo→exchange→expectStatus`
|
||
- API client patterns: `request→header→send→statusCode→jsonPath`
|
||
- Builder chains: `builder→field→value→build→validate`
|
||
|
||
### iDRegEx Findings
|
||
|
||
iDRegEx does NOT help at small scale. On 3-method groups, iDRegEx achieves only 3.8×
|
||
tighter (below the 10× gate threshold). The gate correctly rejects it.
|
||
|
||
| Group size | CRX lang_size | iDRegEx lang_size | Ratio |
|
||
|------------|--------------|-------------------|-------|
|
||
| 3 methods | 15 | 4 | 3.8× |
|
||
| 4 methods (`storage`) | — | — | 91× (outlier) |
|
||
|
||
Bags survive because:
|
||
1. CRX emits one grammar deterministically (no alternative to compare)
|
||
2. `lang_size_score` only ranks **between** algorithms, not within CRX's own output
|
||
3. iDRegEx is too slow for large groups (200s+ timeout on 2036m FastAPI tests)
|
||
|
||
### Key Insight
|
||
|
||
The grammar inference pipeline is fundamentally limited by the input: if methods in a
|
||
package don't share a sequential calling pattern, no algorithm can find one. The ~15%
|
||
structured grammars represent genuinely reusable patterns; the ~85% bags represent
|
||
packages with diverse, unrelated methods grouped only by directory proximity.
|
||
|
||
### Files
|
||
|
||
- `bex/grammar.py`: AST nodes, `_count_concat` memoization, `_COUNT_CAP = 10^30`
|
||
- `bex/crx.py`: CRX algorithm (AST-based)
|
||
- `bex/mdl.py`: `lang_size_score`, `model_cost`, `data_cost`
|
||
- `bex/idregex.py`: iDRegEx algorithm
|
||
- `bex/decompose.py`: sequence decomposition
|
||
- `bex/tag_preprocessor/analyze.py`: pipeline orchestration, all defaults
|
||
- `experiments/results/round20_ast_verify/`: full experiment data (v2/v3/v4)
|