# Experiment Results — RAGSAK Context Strategies

**Date**: 2026-07-11
**Codebase**: RAGSAK (462 .kt files, 1594 methods, 137 packages)

## Summary Table

| Strategy | Contexts | SORE OK | Coverage | Notes |
|----------|----------|---------|----------|-------|
| Baseline (package) | 116 | 2 | 0.6% | Current approach — terrible |
| File path k=1 | 43 | 2 | 1.1% | Structural — no better than baseline |
| File path k=2 | 54 | 2 | 1.1% | Same — structural doesn't help |
| File path k=3 | 54 | 2 | 1.1% | Same — structural doesn't help |
| **First 1 symbol** | 134 | **20** | **4.6%** | Behavioral — 10× better |
| **First 2 symbols** | 141 | **39** | **9.8%** | Behavioral — 20× better |
| **First 3 symbols** | 117 | **47** | **12.0%** | Behavioral — **winner** |
| Two-dim (p1,s1) | 151 | 24 | 6.3% | Hybrid — worse than behavioral alone |
| Two-dim (p1,s2) | 119 | 33 | 8.7% | Hybrid — worse than behavioral alone |
| Two-dim (p2,s1) | 146 | 24 | 6.3% | Hybrid — worse than behavioral alone |
| Two-dim (p2,s2) | 117 | 33 | 8.7% | Hybrid — worse than behavioral alone |
| Return type heuristic | 5 | 0 | 0.0% | Too coarse — useless |

## Key Findings

### 1. Behavioral grouping > Structural grouping
- File path strategies (A) produce same results as baseline — structural similarity doesn't predict behavioral similarity
- First-k-symbols strategies (B) produce 20× more SOREs — behavioral prefix is the right signal
- **Winner: Option B, k=3** — 47 SOREs, 12% coverage

### 2. Two-dimensional doesn't help
- Adding file path to behavioral grouping **reduces** performance
- The structural component drags down the behavioral signal
- Conclusion: **drop the structural dimension entirely**

### 3. The successful SOREs reveal real patterns

**Test patterns** (most common):
- `post.uri.exchange.expectStatus.(expectBody.jsonPath.isEqualTo)?` — HTTP test flow
- `runTest.post.uri.bodyValue.(ChatRequest|SessionChatRequest).exchange.expectStatus` — test setup
- `every.getJobStatus.get.uri.exchange.expectStatus.(expectBody.jsonPath.isEqualTo)?` — polling pattern

**Assertion patterns**:
- `assertEquals.(of.(assertFailsWith)?)+` — assertion chain
- `filesIn.assertTrue.(hasImport)+` — file validation
- `trim.lowercase.(warn)+` — string processing

**Mock patterns**:
- `mockk.every....` — mock setup
- `clearAllMocks` — cleanup

**CRUD patterns**:
- `info.deleteByJobId` — delete operation
- `info.deleteByKnowledgeBaseId` — delete operation
- `update.trimIndent.now.insertRow` — update operation

### 4. Why coverage is still low (12%)
- Most methods have **unique call patterns** — they don't share a 3-symbol prefix with any other method
- The 3-symbol prefix is too specific for diverse codebases
- Need either:
  - More data (more codebases to train on)
  - Looser grouping (k=1 or k=2, but with merging)
  - Symbol coarsening (collapse specific symbols into categories)

### 5. Smart filters work well
- `max_unique_ratio=0.85`: Skips 60-80% of groups (too diverse)
- `max_alphabet=20`: Skips groups with too many unique symbols
- `max_soa_edges=100`: Skips complex SOAs that would be slow
- **Result**: Experiment runs in <1s instead of minutes

## Successful SORE Examples (Top 10 by group size)

| Context | Methods | Unique Ratio | SORE |
|---------|---------|--------------|------|
| `ery {.stKnowledgeBases().gRequest(m` | 12 | 0.333 | `ery {.stKnowledgeBases().gRequest(m.(ckKnowledgeBase(re.ertEquals(Kn."k\|eckKnowledgeBase(r.(sertEquals(K.("\|(sertEquals(` |
| `mockk` | 11 | 0.091 | `mockk` |
| `filesIn.assertTrue.hasImport` | 7 | 0.286 | `filesIn.assertTrue.(hasImport)+` |
| `of` | 7 | 0.143 | `of` |
| `assertEquals.of.assertFailsWith` | 7 | 0.429 | `assertEquals.(of.(assertFailsWith)?)+` |
| `filesIn.filter.contains` | 5 | 0.400 | `filesIn.filter.contains.assertTrue.(hasImport)+` |
| `trim.lowercase.warn` | 5 | 0.400 | `trim.lowercase.(warn)+` |
| `DoclingConfig.assertThatThrownBy.validateCriticalSettings` | 5 | 0.200 | `DoclingConfig.assertThatThrownBy.validateCriticalSettings.isInstanceOf.hasMessageContaining` |
| `defaultCapabilityId` | 4 | 0.250 | `defaultCapabilityId` |
| `clearAllMocks` | 4 | 0.250 | `clearAllMocks` |

## Next Steps

1. **Symbol coarsening**: Collapse specific symbols into categories (GETTER, SETTER, BUILDER, etc.) to reduce fragmentation
2. **Hierarchical grouping**: Group by k=1 first, then sub-group by k=2 within each group
3. **Merge similar contexts**: After initial grouping, merge contexts with similar SOREs (Bex's Reduce algorithm)
4. **Test on more codebases**: RAGSAK may not be representative — need more data
5. **Accept the limit**: Some methods are genuinely unique — 12% coverage may be the ceiling for this approach

## Files Generated

- `experiments/results/summary.json` — metrics for all strategies
- `experiments/results/*.json` — detailed results per strategy
- `experiments/context_eval.py` — experiment runner script
- `Papers/02-06-2026/context-design-analysis.md` — analysis of all options
- `Papers/02-06-2026/bex-research-summary.md` — summary of Bex's papers