grammar-inference-engine/experiments/RESULTS.md

Experiment Results — RAGSAK Context Strategies

Date: 2026-07-11 Codebase: RAGSAK (462 .kt files, 1594 methods, 137 packages)

Summary Table

Strategy	Contexts	SORE OK	Coverage	Notes
Baseline (package)	116	2	0.6%	Current approach — terrible
File path k=1	43	2	1.1%	Structural — no better than baseline
File path k=2	54	2	1.1%	Same — structural doesn't help
File path k=3	54	2	1.1%	Same — structural doesn't help
First 1 symbol	134	20	4.6%	Behavioral — 10× better
First 2 symbols	141	39	9.8%	Behavioral — 20× better
First 3 symbols	117	47	12.0%	Behavioral — winner
Two-dim (p1,s1)	151	24	6.3%	Hybrid — worse than behavioral alone
Two-dim (p1,s2)	119	33	8.7%	Hybrid — worse than behavioral alone
Two-dim (p2,s1)	146	24	6.3%	Hybrid — worse than behavioral alone
Two-dim (p2,s2)	117	33	8.7%	Hybrid — worse than behavioral alone
Return type heuristic	5	0	0.0%	Too coarse — useless

Key Findings

1. Behavioral grouping > Structural grouping

• File path strategies (A) produce same results as baseline — structural similarity doesn't predict behavioral similarity
• First-k-symbols strategies (B) produce 20× more SOREs — behavioral prefix is the right signal
• Winner: Option B, k=3 — 47 SOREs, 12% coverage

2. Two-dimensional doesn't help

• Adding file path to behavioral grouping reduces performance
• The structural component drags down the behavioral signal
• Conclusion: drop the structural dimension entirely

3. The successful SOREs reveal real patterns

Test patterns (most common):

• post.uri.exchange.expectStatus.(expectBody.jsonPath.isEqualTo)? — HTTP test flow
• runTest.post.uri.bodyValue.(ChatRequest|SessionChatRequest).exchange.expectStatus — test setup
• every.getJobStatus.get.uri.exchange.expectStatus.(expectBody.jsonPath.isEqualTo)? — polling pattern

Assertion patterns:

• assertEquals.(of.(assertFailsWith)?)+ — assertion chain
• filesIn.assertTrue.(hasImport)+ — file validation
• trim.lowercase.(warn)+ — string processing

Mock patterns:

• mockk.every.... — mock setup
• clearAllMocks — cleanup

CRUD patterns:

• info.deleteByJobId — delete operation
• info.deleteByKnowledgeBaseId — delete operation
• update.trimIndent.now.insertRow — update operation

4. Why coverage is still low (12%)

• Most methods have unique call patterns — they don't share a 3-symbol prefix with any other method
• The 3-symbol prefix is too specific for diverse codebases
• Need either:
  • More data (more codebases to train on)
  • Looser grouping (k=1 or k=2, but with merging)
  • Symbol coarsening (collapse specific symbols into categories)

5. Smart filters work well

• max_unique_ratio=0.85: Skips 60-80% of groups (too diverse)
• max_alphabet=20: Skips groups with too many unique symbols
• max_soa_edges=100: Skips complex SOAs that would be slow
• Result: Experiment runs in <1s instead of minutes

Successful SORE Examples (Top 10 by group size)

Context	Methods	Unique Ratio	SORE
ery {.stKnowledgeBases().gRequest(m	12	0.333	ery {.stKnowledgeBases().gRequest(m.(ckKnowledgeBase(re.ertEquals(Kn."k|eckKnowledgeBase(r.(sertEquals(K.("|(sertEquals(
mockk	11	0.091	mockk
filesIn.assertTrue.hasImport	7	0.286	filesIn.assertTrue.(hasImport)+
of	7	0.143	of
assertEquals.of.assertFailsWith	7	0.429	assertEquals.(of.(assertFailsWith)?)+
filesIn.filter.contains	5	0.400	filesIn.filter.contains.assertTrue.(hasImport)+
trim.lowercase.warn	5	0.400	trim.lowercase.(warn)+
DoclingConfig.assertThatThrownBy.validateCriticalSettings	5	0.200	DoclingConfig.assertThatThrownBy.validateCriticalSettings.isInstanceOf.hasMessageContaining
defaultCapabilityId	4	0.250	defaultCapabilityId
clearAllMocks	4	0.250	clearAllMocks

Next Steps

1. Symbol coarsening: Collapse specific symbols into categories (GETTER, SETTER, BUILDER, etc.) to reduce fragmentation
2. Hierarchical grouping: Group by k=1 first, then sub-group by k=2 within each group
3. Merge similar contexts: After initial grouping, merge contexts with similar SOREs (Bex's Reduce algorithm)
4. Test on more codebases: RAGSAK may not be representative — need more data
5. Accept the limit: Some methods are genuinely unique — 12% coverage may be the ceiling for this approach

Files Generated

• experiments/results/summary.json — metrics for all strategies
• experiments/results/*.json — detailed results per strategy
• experiments/context_eval.py — experiment runner script
• Papers/02-06-2026/context-design-analysis.md — analysis of all options
• Papers/02-06-2026/bex-research-summary.md — summary of Bex's papers