tobjend
8028570ceb
feat: frequency threshold sweep — 0.01-0.20 across 4 codebases
...
Sweet spot: 0.01-0.05. At 0.01 RAGSAK gets 61 SOREs (26.1% cov) with
real conventions like warn.status.body.ErrorResponse. At 0.05 coverage
jumps to 45.4% but that pattern disappears. Flask dies at 0.15+.
Current default min_coverage=0.2 is too aggressive for most codebases.
2026-07-12 01:26:45 +02:00
tobjend
dd183f6241
feat: 4-codebase evaluation — conventions vs completions tradeoff
...
Added kotlinx.coroutines (1040 .kt files) and FastAPI (1129 .py files).
Key findings across 4 codebases:
- Coarsening trades per-package coverage for cross-package reach
- FastAPI: coverage IMPROVES (12.4% -> 20.7%), cross-pkg +15
- Coroutines: cross-pkg +15, real exception-handling patterns
- RAGSAK: cross-pkg +18, null-check conventions across 8 packages
- Flask: slight cross-pkg loss, rendering conventions still visible
The conventions vs completions tradeoff is real and measurable.
2026-07-12 01:14:09 +02:00
tobjend
8540947eec
docs: Round 6b results — Kotlin capture fix + minimal coarsening
2026-07-12 01:05:19 +02:00
tobjend
d65b78bdf1
fix: bare Kotlin captures + minimal coarsening (RETURN/IF/EXCEPTION/LOOP only)
...
- Added bare Kotlin captures to BEHAVIORAL_PREFIXES: conditional, exception, repeat, property, type
- Dropped variable (too noisy), kept only 4 high-signal categories in coarsen_token
- RAGSAK: 87 cross-package contexts (up from 69), real null-check patterns
- Flask: k=2 coverage 4.5% -> 15.8%, cross-package rendering conventions
2026-07-12 01:04:51 +02:00
tobjend
592974f039
feat: structural coarsening experiment — keeps function names raw, coarsens keywords
...
- coarsen_token() maps tree-sitter captures to categories (RETURN, IF, LOOP, etc.)
- Function calls kept as raw text (they ARE the behavioral content)
- Only ~5% structural tokens coarsened
- Results: Flask coverage 17.5% -> 28.8% (+11.3%), RAGSAK unchanged (95% calls)
- Cross-package shapes emerge: ('IF', 'KW', 'RETURN') in 4 Flask packages
2026-07-12 00:56:09 +02:00
tobjend
8e1c7f3767
docs: experiment log + cross-package analysis
...
- Full experiment history: context strategies, Reduce, scoring, GBNF, cross-package
- Documented failures: per-package sparsity, Reduce at wrong level, exact match rarity
- Designed next experiment: structural coarsening via tree-sitter categories
- Updated RESULTS.md with Round 5 findings and summary table
2026-07-12 00:45:21 +02:00
tobjend
b516b2985d
feat: implement Reduce algorithm (Algorithm 4, TODS 2010)
...
- bex/reduce.py: Faithful implementation of Reduce with support-weighted
SOA edit distance, adjunction, iterative merging, and Minimize
- experiments/context_eval.py: Multi-codebase support (RAGSAK + Flask),
Reduce experiments with thresholds 0.05-0.4
- tests/test_reduce.py: 24 tests covering all Reduce components
- Flask cloned to external_refs/flask for cross-validation
Results:
- RAGSAK: 12.0% coverage (First 3 symbols)
- Flask: 10.7% coverage (First 3 symbols)
- Reduce has minimal impact (1-2 merges per codebase at ε=0.3)
- Coverage ceiling appears to be ~10-12% for prefix-based grouping
2026-07-12 00:11:38 +02:00
tobjend
bbdfe93679
experiments: test all context strategies on RAGSAK — behavioral grouping wins
...
Key findings:
- File path grouping (structural) = baseline (0.6% coverage) — no improvement
- First-k-symbols grouping (behavioral) = 20× improvement (12% coverage at k=3)
- Two-dimensional (path + symbols) = worse than behavioral alone
- Winner: Option B k=3 — 47 SOREs, 12% coverage
Smart filters make experiments fast (<1s vs minutes):
- max_unique_ratio=0.85
- max_alphabet=20
- max_soa_edges=100
Preserved in experiments/results/ for future reference.
2026-07-11 23:40:21 +02:00
