tobi/grammar-inference-engine
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
134 commits 4 branches 0 tags 7.1 MiB
Commit graph

13 commits

Author SHA1 Message Date
tobjend
becbd82c56 feat: add post-hoc noise filtering for grammars
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
Details
ci/woodpecker/pr/woodpecker Pipeline failed
Details 
- Add filter_noise() to remove test/stdlib tokens from AST
- Add grammar_noise_ratio() to calculate noise fraction
- Integrate into _build_json_output() and _build_yaml_output()
- Precision improvement: 28-55% → 84-94% across all 3 codebases
- Grammar count unchanged (233) — filtering is post-hoc, preserves recall
 2026-07-13 02:23:37 +02:00
tobjend
8b3a454a15 feat: enable idregex_refine by default + AST rewrite of _count_optionals 
- idregex_refine=True default across all entry points + CLI
- Added --no-idregex-refine flag to CLI
- Rewrote _count_optionals from SORE string parser to AST walker
- Added _is_pure_bag() helper for bag detection
- Updated _should_try_idregex() to trigger on pure Plus(Alt) bags
- Updated tests for AST-based _count_optionals
- Added RAGSAK v4 experiment results
 2026-07-13 01:13:49 +02:00
tobjend
8460c3c862 experiments: round 20 v3 results (decompose default + cap fix) 
v3 runs for RAGSAK/fastapi/zod with decompose default-on and _COUNT_CAP
raised. Updated SUMMARY.md with before/after quality table.
 2026-07-13 01:13:48 +02:00
tobjend
1810a695d0 experiments: round 20 AST-migration verification across 3 languages 
Run RAGSAK (Kotlin), fastapi (Python), zod (TypeScript) with --slice
package --min-structure 0.5, verbose+timeout, results saved for later.

After _count_concat memoization fix: RAGSAK full run 54.9s -> 6.7s,
no tail stalls. RAGSAK: 121 packages (28 with grammar). fastapi/zod
complete in ~25s (dominated by arg-extraction on 2000+ method test
groups, not grammar explosion). Includes SUMMARY.md with before/after.
 2026-07-13 01:13:48 +02:00
tobjend
e62fffc6e0 feat: grammar_structure_score + min_structure filter 
Quantifies how structured a SORE is (0.0=flat bag, 1.0=fully structured).
Flat bags are CRX over-approximations — they list symbols without ordering.

With min_structure=0.2:
  Flask:   2 kept (was 5),  9 dropped
  RAGSAK: 10 kept (was 19), 114 dropped
  FastAPI: 47 kept (was 106), 95 dropped
  Total: 59 useful grammars, 218 noise removed

CLI: --min-structure 0.2 (default: 0, keep all)
 2026-07-12 03:02:50 +02:00
tobjend
3468813ec8 fix: validate SORE before returning, skip malformed grammars 
- Add validate_sore() to gbnf.py — checks parseability without converting
- _infer_group now validates grammar and returns skip_reason='malformed_grammar'
  for SOREs containing raw code (e.g. w_body=, sult=, (+,+:N+...)
- Results: 130 grammars, 130 GBNF OK, 0 GBNF FAIL
  - Flask: 5 OK, 0 FAIL, 3.4s
  - RAGSAK: 19 OK, 0 FAIL, 11 malformed, 12.9s
  - FastAPI: 106 OK, 0 FAIL, 6 malformed, 30.5s
 2026-07-12 02:56:13 +02:00
tobjend
bc7d3b6ca1 perf: iDRegEx opt-in, GBNF newline fix, OverflowError fix 
- iDRegEx now opt-in via --idregex flag (was running on every group,
  causing 55s+ on Flask alone — src/flask/json took 55s in iDRegEx)
- GBNF tokenizer strips newlines from literals (multi-line symbols)
- Fix OverflowError: lang_size_score produces huge ints for large
  disjunctions, format as string not float
- Flask: 2.7s (was 55s+), RAGSAK: 13s (was 74s)
 2026-07-12 02:52:05 +02:00
tobjend
8028570ceb feat: frequency threshold sweep — 0.01-0.20 across 4 codebases 
Sweet spot: 0.01-0.05. At 0.01 RAGSAK gets 61 SOREs (26.1% cov) with
real conventions like warn.status.body.ErrorResponse. At 0.05 coverage
jumps to 45.4% but that pattern disappears. Flask dies at 0.15+.

Current default min_coverage=0.2 is too aggressive for most codebases.
 2026-07-12 01:26:45 +02:00
tobjend
dd183f6241 feat: 4-codebase evaluation — conventions vs completions tradeoff 
Added kotlinx.coroutines (1040 .kt files) and FastAPI (1129 .py files).

Key findings across 4 codebases:
- Coarsening trades per-package coverage for cross-package reach
- FastAPI: coverage IMPROVES (12.4% -> 20.7%), cross-pkg +15
- Coroutines: cross-pkg +15, real exception-handling patterns
- RAGSAK: cross-pkg +18, null-check conventions across 8 packages
- Flask: slight cross-pkg loss, rendering conventions still visible

The conventions vs completions tradeoff is real and measurable.
 2026-07-12 01:14:09 +02:00
tobjend
d65b78bdf1 fix: bare Kotlin captures + minimal coarsening (RETURN/IF/EXCEPTION/LOOP only) 
- Added bare Kotlin captures to BEHAVIORAL_PREFIXES: conditional, exception, repeat, property, type
- Dropped variable (too noisy), kept only 4 high-signal categories in coarsen_token
- RAGSAK: 87 cross-package contexts (up from 69), real null-check patterns
- Flask: k=2 coverage 4.5% -> 15.8%, cross-package rendering conventions
 2026-07-12 01:04:51 +02:00
tobjend
592974f039 feat: structural coarsening experiment — keeps function names raw, coarsens keywords 
- coarsen_token() maps tree-sitter captures to categories (RETURN, IF, LOOP, etc.)
- Function calls kept as raw text (they ARE the behavioral content)
- Only ~5% structural tokens coarsened
- Results: Flask coverage 17.5% -> 28.8% (+11.3%), RAGSAK unchanged (95% calls)
- Cross-package shapes emerge: ('IF', 'KW', 'RETURN') in 4 Flask packages
 2026-07-12 00:56:09 +02:00
tobjend
b516b2985d feat: implement Reduce algorithm (Algorithm 4, TODS 2010) 
- bex/reduce.py: Faithful implementation of Reduce with support-weighted
  SOA edit distance, adjunction, iterative merging, and Minimize
- experiments/context_eval.py: Multi-codebase support (RAGSAK + Flask),
  Reduce experiments with thresholds 0.05-0.4
- tests/test_reduce.py: 24 tests covering all Reduce components
- Flask cloned to external_refs/flask for cross-validation

Results:
- RAGSAK: 12.0% coverage (First 3 symbols)
- Flask: 10.7% coverage (First 3 symbols)
- Reduce has minimal impact (1-2 merges per codebase at ε=0.3)
- Coverage ceiling appears to be ~10-12% for prefix-based grouping
 2026-07-12 00:11:38 +02:00
tobjend
bbdfe93679 experiments: test all context strategies on RAGSAK — behavioral grouping wins 
Key findings:
- File path grouping (structural) = baseline (0.6% coverage) — no improvement
- First-k-symbols grouping (behavioral) = 20× improvement (12% coverage at k=3)
- Two-dimensional (path + symbols) = worse than behavioral alone
- Winner: Option B k=3 — 47 SOREs, 12% coverage

Smart filters make experiments fast (<1s vs minutes):
- max_unique_ratio=0.85
- max_alphabet=20
- max_soa_edges=100

Preserved in experiments/results/ for future reference.
 2026-07-11 23:40:21 +02:00