Commit graph

9 commits

Author SHA1 Message Date
tobjend
ea6cac53e3 wip: AST foundation — grammar.py, expr.py→AST, soa.py→AST labels 2026-07-13 01:13:48 +02:00
tobjend
bc7d3b6ca1 perf: iDRegEx opt-in, GBNF newline fix, OverflowError fix
- iDRegEx now opt-in via --idregex flag (was running on every group,
  causing 55s+ on Flask alone — src/flask/json took 55s in iDRegEx)
- GBNF tokenizer strips newlines from literals (multi-line symbols)
- Fix OverflowError: lang_size_score produces huge ints for large
  disjunctions, format as string not float
- Flask: 2.7s (was 55s+), RAGSAK: 13s (was 74s)
2026-07-12 02:52:05 +02:00
tobjend
dfb56a083a WIP: language size scoring + diversity threshold (step 1 pending) 2026-07-11 22:56:42 +02:00
tobjend
ce6521ad5e Revert "parallelize kORE outer (k, n) trials via ProcessPoolExecutor"
This reverts commit 0b5b0e623b.
2026-07-11 20:36:48 +02:00
tobjend
0b5b0e623b parallelize kORE outer (k, n) trials via ProcessPoolExecutor
Add _kore_trial() module-level worker for pickling. infer() accepts
n_workers param — >1 runs all (k x N) trials concurrently instead
of serial. Default 1 preserves existing behavior.

Thread n_workers through _run_kore() and infer_ensemble() to support
--kore flag with parallel kORE inference.
2026-07-04 03:04:54 +02:00
tobjend
710da56916 drop kORE from default ensemble, add --kore flag to opt in
- Remove kORE from top-level imports, _ALGORITHMS, and infer_ensemble body
- Add include_kore=False parameter, runs kORE only when opted in
- Add --kore CLI flag threaded through all pipeline layers
- Remove koreinference from --prefer choices
- Keep kore.py module in repo for reference/future use
- Update tests: remove test_prefer_koreinference, update algorithm assertions
2026-07-04 02:58:07 +02:00
tobjend
9045769d57 feat: core+outlier analysis via min_coverage parameter, 6 new tests
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
ci/woodpecker/pr/woodpecker Pipeline was successful
2026-07-01 15:09:10 +02:00
tobjend
edd6d9d4dd feat: implement kOREInference (Algorithm 4) with MDL scoring, add to ensemble, 79 tests
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
ci/woodpecker/pr/woodpecker Pipeline was successful
2026-07-01 14:50:09 +02:00
tobjend
0e2aec582b Grammar inference engine: CRX + iDRegEx ensemble with MDL scoring, MCP server, showcase, and blog post
- Ensemble inference (infer_ensemble) runs both CRX and iDRegEx, picks best by MDL
- CRX: CRX algorithm for wide coverage (accepts all sequences, large vocabulary)
- iDRegEx: iDRegEx for minimal core grammar (tightest common pattern)
- MDL scoring: fixed model_cost to count alphabet symbol occurrences, fixed dispatch order in _count_words_fast
- Fixed _match_tokens: rewritten as _match_possible with proper backtracking
- Fixed _parse_parts disjunction: children use _parse_flat_symbol to avoid dot-splitting
- MCP server: infer_best_grammar and infer_grammar tools
- Added prefer parameter (crx/idregex) to skip ensemble
- 28 passing tests
- SHOWCASE.md with Geerlingguy Galaxy demonstration
- blog_post.md with full technical deep-dive
2026-07-01 09:51:41 +02:00