Commit graph

22 commits

Author SHA1 Message Date
tobjend
710da56916 drop kORE from default ensemble, add --kore flag to opt in
- Remove kORE from top-level imports, _ALGORITHMS, and infer_ensemble body
- Add include_kore=False parameter, runs kORE only when opted in
- Add --kore CLI flag threaded through all pipeline layers
- Remove koreinference from --prefer choices
- Keep kore.py module in repo for reference/future use
- Update tests: remove test_prefer_koreinference, update algorithm assertions
2026-07-04 02:58:07 +02:00
tobjend
e3ad256321 feat: add --slice package (per-directory grouping), --include/--exclude, --verbose, drop root group inference, run full ensemble on all groups 2026-07-04 02:43:03 +02:00
tobjend
4a506cd39b fix: add bare function fallback to CALL_PREFIXES, fix IMPORT_PATTERNS order, update pipeline diagram
- CALL_PREFIXES: add bare "function" as last fallback — Kotlin uses
  @function for both calls and definitions (no function.call capture)
- _find_arglist_node: remove template_string from arglist detection
- IMPORT_PATTERNS: require_relative before require (Ruby fix)
- pipeline-overview.txt: remove .gitignore, add min_coverage=0.8 + core/outlier
- DEFAULT_COVERAGE=0.8, frequency_filter at fixed 0.2
2026-07-04 01:28:35 +02:00
tobjend
eb2173442b fix: restore frequency_filter at 0.2, add dual 0.8 BEX coverage
- frequency_filter(min_coverage=0.2) strips rare symbols before clustering/inference
- DEFAULT_COVERAGE=0.8 passed to infer_ensemble for sequence-level core/outlier detection
- Both filters active: symbol-level (0.2) then BEX sequence-level (0.8)
- Fix seq_of_file tracking broken by filter creating new list objects
2026-07-04 01:13:48 +02:00
tobjend
e23922a1b7 feat: adaptive multi-assignment clustering; add ADRs 1-10
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
- Multi-assignment clustering (no greedy 'used' set)
- Adaptive ngram fallback (shrink when (other) > 60%)
- Add docs/adr/ with 10 architecture decision records
- Fix ADR 1 (query modification description)
- Fix ADR 3 (multi-assignment + adaptive shrink)
- Fix ADR 5 (import sort order clarification)
- Fix ADR 6 (remove Kotlin call_suffix references)
- New ADR 9 (adaptive clustering rationale)
- New ADR 10 (universal package mapping via relpath)
2026-07-03 22:58:09 +02:00
tobjend
fd574da53d feat: language-agnostic import/arg/package extraction
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
- Add _classify_arg_node, _find_arglist_node, _iterate_arg_nodes,
  extract_arg_info, _summarize_arg_info (no per-language branches)
- _find_arglist_node: drop call_suffix from fallback list
- _iterate_arg_nodes: generic named-child iterator only
- Add IMPORT_PATTERNS + _extract_imports (regex-based, all languages)
- Add _file_to_package / _top_packages (pure relpath from project root)
- Add --json output format for prompt injection
- analyze_clusters returns 4-tuple with meta (files, imports, arg_patterns, packages)
2026-07-03 22:41:04 +02:00
tobjend
73b94af959 feat: cluster methods by n-gram patterns before inference
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
- _extract_call_tokens filters to call-like captures only
- cluster_methods groups sequences by shared 3-gram call patterns
- analyze_clusters runs per-cluster CRX/iDRegEx/kORE inference
- Lists 17 convention clusters for RAGSAK test code
- iDRegEx and kOREInference now produce ordered grammars per cluster
- (other) cluster captures diverse conventions as CRX vocabulary
- min-cluster-size (default 3) and ngram-size (default 3) CLI flags
2026-07-03 21:41:46 +02:00
tobjend
2620b6e49f feat: method-level sequence extraction via child_by_field_name('body')
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
- _find_method_bodies uses tree-sitter's universal body field (9/10 grammars)
- Kotlin fallback: scan children for body-like types
- preprocess_by_method groups highlight captures by enclosing method body
- Returns per-method sequences for k-ORE ordering analysis
- analyze.py infer() now uses preprocess_by_method
- Document findings in ANALYSIS.md
- 97 tests pass
2026-07-03 21:34:25 +02:00
tobjend
0c7703f63b feat: universal tag-preprocessor orchestrator with frequency filter
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
- Rename bex/tag-preprocessor/ to bex/tag_preprocessor/ (valid Python package)
- Add analyze.py: scan_directory, frequency_filter, infer, analyze_directory
- Frequency filter removes symbols below min_coverage threshold (Phase 1.0 step 4)
- infer() wires preprocess → frequency_filter → ensemble (Phase 1.0 step 5)
- --dir CLI mode for running full pipeline on directories (Phase 1.0 step 6)
- 9 new tests (test_analyze.py), all 94 tests pass
2026-07-03 20:58:18 +02:00
tobjend
ee5ebc9eb4 feat: universal AST preprocessor with nvim-treesitter queries
- Replace old per-language query files with unified tag-preprocessor
- Universal preprocessor uses nvim-treesitter highlights.scm for 10 langs
- Resolves ; inherits: directives (ecma, jsx) for JS/TS
- ts-kotlin bundled query for Kotlin compatibility
- Strips unsupported #set! 3-arg predicates for jsx
- archive old community queries
2026-07-03 20:48:47 +02:00
tobjend
8d06ac2d52 chore: add downloaded community queries for kotlin + typescript
Kotlin (fwcd/tree-sitter-kotlin) and TypeScript packages
don't bundle queries. These are downloaded from upstream repos.
2026-07-03 18:05:19 +02:00
tobjend
036a84cc76 docs: add min_coverage to MCP tool + README, include core in output
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
ci/woodpecker/pr/woodpecker Pipeline was successful
2026-07-01 15:16:24 +02:00
tobjend
9045769d57 feat: core+outlier analysis via min_coverage parameter, 6 new tests
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
ci/woodpecker/pr/woodpecker Pipeline was successful
2026-07-01 15:09:10 +02:00
tobjend
edd6d9d4dd feat: implement kOREInference (Algorithm 4) with MDL scoring, add to ensemble, 79 tests
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
ci/woodpecker/pr/woodpecker Pipeline was successful
2026-07-01 14:50:09 +02:00
tobjend
dc559a4aee fix badge position; purge remaining German user-reference comments
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
2026-07-01 13:28:55 +02:00
tobjend
6b60e0131f clean up agent-betraying comments; fix stale test names
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
2026-07-01 13:26:03 +02:00
tobjend
b8cc40177c remove redundant infer_grammar tool; update docs to single-tool MCP 2026-07-01 13:15:19 +02:00
tobjend
6bf7a681ce purge make_charts.py, examples/, full-text papers, blog_post (moved to ~/Desktop/kesai/); translate German CLI to English 2026-07-01 11:28:42 +02:00
tobjend
d7477344a6 move format-specific adapters to examples/, purge format-specific MCP tools 2026-07-01 10:36:14 +02:00
tobjend
0e2aec582b Grammar inference engine: CRX + iDRegEx ensemble with MDL scoring, MCP server, showcase, and blog post
- Ensemble inference (infer_ensemble) runs both CRX and iDRegEx, picks best by MDL
- CRX: CRX algorithm for wide coverage (accepts all sequences, large vocabulary)
- iDRegEx: iDRegEx for minimal core grammar (tightest common pattern)
- MDL scoring: fixed model_cost to count alphabet symbol occurrences, fixed dispatch order in _count_words_fast
- Fixed _match_tokens: rewritten as _match_possible with proper backtracking
- Fixed _parse_parts disjunction: children use _parse_flat_symbol to avoid dot-splitting
- MCP server: infer_best_grammar and infer_grammar tools
- Added prefer parameter (crx/idregex) to skip ensemble
- 28 passing tests
- SHOWCASE.md with Geerlingguy Galaxy demonstration
- blog_post.md with full technical deep-dive
2026-07-01 09:51:41 +02:00
tobjend
adc52c99ec Add MCP server: grammar inference via FastMCP
- bex/mcp_server.py: FastMCP server with 3 tools:
  * infer_grammar(sequences, method='crx'|'idregex')
  * infer_yaml_grammar(yaml_dir, pattern, method)
  * infer_ansible_role_grammar(roles_dir)
- pyproject.toml: add bex-mcp console_scripts entry point
2026-07-01 08:03:10 +02:00
tobjend
7c00c6713d Initial commit: BEX-based grammar inference engine
- CRX: direct CHARE inference (Algorithm 7, TODS 2010)
- iDRegEx: k-ORE inference (Algorithm 4, arXiv 2010)
- RWR₀: SORE repair (Algorithm 6, TODS 2010)
- rwr²: k-ORE extraction (Algorithm 3, arXiv 2010)
- SOA, k-OA, iKoa, 2T-INF, Baum-Welch
- Ansible role grammar adapter
- Generic YAML key-path converter
- 28 tests, all passing
2026-07-01 08:01:16 +02:00