73 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
tobjend	5906adfc95	parallelize preprocessing across files via ProcessPoolExecutor ... Replace serial file-read + tree-sitter parse loop with _preprocess_files() using ProcessPoolExecutor. Module-level _preprocess_file() for pickling. Covers all three callers: analyze_clusters, analyze_by_package, infer.	2026-07-04 02:58:47 +02:00
tobjend	710da56916	drop kORE from default ensemble, add --kore flag to opt in ... - Remove kORE from top-level imports, _ALGORITHMS, and infer_ensemble body - Add include_kore=False parameter, runs kORE only when opted in - Add --kore CLI flag threaded through all pipeline layers - Remove koreinference from --prefer choices - Keep kore.py module in repo for reference/future use - Update tests: remove test_prefer_koreinference, update algorithm assertions	2026-07-04 02:58:07 +02:00
tobjend	e3ad256321	feat: add --slice package (per-directory grouping), --include/--exclude, --verbose, drop root group inference, run full ensemble on all groups	2026-07-04 02:43:03 +02:00
tobjend	4a506cd39b	fix: add bare function fallback to CALL_PREFIXES, fix IMPORT_PATTERNS order, update pipeline diagram ... - CALL_PREFIXES: add bare "function" as last fallback — Kotlin uses @function for both calls and definitions (no function.call capture) - _find_arglist_node: remove template_string from arglist detection - IMPORT_PATTERNS: require_relative before require (Ruby fix) - pipeline-overview.txt: remove .gitignore, add min_coverage=0.8 + core/outlier - DEFAULT_COVERAGE=0.8, frequency_filter at fixed 0.2	2026-07-04 01:28:35 +02:00
tobjend	eb2173442b	fix: restore frequency_filter at 0.2, add dual 0.8 BEX coverage ... - frequency_filter(min_coverage=0.2) strips rare symbols before clustering/inference - DEFAULT_COVERAGE=0.8 passed to infer_ensemble for sequence-level core/outlier detection - Both filters active: symbol-level (0.2) then BEX sequence-level (0.8) - Fix seq_of_file tracking broken by filter creating new list objects	2026-07-04 01:13:48 +02:00
tobjend	55d21af48e	docs: pipeline overview ASCII diagram	2026-07-03 22:59:51 +02:00
tobjend	e23922a1b7	feat: adaptive multi-assignment clustering; add ADRs 1-10 ... - Multi-assignment clustering (no greedy 'used' set) - Adaptive ngram fallback (shrink when (other) > 60%) - Add docs/adr/ with 10 architecture decision records - Fix ADR 1 (query modification description) - Fix ADR 3 (multi-assignment + adaptive shrink) - Fix ADR 5 (import sort order clarification) - Fix ADR 6 (remove Kotlin call_suffix references) - New ADR 9 (adaptive clustering rationale) - New ADR 10 (universal package mapping via relpath)	2026-07-03 22:58:09 +02:00
tobjend	fd574da53d	feat: language-agnostic import/arg/package extraction ... - Add _classify_arg_node, _find_arglist_node, _iterate_arg_nodes, extract_arg_info, _summarize_arg_info (no per-language branches) - _find_arglist_node: drop call_suffix from fallback list - _iterate_arg_nodes: generic named-child iterator only - Add IMPORT_PATTERNS + _extract_imports (regex-based, all languages) - Add _file_to_package / _top_packages (pure relpath from project root) - Add --json output format for prompt injection - analyze_clusters returns 4-tuple with meta (files, imports, arg_patterns, packages)	2026-07-03 22:41:04 +02:00
tobjend	73b94af959	feat: cluster methods by n-gram patterns before inference ... - _extract_call_tokens filters to call-like captures only - cluster_methods groups sequences by shared 3-gram call patterns - analyze_clusters runs per-cluster CRX/iDRegEx/kORE inference - Lists 17 convention clusters for RAGSAK test code - iDRegEx and kOREInference now produce ordered grammars per cluster - (other) cluster captures diverse conventions as CRX vocabulary - min-cluster-size (default 3) and ngram-size (default 3) CLI flags	2026-07-03 21:41:46 +02:00
tobjend	2620b6e49f	feat: method-level sequence extraction via child_by_field_name('body') ... - _find_method_bodies uses tree-sitter's universal body field (9/10 grammars) - Kotlin fallback: scan children for body-like types - preprocess_by_method groups highlight captures by enclosing method body - Returns per-method sequences for k-ORE ordering analysis - analyze.py infer() now uses preprocess_by_method - Document findings in ANALYSIS.md - 97 tests pass	2026-07-03 21:34:25 +02:00
tobjend	0c7703f63b	feat: universal tag-preprocessor orchestrator with frequency filter ... - Rename bex/tag-preprocessor/ to bex/tag_preprocessor/ (valid Python package) - Add analyze.py: scan_directory, frequency_filter, infer, analyze_directory - Frequency filter removes symbols below min_coverage threshold (Phase 1.0 step 4) - infer() wires preprocess → frequency_filter → ensemble (Phase 1.0 step 5) - --dir CLI mode for running full pipeline on directories (Phase 1.0 step 6) - 9 new tests (test_analyze.py), all 94 tests pass	2026-07-03 20:58:18 +02:00
tobjend	8bda174293	docs: document Kotlin and JS/TS special cases in analysis	2026-07-03 20:54:03 +02:00
tobjend	70300ff917	chore: add Gemini session chat logs as reference	2026-07-03 20:48:50 +02:00
tobjend	ee5ebc9eb4	feat: universal AST preprocessor with nvim-treesitter queries ... - Replace old per-language query files with unified tag-preprocessor - Universal preprocessor uses nvim-treesitter highlights.scm for 10 langs - Resolves ; inherits: directives (ecma, jsx) for JS/TS - ts-kotlin bundled query for Kotlin compatibility - Strips unsupported #set! 3-arg predicates for jsx - archive old community queries	2026-07-03 20:48:47 +02:00
tobjend	986eaa7f83	feat: add community tags findings audit and kotlin tags query	2026-07-03 18:40:53 +02:00
tobjend	8d06ac2d52	chore: add downloaded community queries for kotlin + typescript ... Kotlin (fwcd/tree-sitter-kotlin) and TypeScript packages don't bundle queries. These are downloaded from upstream repos.	2026-07-03 18:05:19 +02:00
tobjend	069c63f2c8	feat: inspect TreeSitter tag queries for Dervish universal AST extraction ... - Fetch TAGS_QUERY and HIGHLIGHTS_QUERY from 8 official TreeSitter language packages - Save all curated queries as .scm files - Analysis confirms universal query approach is viable - TreeSitter silently ignores non-existent node types per language - Add full conversation references (gemini chat markdown files) - Add comprehensive conversation summary with roadmap	2026-07-03 17:46:33 +02:00
tobi	28f5f897d5	Update README.md	2026-07-02 18:17:16 +00:00
tobi	d74b36e563	Update README.md	2026-07-02 18:14:48 +00:00
tobi	197a0a3c22	Update README.md	2026-07-02 18:08:42 +00:00
tobi	136ae08fe3	Update README.md	2026-07-02 18:06:38 +00:00
tobi	ea8e2f1db7	Update README.md	2026-07-02 16:33:51 +00:00
tobi	16cbff61a8	Update README.md	2026-07-02 16:33:02 +00:00
tobjend	b037098730	fix: regenerate chart with Comic Neue + path.sketch xkcd style	2026-07-01 16:14:24 +02:00
tobi	d2d57bc431	Merge pull request 'feat: kOREInference — Algorithm 4 iDRegEx with MDL scoring + ensemble integration' (#1) from feature/kore-inference into main	2026-07-01 14:08:18 +00:00
tobjend	90d8c69aa7	fix: broken y-axis bar chart for readability	2026-07-01 16:05:49 +02:00
tobjend	2562519718	fix: grouped bar chart with Without Dervish vs With Dervish	2026-07-01 16:04:55 +02:00
tobjend	ce4088e705	fix: chart bars use Dervish purple #853E91	2026-07-01 16:03:20 +02:00
tobjend	d96e99d84f	docs: xkcd-style chart	2026-07-01 16:02:14 +02:00
tobjend	b34e39d4b9	feat: replace single-chart Helm with cross-project convention (15 charts, 6 publishers)	2026-07-01 16:00:04 +02:00
tobjend	fbdb39bf7c	docs: syntax highlighting tags in SHOWCASE.md	2026-07-01 15:48:36 +02:00
tobjend	bc81bbdce1	docs: extra blank line above coffee button	2026-07-01 15:42:14 +02:00
tobjend	bb193d000f	docs: bump logo 180→216px	2026-07-01 15:41:55 +02:00
tobjend	4969819dbb	docs: center logo, move coffee button below nav links	2026-07-01 15:41:35 +02:00
tobjend	a70024397f	chore: ignore examples/	2026-07-01 15:40:02 +02:00
tobjend	929a50c95d	docs: add Buy Me a Coffee button	2026-07-01 15:37:09 +02:00
tobjend	0be1a7fd79	docs: badges row, nav links, language tags on code blocks	2026-07-01 15:24:00 +02:00
tobjend	5e0674bf77	docs: fix Go lint description (both optional), format outliers in SHOWCASE usage	2026-07-01 15:21:55 +02:00
tobjend	0886e5f3bc	docs: update README and SHOWCASE for kOREInference + core/outlier analysis	2026-07-01 15:18:32 +02:00
tobjend	036a84cc76	docs: add min_coverage to MCP tool + README, include core in output	2026-07-01 15:16:24 +02:00
tobjend	9045769d57	feat: core+outlier analysis via min_coverage parameter, 6 new tests	2026-07-01 15:09:10 +02:00
tobjend	edd6d9d4dd	feat: implement kOREInference (Algorithm 4) with MDL scoring, add to ensemble, 79 tests	2026-07-01 14:50:09 +02:00
tobjend	dc559a4aee	fix badge position; purge remaining German user-reference comments	2026-07-01 13:28:55 +02:00
tobjend	6b60e0131f	clean up agent-betraying comments; fix stale test names	2026-07-01 13:26:03 +02:00
tobjend	dcfd8b054e	remove pull:true (image cached now)	2026-07-01 13:23:29 +02:00
tobjend	79c6a0734c	force pull of dervish-ci:latest	2026-07-01 13:22:39 +02:00
tobjend	701c5649c7	use pre-built dervish-ci image with pytest; no pip install in pipeline	2026-07-01 13:20:56 +02:00
tobjend	6191427983	fix pipeline: editable install; add CI badge to README	2026-07-01 13:19:11 +02:00
tobjend	416527c209	add Woodpecker CI: run tests on push/pull/manual	2026-07-01 13:15:53 +02:00
tobjend	b8cc40177c	remove redundant infer_grammar tool; update docs to single-tool MCP	2026-07-01 13:15:19 +02:00