diff --git a/docs/adr/0001-use-nvim-treesitter-highlights-scm.md b/docs/adr/0001-use-nvim-treesitter-highlights-scm.md new file mode 100644 index 0000000..ed622bf --- /dev/null +++ b/docs/adr/0001-use-nvim-treesitter-highlights-scm.md @@ -0,0 +1,41 @@ +# 1. Use nvim-treesitter `highlights.scm` as behavioral capture source + +**Date:** 2026-07-03 + +**Status:** Accepted + +## Context + +We need a universal source of behavioral code tokens (function calls, references, definitions) across multiple programming languages. Options: + +- **`tags.scm`** (nvim-treesitter): Purpose-built for symbol tagging. Covers definitions and references. +- **`highlights.scm`** (nvim-treesitter): Built for syntax highlighting. Covers a wider range of tokens including keywords, operators, and built-ins. +- **Custom per-language queries**: Write and maintain our own query files for each language. + +We need tokens that represent *what the code does at runtime* — not just structure. + +## Decision + +Use nvim-treesitter `highlights.scm` as the capture source for all 10 languages. + +We filter captures to a `BEHAVIORAL_PREFIXES` set: `definition.`, `reference.`, `keyword.`, `function`, `attribute`, `constructor`, `label`, `type.definition`, `module`. + +For Kotlin, use the `ts-kotlin` (fwcd fork) bundled `highlights.scm` instead of nvim-treesitter's, because nvim-treesitter's Kotlin query references a duplicate `annotation` node type that doesn't exist in the grammar. + +## Consequences + +**Positive:** +- `highlights.scm` covers 4 out of 5 behavioral capture types that `tags.scm` misses, across all 10 languages. +- No per-language custom code or adapters needed. +- Community-maintained queries stay fresh with language evolution. +- Same query files work for both parsing and tokenizing. + +**Negative:** +- `highlights.scm` includes non-behavioral captures (comments, punctuation, operators) — we filter these out. +- Two `jsx` captures use `#set!` with 3 arguments, which `py-tree-sitter` 0.26 rejects. Strip these 2 patterns. +- Kotlin requires a separate grammar package (`ts-kotlin`) because the nvim-treesitter Kotlin grammar is incompatible. + +## Alternatives Considered + +- **`tags.scm`**: Cleaner signal-to-noise ratio, but misses `function`, `attribute`, `constructor`, `module` captures that are essential for behavioral understanding. +- **Custom queries**: Would give full control but require per-language maintenance — violates our universal-preprocessor constraint. diff --git a/docs/adr/0002-language-agnostic-method-extraction.md b/docs/adr/0002-language-agnostic-method-extraction.md new file mode 100644 index 0000000..48b2762 --- /dev/null +++ b/docs/adr/0002-language-agnostic-method-extraction.md @@ -0,0 +1,50 @@ +# 2. Language-agnostic method extraction via `child_by_field_name("body")` + +**Date:** 2026-07-03 + +**Status:** Accepted + +## Context + +To analyze method-level behavioral conventions, we must extract the body of each function/method from the AST. The standard tree-sitter approach is `node.child_by_field_name("body")`, but this named field is not universal across all language grammars. + +We need one code path that works for all 10 supported languages without per-language branches. + +## Decision + +Use `node.child_by_field_name("body")` as the primary extraction method. When it returns `None`, fall back to scanning the node's children for any child with a type containing `body`, `block`, or `compound_statement`. + +Parent nodes are further filtered to only include nodes whose type contains `function` or `method` — avoiding class bodies, loop bodies, and conditional blocks. + +This logic lives in `_find_method_bodies()` in `code.py`: + +```python +def walk(node): + body = node.child_by_field_name("body") + if not body: + for child in node.children: + ctype = child.type.lower() + if "body" in ctype or "block" in ctype or ctype == "compound_statement": + body = child; break + if body: + ptype = node.type.lower() + if "function" in ptype or "method" in ptype: + bodies.append(body) + for child in node.children: walk(child) +``` + +## Consequences + +**Positive:** +- Works for 9/10 grammars via `child_by_field_name("body")` alone (Python, Go, Rust, JS, TS, Ruby, Java, C, C++). +- Kotlin fallback works because the fwcd Kotlin grammar uses `function_body` as a child node type. +- Zero per-language case analysis — just pattern matching on type strings. + +**Negative:** +- Fallback relies on string matching (`"body" in ctype`) which could produce false positives if future grammar versions introduce new body-like types. +- C/C++ `function_definition` uses `declarator` field for the function name, not `name` — affects name extraction but not body extraction. + +## Alternatives Considered + +- **Grammar-specific field names**: Map each language to its body field name. Rejected because it creates a maintenance burden and violates the zero-adapters constraint. +- **Top-down sibling traversal**: Walk from node start to next sibling to find the body. Fragile across grammars with different compound statement structures. diff --git a/docs/adr/0003-method-level-n-gram-clustering.md b/docs/adr/0003-method-level-n-gram-clustering.md new file mode 100644 index 0000000..27fe644 --- /dev/null +++ b/docs/adr/0003-method-level-n-gram-clustering.md @@ -0,0 +1,44 @@ +# 3. Method-level n-gram clustering before inference + +**Date:** 2026-07-03 + +**Status:** Accepted + +## Context + +The BEX ensemble (CRX, iDRegEx, kORE) infers grammars from sets of symbol sequences. When we run inference on *all methods in a codebase*, the sequences are too diverse — each file has different conventions, and the ensemble produces only a flat vocabulary bag like `(any+assertEquals+assertTrue+every+listOf+verify)+`. + +This doesn't capture the *ordering* of calls or the distinct methodological styles present in the codebase. + +## Decision + +Group methods by shared n-gram (default: 3-gram) call patterns *before* running inference. + +Pipeline: `preprocess_by_method` → `frequency_filter` → `cluster_methods` → per-cluster `infer_ensemble` + +The clustering algorithm: +1. Extract call tokens from each method sequence (filter to `function`, `reference.call`, `reference.class` captures). +2. Build an n-gram index: for each method, for each sliding window of size N, record the n-gram. +3. Sort n-grams by frequency (most shared first). +4. Assign each method to the largest matching cluster, then remove assigned methods. +5. Remaining unclustered methods go to `(other)`. + +This produces 10-30 clusters for a typical test suite, each with 3-100+ methods sharing a call-order pattern. + +## Consequences + +**Positive:** +- iDRegEx and kOREInference now produce ordered grammars (e.g. `every+.assertEquals.verify+.any?`) because small, focused clusters have enough signal. +- Each cluster reveals a distinct *methodological style* in the codebase (mockist TDD vs data-driven testing vs pure assertion). +- The `(other)` cluster still captures the full vocabulary bag for diverse methods. + +**Negative:** +- Clustering adds a hyperparameter (`ngram_size`, default 3). Wrong value can produce too many tiny clusters or one giant cluster. +- `min_cluster_size` (default 3) filters out tiny but potentially interesting patterns. +- Methods in `(other)` never get ordered grammar inference — just vocabulary. + +## Alternatives Considered + +- **Infer on all methods (no clustering)**: Produces flat vocabulary only. CRX works at 100% coverage, but iDRegEx and kORE fail on diverse inputs. +- **Infer per file**: Too fine-grained — most files have 1-5 methods, not enough for inference. +- **Infer per directory**: Better, but directories mix unrelated conventions (setup/teardown vs actual test logic). diff --git a/docs/adr/0004-frequency-filter-with-min-coverage.md b/docs/adr/0004-frequency-filter-with-min-coverage.md new file mode 100644 index 0000000..282ae2b --- /dev/null +++ b/docs/adr/0004-frequency-filter-with-min-coverage.md @@ -0,0 +1,51 @@ +# 4. Frequency filter with `min_coverage` threshold + +**Date:** 2026-07-03 + +**Status:** Accepted + +## Context + +A raw method sequence can contain hundreds of unique call tokens, many of which appear in only 1-2 methods. These rare symbols are noise — they inflate grammar size, confuse the inference algorithm, and dilute the signal of common conventions. + +We need a principled way to discard rare symbols while keeping the behavioral patterns that define the codebase. + +## Decision + +Apply a frequency filter *before* clustering: remove any symbol that appears in fewer than `min_coverage` fraction of method sequences. + +Default threshold: `0.2` (20% of methods must contain the symbol). + +Filtering is done by `frequency_filter()` in `analyze.py`: +```python +n_files = len(sequences) +threshold = max(1, int(n_files * min_coverage)) +symbol_file_count = Counter() +for seq in sequences: + seen = set() + for _, text, _ in seq: + symbol_file_count[text] += 1 if text not in seen else 0 + seen.add(text) +keep = {text for text, count in symbol_file_count.items() if count >= threshold} +``` + +A symbol is counted once per file (not once per occurrence) to avoid skew from files that repeat the same symbol many times. + +## Consequences + +**Positive:** +- Removes noise before clustering, improving cluster quality. +- Prevents rare one-off function calls from creating spurious n-gram matches. +- The `common` vs `other` cluster distinction is sharper because the filter removes tokens that would appear in neither. + +**Negative:** +- With large codebases (600+ methods), 20% threshold may be too aggressive — a symbol needs 120+ occurrences to survive. + - Mitigation: `--min-coverage` flag lets users tune per codebase. + - Production code often needs lower values (`0.05`) because methods are more diverse than tests. +- The threshold is relative, not absolute. A 3-file project keeps anything in 1+ files (`max(1, 3*0.2)` = 1). + +## Alternatives Considered + +- **No filter**: CRX produces `(a+b+c+d+e+f+g+h+i+j+...)+` — the vocabulary is too large to be informative. +- **Absolute threshold**: `min_occurrences=5`. Doesn't scale — works for small projects, wrong for large ones. +- **TF-IDF style weighting**: More sophisticated but adds complexity. The simple coverage filter works well in practice. diff --git a/docs/adr/0005-import-extraction-per-cluster.md b/docs/adr/0005-import-extraction-per-cluster.md new file mode 100644 index 0000000..8fab5e9 --- /dev/null +++ b/docs/adr/0005-import-extraction-per-cluster.md @@ -0,0 +1,45 @@ +# 5. Import extraction per cluster + +**Date:** 2026-07-03 + +**Status:** Accepted + +## Context + +An LLM prompted with a behavioral convention like `every → assertEquals → verify` still needs to know *which imports to use*. Without imports, it will guess the wrong library — writing `from unittest.mock import patch` instead of `import io.mockk.every`, or importing from `jest` instead of `vitest`. + +Imports are the bridge between abstract conventions and actionable code. + +## Decision + +For each cluster, scan the source files whose methods belong to that cluster and extract all unique import lines. + +Language-agnostic approach: match lines against common import patterns: +- `import ...` (Java, Kotlin, Python, Go, JS/TS) +- `from ... import ...` (Python) +- `require ...` / `require_relative ...` (Ruby, JS) +- `#include ...` (C/C++) +- `use ...` (Rust) +- `include ...` (Ruby) + +Scan the first 200 lines of each file (imports are always at the top), deduplicate across files, and sort the result. + +File-to-cluster mapping is preserved by tracking `(file_path, sequence)` pairs through the pipeline. After `frequency_filter` (which preserves order and count), we use object identity to map each clustered sequence back to its source file. + +## Consequences + +**Positive:** +- Each cluster shows exact import lines used by its methods. +- An LLM can copy these directly — no guessing. +- Reveals *library choice conventions*: `kotlin.test.*` vs `org.junit.jupiter.api.*`, `io.mockk.coEvery` vs `io.mockk.every`. + +**Negative:** +- Import scanning re-reads files (second pass). Negligible cost since files are small and OS-cached. +- 200-line scan limit might miss imports in files with very long license headers. +- Lines containing `import` in prose (comments, strings) may produce false positives — rare in practice. + +## Alternatives Considered + +- **Single global import list**: Simpler but useless — conflates imports from unrelated clusters. +- **No imports**: LLM must guess. Leads to wrong imports and broken code. +- **Per-file imports (not per-cluster)**: Too granular — mixes test imports with production imports in the same file. diff --git a/docs/adr/0006-argument-pattern-extraction.md b/docs/adr/0006-argument-pattern-extraction.md new file mode 100644 index 0000000..cf06cf9 --- /dev/null +++ b/docs/adr/0006-argument-pattern-extraction.md @@ -0,0 +1,53 @@ +# 6. Argument pattern extraction via AST node classification + +**Date:** 2026-07-03 + +**Status:** Accepted + +## Context + +A behavioral token like `assertEquals` tells the LLM that the function is called, but not *how*. Two codebases both use `assertEquals` — one writes `assertEquals(expected, actual)` and the other writes `assertEquals(actual, expected)` with swapped argument order. An LLM guessing the wrong order writes broken tests. + +The highlights.scm captures tell us *that* a function is called. We need the argument *structure* — number of arguments, their types, and the common patterns. + +## Decision + +For each behavioral capture node, walk up to its parent `call_expression` (or equivalent), find the argument list node, and classify each argument by structural role. + +Argument classification is language-agnostic: + +| Classification | Matches | +|---|---| +| `lit` | string, number, boolean, null | +| `var` | identifiers, names | +| `call` | nested call expressions, method invocations | +| `lambda` | lambda expressions, blocks, do-blocks | +| `kwarg` | keyword/named arguments | +| `expr` | binary/unary/ternary/operator expressions | +| `template` | string interpolation, template literals | +| `other` | anything else (fallback) | + +Argument list node detection uses a tiered approach: +1. `child_by_field_name("arguments")` — works for Python, JS, TS, Java, Go, Ruby, Rust. +2. Fallback: scan children for `argument_list`, `arguments`, `call_suffix` (Kotlin), `template_string` (JS tagged templates). +3. Kotlin special case: `call_suffix` may contain a direct `lambda_expression` child (for `every { ... }` syntax) or a `value_arguments → value_argument` chain (for `func(a, b)` syntax). + +Results are aggregated per cluster into a summary showing min/max/common arg counts and the top argument-type patterns. + +## Consequences + +**Positive:** +- Reveals argument ordering conventions: `assertEquals: n=2 [lit,var]` means expected-first. +- Reveals calling convention variance: `verify: n=0 [] | n=1 [lambda] | n=1 [var]` means three styles coexist. +- No per-language branches — the tiered arglist detection handles all 10 grammars. + +**Negative:** +- `kwarg` detection only covers named arguments, not default values or spread operators. +- Nested destructuring patterns fall into `other` bucket — no granularity for complex argument shapes. +- `other` is a catch-all that can hide meaningful distinctions we haven't classified yet. + +## Alternatives Considered + +- **Extract raw argument text**: Language-agnostic but fragile — variable names change per test, producing high variance and low signal. +- **No argument extraction**: The LLM sees `assertEquals` but doesn't know argument order. Leads to wrong code. +- **Per-language argument extractors**: Would be more precise but violate the zero-adapters constraint. diff --git a/docs/adr/0007-json-output-for-llm-prompt-injection.md b/docs/adr/0007-json-output-for-llm-prompt-injection.md new file mode 100644 index 0000000..0b2dccb --- /dev/null +++ b/docs/adr/0007-json-output-for-llm-prompt-injection.md @@ -0,0 +1,63 @@ +# 7. JSON output for LLM prompt injection + +**Date:** 2026-07-03 + +**Status:** Accepted + +## Context + +The text table output is human-readable but not directly usable by an LLM. To use Dervish conventions in another agent or coding session, the output must be parsed, reformatted, and injected into a prompt — an extra friction step. + +An LLM consuming conventions needs: +- Structured data it can read directly (no parsing). +- All metadata per convention (grammar, imports, args, files, packages). +- Compact enough to fit in context without overflow. + +## Decision + +Add a `--json` flag that outputs a structured JSON array instead of the text table. + +JSON structure: +```json +[{ + "language": ".kt", + "conventions": [{ + "label": "every → assertEquals → verify", + "method_count": 16, + "algorithm": "CRX", + "grammar": "every+.assertEquals.verify+.any?", + "mdl_score": 8.64, + "imports": ["import io.mockk.every", "..."], + "packages": ["eu/corentic/springrag/agent/capability"], + "arg_patterns": { + "assertEquals": { + "occurrences": 42, + "arg_count": {"min": 2, "max": 3, "common": 2}, + "patterns": [{"count": 30, "args": 2, "types": ["lit", "var"]}] + } + } + }], + "total_methods": 665 +}] +``` + +Also accepts `--format json` and `--format text` for explicit control. + +## Consequences + +**Positive:** +- LLM consumes the JSON directly — no parsing step needed. +- All metadata in one object per convention — imports, args, files, packages all together. +- `--json` is a single flag — the default text output remains for human review. + +**Negative:** +- JSON is more verbose than text (full import list instead of truncated preview). +- No easy way to limit output size — a large codebase produces JSON that may overflow context. + - Mitigation: `--include` flag filters files before analysis. + +## Alternatives Considered + +- **YAML output**: More readable, but less universally parseable by LLMs. +- **CSV output**: Too flat for nested data (arg_patterns, imports list). +- **Custom prompt template**: Would need per-framework templates. JSON is framework-agnostic. +- **No structured output**: User must pipe through `jq` or manual reformatting. Bad UX. diff --git a/docs/adr/0008-bex-ensemble-for-grammar-inference.md b/docs/adr/0008-bex-ensemble-for-grammar-inference.md new file mode 100644 index 0000000..4ed0563 --- /dev/null +++ b/docs/adr/0008-bex-ensemble-for-grammar-inference.md @@ -0,0 +1,60 @@ +# 8. BEX ensemble for grammar inference + +**Date:** 2026-07-03 + +**Status:** Accepted + +## Context + +Given a set of symbol sequences (e.g. `["every", "assertEquals", "verify"]`), we need to infer a grammar that concisely describes the pattern. Three algorithms are available: + +- **CRX**: Fast, produces unordered CHAREs (e.g. `(a+b+c)+`). Best for vocabulary discovery. +- **iDRegEx**: Slower, produces ordered regex with alternation and optionality (e.g. `a.b.(c|d)?`). Best for small, clean sequences. +- **kOREInference**: Probabilistic, handles noise well (e.g. `a.b.(b?(a|c))`). Best for diverse sequences with outliers. + +No single algorithm works best for all codebases. We need to pick the right one for each cluster automatically. + +## Decision + +Run all three algorithms (ensemble), compute MDL (Minimum Description Length) for each, and select the one with the lowest MDL score. + +MDL = grammar_length + sum of per-example encoding costs. Lower is better — the grammar explains the data most compactly. + +Ensemble logic in `infer_ensemble()`: +```python +def infer_ensemble(sequences, kmax=2, N=3, prefer=None): + best = None + best_score = float('inf') + for name, fn in [('CRX', crx), ('iDRegEx', idregex), ('kOREInference', kore)]: + if prefer and name.lower() != prefer.lower(): + continue + grammar = fn(sequences, ...) + mdl = compute_mdl(grammar, sequences) + if mdl < best_score: + best_score = mdl + best = {'algorithm': name, 'grammar': grammar, 'mdl_score': mdl} + return {'best': best, 'all': all_results, 'why': {...}} +``` + +Default `kmax=2`, `N=3` (max k for k-ORE, random trials). + +## Consequences + +**Positive:** +- CRX handles large clusters with diverse vocabulary — produces useful vocabulary bags. +- iDRegEx fires on small, focused clusters (3-12 methods) — produces ordered grammars with exact subsequences. +- kOREInference handles noisy clusters where methods share a theme but vary in exact call order. +- MDL provides a principled, automatic selection criterion. + +**Negative:** +- k-ORE algorithms fail on real code when sequences are too diverse (per-file sequences differ more than per-log sequences they were designed for). + - Clustering helps by grouping similar methods before inference. +- iDRegEx can produce overfit grammars on very small clusters (3 methods) — e.g. `every.every.verify.(assertEquals)?` for 3 methods that happen to share an exact sequence. +- MDL comparison assumes grammars are comparable — CRX CHAREs and iDRegEx regex use different notation, so length comparison is approximate. + +## Alternatives Considered + +- **Single algorithm (CRX only)**: Fast but produces only unordered vocab — misses ordering conventions entirely. +- **Single algorithm (iDRegEx only)**: Produces ordered grammars but fails on diverse inputs (returns `ε`). +- **Single algorithm (kORE only)**: Most robust to noise but slowest, and still fails on highly diverse code sequences. +- **Algorithm per cluster size**: Manual heuristic (CRX for >20 methods, iDRegEx for <10). Harder to tune than MDL-driven selection.