feat: adaptive multi-assignment clustering; add ADRs 1-10
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed

- Multi-assignment clustering (no greedy 'used' set)
- Adaptive ngram fallback (shrink when (other) > 60%)
- Add docs/adr/ with 10 architecture decision records
- Fix ADR 1 (query modification description)
- Fix ADR 3 (multi-assignment + adaptive shrink)
- Fix ADR 5 (import sort order clarification)
- Fix ADR 6 (remove Kotlin call_suffix references)
- New ADR 9 (adaptive clustering rationale)
- New ADR 10 (universal package mapping via relpath)
This commit is contained in:
tobjend 2026-07-03 22:58:09 +02:00
parent fd574da53d
commit e23922a1b7
11 changed files with 545 additions and 8 deletions

View file

@ -190,17 +190,18 @@ def frequency_filter(sequences, min_coverage=0.2):
return filtered return filtered
def cluster_methods(sequences, min_cluster_size=3, ngram_size=3): def cluster_methods(sequences, min_cluster_size=3, ngram_size=3, max_clusters=20):
"""Group method sequences by shared n-gram call patterns. """Group method sequences by shared n-gram call patterns.
Extracts call tokens from each sequence, builds an n-gram index, Extracts call tokens from each sequence, builds an n-gram index,
and assigns methods to the largest matching clusters first. and assigns methods to all matching clusters (multi-assignment).
Remaining methods go to an '(other)' cluster. Methods that match NO n-gram pattern go to an '(other)' cluster.
Args: Args:
sequences: list of (capture, text, line) lists. sequences: list of (capture, text, line) lists.
min_cluster_size: minimum methods to form a cluster. min_cluster_size: minimum methods to form a cluster.
ngram_size: length of n-grams to match (default 3). ngram_size: length of n-grams to match (default 3).
max_clusters: max clusters before dumping rest to (other).
Returns: Returns:
list of (label, [sequences]) tuples. list of (label, [sequences]) tuples.
@ -217,23 +218,45 @@ def cluster_methods(sequences, min_cluster_size=3, ngram_size=3):
if len(ngram) == ngram_size: if len(ngram) == ngram_size:
ngram_to_methods.setdefault(ngram, set()).add(idx) ngram_to_methods.setdefault(ngram, set()).add(idx)
used = set()
clusters = [] clusters = []
assigned = set()
for ngram, indices in sorted(ngram_to_methods.items(), key=lambda x: -len(x[1])): for ngram, indices in sorted(ngram_to_methods.items(), key=lambda x: -len(x[1])):
indices = indices - used
if len(indices) >= min_cluster_size: if len(indices) >= min_cluster_size:
if len(clusters) >= max_clusters:
break
label = "".join(ngram) label = "".join(ngram)
cluster_seqs = [sequences[i] for i in indices] cluster_seqs = [sequences[i] for i in indices]
clusters.append((label, cluster_seqs)) clusters.append((label, cluster_seqs))
used.update(indices) assigned.update(indices)
remaining = [i for i in range(len(sequences)) if i not in used] remaining = [i for i in range(len(sequences)) if i not in assigned]
if remaining: if remaining:
clusters.append(("(other)", [sequences[i] for i in remaining])) clusters.append(("(other)", [sequences[i] for i in remaining]))
return clusters return clusters
def cluster_methods_adaptive(sequences, min_cluster_size=3, ngram_size=3, other_threshold=0.6):
"""Adaptive clustering: shrink ngram until (other) <= threshold or ngram=1.
When the (other) cluster swallows > other_threshold of methods,
retry with ngram-1. Keeps the smallest ngram that gives acceptable coverage.
"""
for n in range(ngram_size, 0, -1):
clusters = cluster_methods(sequences, min_cluster_size=min_cluster_size, ngram_size=n)
other_count = 0
total = 0
for label, seqs in clusters:
total += len(seqs)
if label == "(other)":
other_count = len(seqs)
if total == 0:
return clusters
if other_count / total <= other_threshold:
break
return clusters
def analyze_clusters(file_paths, extension, project_root="", min_coverage=0.2, prefer=None, kmax=2, N=3): def analyze_clusters(file_paths, extension, project_root="", min_coverage=0.2, prefer=None, kmax=2, N=3):
"""Run full pipeline with clustering: preprocess → cluster → per-cluster infer. """Run full pipeline with clustering: preprocess → cluster → per-cluster infer.
@ -255,7 +278,7 @@ def analyze_clusters(file_paths, extension, project_root="", min_coverage=0.2, p
return [] return []
sequences = frequency_filter(sequences, min_coverage) sequences = frequency_filter(sequences, min_coverage)
clusters = cluster_methods(sequences) clusters = cluster_methods_adaptive(sequences)
results = [] results = []
for label, cluster_seqs in clusters: for label, cluster_seqs in clusters:

View file

@ -0,0 +1,41 @@
# 1. Use nvim-treesitter `highlights.scm` as behavioral capture source
**Date:** 2026-07-03
**Status:** Accepted
## Context
We need a universal source of behavioral code tokens (function calls, references, definitions) across multiple programming languages. Options:
- **`tags.scm`** (nvim-treesitter): Purpose-built for symbol tagging. Covers definitions and references.
- **`highlights.scm`** (nvim-treesitter): Built for syntax highlighting. Covers a wider range of tokens including keywords, operators, and built-ins.
- **Custom per-language queries**: Write and maintain our own query files for each language.
We need tokens that represent *what the code does at runtime* — not just structure.
## Decision
Use nvim-treesitter `highlights.scm` as the capture source for all 10 languages.
We filter captures to a `BEHAVIORAL_PREFIXES` set: `definition.`, `reference.`, `keyword.`, `function`, `attribute`, `constructor`, `label`, `type.definition`, `module`.
For Kotlin, use the `ts-kotlin` (fwcd fork) bundled `highlights.scm` instead of nvim-treesitter's, because nvim-treesitter's Kotlin query references a duplicate `annotation` node type that doesn't exist in the grammar.
## Consequences
**Positive:**
- `highlights.scm` covers 4 out of 5 behavioral capture types that `tags.scm` misses, across all 10 languages.
- No per-language custom code or adapters needed.
- Community-maintained queries stay fresh with language evolution.
- Same query files work for both parsing and tokenizing.
**Negative:**
- `highlights.scm` includes non-behavioral captures (comments, punctuation, operators) — we filter these out.
- Some nvim-treesitter queries use `#set!` directives (`#set! priority`, `#eq?`) that `py-tree-sitter` doesn't support. These patterns are removed in the bundled copies under `queries/`.
- Kotlin requires a separate grammar package (`ts-kotlin`) because the nvim-treesitter Kotlin grammar is incompatible.
## Alternatives Considered
- **`tags.scm`**: Cleaner signal-to-noise ratio, but misses `function`, `attribute`, `constructor`, `module` captures that are essential for behavioral understanding.
- **Custom queries**: Would give full control but require per-language maintenance — violates our universal-preprocessor constraint.

View file

@ -0,0 +1,50 @@
# 2. Language-agnostic method extraction via `child_by_field_name("body")`
**Date:** 2026-07-03
**Status:** Accepted
## Context
To analyze method-level behavioral conventions, we must extract the body of each function/method from the AST. The standard tree-sitter approach is `node.child_by_field_name("body")`, but this named field is not universal across all language grammars.
We need one code path that works for all 10 supported languages without per-language branches.
## Decision
Use `node.child_by_field_name("body")` as the primary extraction method. When it returns `None`, fall back to scanning the node's children for any child with a type containing `body`, `block`, or `compound_statement`.
Parent nodes are further filtered to only include nodes whose type contains `function` or `method` — avoiding class bodies, loop bodies, and conditional blocks.
This logic lives in `_find_method_bodies()` in `code.py`:
```python
def walk(node):
body = node.child_by_field_name("body")
if not body:
for child in node.children:
ctype = child.type.lower()
if "body" in ctype or "block" in ctype or ctype == "compound_statement":
body = child; break
if body:
ptype = node.type.lower()
if "function" in ptype or "method" in ptype:
bodies.append(body)
for child in node.children: walk(child)
```
## Consequences
**Positive:**
- Works for 9/10 grammars via `child_by_field_name("body")` alone (Python, Go, Rust, JS, TS, Ruby, Java, C, C++).
- Kotlin fallback works because the fwcd Kotlin grammar uses `function_body` as a child node type.
- Zero per-language case analysis — just pattern matching on type strings.
**Negative:**
- Fallback relies on string matching (`"body" in ctype`) which could produce false positives if future grammar versions introduce new body-like types.
- C/C++ `function_definition` uses `declarator` field for the function name, not `name` — affects name extraction but not body extraction.
## Alternatives Considered
- **Grammar-specific field names**: Map each language to its body field name. Rejected because it creates a maintenance burden and violates the zero-adapters constraint.
- **Top-down sibling traversal**: Walk from node start to next sibling to find the body. Fragile across grammars with different compound statement structures.

View file

@ -0,0 +1,46 @@
# 3. Method-level n-gram clustering before inference
**Date:** 2026-07-03
**Status:** Accepted
## Context
The BEX ensemble (CRX, iDRegEx, kORE) infers grammars from sets of symbol sequences. When we run inference on *all methods in a codebase*, the sequences are too diverse — each file has different conventions, and the ensemble produces only a flat vocabulary bag like `(any+assertEquals+assertTrue+every+listOf+verify)+`.
This doesn't capture the *ordering* of calls or the distinct methodological styles present in the codebase.
## Decision
Group methods by shared n-gram (default: 3-gram) call patterns *before* running inference.
Pipeline: `preprocess_by_method``frequency_filter``cluster_methods` → per-cluster `infer_ensemble`
The clustering algorithm:
1. Extract call tokens from each method sequence (filter to `function`, `reference.call`, `reference.class` captures).
2. Build an n-gram index: for each method, for each sliding window of size N, record the n-gram.
3. Sort n-grams by frequency (most shared first).
4. **Multi-assignment**: methods can belong to every cluster whose n-gram they match (no greedy `used` subtraction). This avoids the first-pattern-hoards-all problem.
5. Capped at 20 clusters (`max_clusters=20`) to prevent output bloat from many single-token n-grams.
6. Methods matching NO n-gram (sequences shorter than N, or no peers sharing their n-grams) go to `(other)`.
Adaptive ngram fallback (`cluster_methods_adaptive`): when `(other)` exceeds 60% of total methods, retry with ngram-1. Repeats down to ngram=1. This prevents a single dominant call token from leaving 95% of methods unclustered.
## Consequences
**Positive:**
- iDRegEx and kOREInference now produce ordered grammars (e.g. `every+.assertEquals.verify+.any?`) because small, focused clusters have enough signal.
- Each cluster reveals a distinct *methodological style* in the codebase (mockist TDD vs data-driven testing vs pure assertion).
- Multi-assignment means a method can reveal multiple patterns simultaneously (e.g., both `assertEquals`-heavy and `mockk`-heavy clusters).
- Adaptive ngram shrink finds the right granularity automatically.
**Negative:**
- Multi-assignment inflates total `method_count` across clusters (one method counted in N clusters).
- Clustering adds a hyperparameter (`ngram_size`, default 3). Adaptive shrink mitigates the tuning burden.
- `min_cluster_size` (default 3) filters out tiny but potentially interesting patterns.
## Alternatives Considered
- **Infer on all methods (no clustering)**: Produces flat vocabulary only. CRX works at 100% coverage, but iDRegEx and kORE fail on diverse inputs.
- **Infer per file**: Too fine-grained — most files have 1-5 methods, not enough for inference.
- **Infer per directory**: Better, but directories mix unrelated conventions (setup/teardown vs actual test logic).

View file

@ -0,0 +1,51 @@
# 4. Frequency filter with `min_coverage` threshold
**Date:** 2026-07-03
**Status:** Accepted
## Context
A raw method sequence can contain hundreds of unique call tokens, many of which appear in only 1-2 methods. These rare symbols are noise — they inflate grammar size, confuse the inference algorithm, and dilute the signal of common conventions.
We need a principled way to discard rare symbols while keeping the behavioral patterns that define the codebase.
## Decision
Apply a frequency filter *before* clustering: remove any symbol that appears in fewer than `min_coverage` fraction of method sequences.
Default threshold: `0.2` (20% of methods must contain the symbol).
Filtering is done by `frequency_filter()` in `analyze.py`:
```python
n_files = len(sequences)
threshold = max(1, int(n_files * min_coverage))
symbol_file_count = Counter()
for seq in sequences:
seen = set()
for _, text, _ in seq:
symbol_file_count[text] += 1 if text not in seen else 0
seen.add(text)
keep = {text for text, count in symbol_file_count.items() if count >= threshold}
```
A symbol is counted once per file (not once per occurrence) to avoid skew from files that repeat the same symbol many times.
## Consequences
**Positive:**
- Removes noise before clustering, improving cluster quality.
- Prevents rare one-off function calls from creating spurious n-gram matches.
- The `common` vs `other` cluster distinction is sharper because the filter removes tokens that would appear in neither.
**Negative:**
- With large codebases (600+ methods), 20% threshold may be too aggressive — a symbol needs 120+ occurrences to survive.
- Mitigation: `--min-coverage` flag lets users tune per codebase.
- Production code often needs lower values (`0.05`) because methods are more diverse than tests.
- The threshold is relative, not absolute. A 3-file project keeps anything in 1+ files (`max(1, 3*0.2)` = 1).
## Alternatives Considered
- **No filter**: CRX produces `(a+b+c+d+e+f+g+h+i+j+...)+` — the vocabulary is too large to be informative.
- **Absolute threshold**: `min_occurrences=5`. Doesn't scale — works for small projects, wrong for large ones.
- **TF-IDF style weighting**: More sophisticated but adds complexity. The simple coverage filter works well in practice.

View file

@ -0,0 +1,45 @@
# 5. Import extraction per cluster
**Date:** 2026-07-03
**Status:** Accepted
## Context
An LLM prompted with a behavioral convention like `every → assertEquals → verify` still needs to know *which imports to use*. Without imports, it will guess the wrong library — writing `from unittest.mock import patch` instead of `import io.mockk.every`, or importing from `jest` instead of `vitest`.
Imports are the bridge between abstract conventions and actionable code.
## Decision
For each cluster, scan the source files whose methods belong to that cluster and extract all unique import lines.
Language-agnostic approach: match lines against common import patterns:
- `import ...` (Java, Kotlin, Python, Go, JS/TS)
- `from ... import ...` (Python)
- `require ...` / `require_relative ...` (Ruby, JS)
- `#include ...` (C/C++)
- `use ...` (Rust)
- `include ...` (Ruby)
Scan the first 200 lines of each file (imports are always at the top), deduplicate across files, yielding stable file-visit order (files sorted by path).
File-to-cluster mapping is preserved by tracking `(file_path, sequence)` pairs through the pipeline. With multi-assignment clustering, methods may belong to multiple clusters — each gets the full import set from its source file.
## Consequences
**Positive:**
- Each cluster shows exact import lines used by its methods.
- An LLM can copy these directly — no guessing.
- Reveals *library choice conventions*: `kotlin.test.*` vs `org.junit.jupiter.api.*`, `io.mockk.coEvery` vs `io.mockk.every`.
**Negative:**
- Import scanning re-reads files (second pass). Negligible cost since files are small and OS-cached.
- 200-line scan limit might miss imports in files with very long license headers.
- Lines containing `import` in prose (comments, strings) may produce false positives — rare in practice.
## Alternatives Considered
- **Single global import list**: Simpler but useless — conflates imports from unrelated clusters.
- **No imports**: LLM must guess. Leads to wrong imports and broken code.
- **Per-file imports (not per-cluster)**: Too granular — mixes test imports with production imports in the same file.

View file

@ -0,0 +1,54 @@
# 6. Argument pattern extraction via AST node classification
**Date:** 2026-07-03
**Status:** Accepted
## Context
A behavioral token like `assertEquals` tells the LLM that the function is called, but not *how*. Two codebases both use `assertEquals` — one writes `assertEquals(expected, actual)` and the other writes `assertEquals(actual, expected)` with swapped argument order. An LLM guessing the wrong order writes broken tests.
The highlights.scm captures tell us *that* a function is called. We need the argument *structure* — number of arguments, their types, and the common patterns.
## Decision
For each behavioral capture node, walk up to its parent `call_expression` (or equivalent), find the argument list node, and classify each argument by structural role.
Argument classification is language-agnostic:
| Classification | Matches |
|---|---|
| `lit` | string, number, boolean, null |
| `var` | identifiers, names |
| `call` | nested call expressions, method invocations |
| `lambda` | lambda expressions, blocks, do-blocks |
| `kwarg` | keyword/named arguments |
| `expr` | binary/unary/ternary/operator expressions |
| `template` | string interpolation, template literals |
| `other` | anything else (fallback) |
Argument list node detection uses a tiered approach:
1. `child_by_field_name("arguments")` — works for Python, JS, TS, Java, Go, Ruby, Rust.
2. Fallback: scan children for `argument_list`, `arguments`, `template_string` (JS tagged templates).
The argument iterator is a simple generic traversal: yield all named children of the arglist node. No per-language special cases. This works for positional args, keyword args, lambdas inside argument lists, and template expressions.
Results are aggregated per cluster into a summary showing min/max/common arg counts and the top argument-type patterns.
## Consequences
**Positive:**
- Reveals argument ordering conventions: `assertEquals: n=2 [lit,var]` means expected-first.
- Reveals calling convention variance: `verify: n=0 [] | n=1 [lambda] | n=1 [var]` means three styles coexist.
- Zero per-language branches — generic tiered detection and iteration handles all 10 grammars.
**Negative:**
- `kwarg` detection only covers named arguments, not default values or spread operators.
- Nested destructuring patterns fall into `other` bucket — no granularity for complex argument shapes.
- `other` is a catch-all that can hide meaningful distinctions we haven't classified yet.
## Alternatives Considered
- **Extract raw argument text**: Language-agnostic but fragile — variable names change per test, producing high variance and low signal.
- **No argument extraction**: The LLM sees `assertEquals` but doesn't know argument order. Leads to wrong code.
- **Per-language argument extractors**: Would be more precise but violate the zero-adapters constraint.

View file

@ -0,0 +1,62 @@
# 7. JSON output for LLM prompt injection
**Date:** 2026-07-03
**Status:** Accepted
## Context
The text table output is human-readable but not directly usable by an LLM. To use behavioral conventions in another agent or coding session, the output must be parsed, reformatted, and injected into a prompt — an extra friction step.
An LLM consuming conventions needs:
- Structured data it can read directly (no parsing).
- All metadata per convention (grammar, imports, args, files, packages).
- Compact enough to fit in context without overflow.
## Decision
Add a `--json` flag that outputs a structured JSON array instead of the text table.
JSON structure:
```json
[{
"language": ".kt",
"conventions": [{
"label": "assertEquals",
"method_count": 327,
"algorithm": "CRX",
"grammar": "assertEquals+",
"mdl_score": 1.0,
"imports": ["import io.mockk.every", "..."],
"arg_patterns": {
"assertEquals": {
"occurrences": 42,
"arg_count": {"min": 2, "max": 3, "common": 2},
"patterns": [{"count": 30, "args": 2, "types": ["lit", "var"]}]
}
}
}],
"total_methods": 1581
}]
```
Also accepts `--format json` and `--format text` for explicit control.
## Consequences
**Positive:**
- LLM consumes the JSON directly — no parsing step needed.
- All metadata in one object per convention — imports, args, packages all together.
- `--json` is a single flag — the default text output remains for human review.
**Negative:**
- JSON is more verbose than text (full import list instead of truncated preview).
- No easy way to limit output size — a large codebase produces JSON that may overflow context.
- Mitigation: `--include` flag filters files before analysis, and `max_clusters=20` caps cluster count.
## Alternatives Considered
- **YAML output**: More readable, but less universally parseable by LLMs.
- **CSV output**: Too flat for nested data (arg_patterns, imports list).
- **Custom prompt template**: Would need per-framework templates. JSON is framework-agnostic.
- **No structured output**: User must pipe through `jq` or manual reformatting. Bad UX.

View file

@ -0,0 +1,60 @@
# 8. BEX ensemble for grammar inference
**Date:** 2026-07-03
**Status:** Accepted
## Context
Given a set of symbol sequences (e.g. `["every", "assertEquals", "verify"]`), we need to infer a grammar that concisely describes the pattern. Three algorithms are available:
- **CRX**: Fast, produces unordered CHAREs (e.g. `(a+b+c)+`). Best for vocabulary discovery.
- **iDRegEx**: Slower, produces ordered regex with alternation and optionality (e.g. `a.b.(c|d)?`). Best for small, clean sequences.
- **kOREInference**: Probabilistic, handles noise well (e.g. `a.b.(b?(a|c))`). Best for diverse sequences with outliers.
No single algorithm works best for all codebases. We need to pick the right one for each cluster automatically.
## Decision
Run all three algorithms (ensemble), compute MDL (Minimum Description Length) for each, and select the one with the lowest MDL score.
MDL = grammar_length + sum of per-example encoding costs. Lower is better — the grammar explains the data most compactly.
Ensemble logic in `infer_ensemble()`:
```python
def infer_ensemble(sequences, kmax=2, N=3, prefer=None):
best = None
best_score = float('inf')
for name, fn in [('CRX', crx), ('iDRegEx', idregex), ('kOREInference', kore)]:
if prefer and name.lower() != prefer.lower():
continue
grammar = fn(sequences, ...)
mdl = compute_mdl(grammar, sequences)
if mdl < best_score:
best_score = mdl
best = {'algorithm': name, 'grammar': grammar, 'mdl_score': mdl}
return {'best': best, 'all': all_results, 'why': {...}}
```
Default `kmax=2`, `N=3` (max k for k-ORE, random trials).
## Consequences
**Positive:**
- CRX handles large clusters with diverse vocabulary — produces useful vocabulary bags.
- iDRegEx fires on small, focused clusters (3-12 methods) — produces ordered grammars with exact subsequences.
- kOREInference handles noisy clusters where methods share a theme but vary in exact call order.
- MDL provides a principled, automatic selection criterion.
**Negative:**
- k-ORE algorithms fail on real code when sequences are too diverse (per-file sequences differ more than per-log sequences they were designed for).
- Clustering helps by grouping similar methods before inference.
- iDRegEx can produce overfit grammars on very small clusters (3 methods) — e.g. `every.every.verify.(assertEquals)?` for 3 methods that happen to share an exact sequence.
- MDL comparison assumes grammars are comparable — CRX CHAREs and iDRegEx regex use different notation, so length comparison is approximate.
## Alternatives Considered
- **Single algorithm (CRX only)**: Fast but produces only unordered vocab — misses ordering conventions entirely.
- **Single algorithm (iDRegEx only)**: Produces ordered grammars but fails on diverse inputs (returns `ε`).
- **Single algorithm (kORE only)**: Most robust to noise but slowest, and still fails on highly diverse code sequences.
- **Algorithm per cluster size**: Manual heuristic (CRX for >20 methods, iDRegEx for <10). Harder to tune than MDL-driven selection.

View file

@ -0,0 +1,49 @@
# 9. Adaptive clustering with multi-assignment
**Date:** 2026-07-03
**Status:** Accepted
## Context
The original clustering (ADR 3) assigned each method to exactly one cluster — the first matching n-gram sorted by frequency. This caused a "winner-takes-all" problem: the most common token (e.g., `assertEquals`) claimed 327 methods, leaving 1,254 methods in `(other)` even when they shared other patterns like `mockk` or `every`.
Additionally, the optimal ngram_size varies per codebase. A small JS test suite benefits from 3-grams (catches multi-step Playwright patterns), while a large Kotlin monorepo needs 1-grams (or even overlapping patterns) to escape the `(other)` blob.
## Decision
Two changes to `cluster_methods`:
### Multi-assignment
Remove the `used` set. A method belongs to every cluster whose n-gram appears in its call sequence. This reveals overlapping patterns — e.g., a method containing both `assertEquals` and `mockk` appears in both clusters, telling the LLM "this method is both assertion-heavy AND mock-heavy."
Add `max_clusters=20` to prevent output bloat from many single-token n-grams. The 20 most frequent n-grams form clusters; the rest go to `(other)`.
### Adaptive ngram fallback
New `cluster_methods_adaptive()` wrapper:
1. Run `cluster_methods` with `ngram_size=N`.
2. If `(other)` exceeds 60% of total methods, retry with `ngram_size=N-1`.
3. Repeat down to `ngram_size=1`.
This ensures the clustering adapts to codebase diversity without manual tuning. A diverse monorepo with 1,500+ methods that share few 3-grams automatically falls back to 2-gram or 1-gram clustering.
## Consequences
**Positive:**
- Overlapping patterns surface richer signals: "327 methods call `assertEquals`, 200 methods call `mockk` (some are both)."
- Adaptive fallback eliminates manual tuning for diverse codebases.
- `max_clusters=20` keeps output concise for LLM context windows.
**Negative:**
- `method_count` sums to more than total methods (one method counted in N clusters). Users must interpret counts as "methods matching this pattern," not "methods exclusive to this cluster."
- `(other)` may still be large at ngram=1 if most methods share no single call token with ≥3 peers (rare but possible).
- Adaptive fallback adds a re-clustering pass (negligible cost — clustering is cheap vs inference).
## Alternatives Considered
- **Greedy assignment (ADR 3 original)**: Creates clean mutually exclusive clusters, but loses signal from overlapping patterns. The `(other)` blob grows uncontrollably.
- **Hierarchical clustering**: More sophisticated grouping but adds complexity — no clear benefit for our use case (clusters are consumed by an LLM, not analyzed by a human).
- **Fixed ngram_size with manual flag**: Passes the tuning burden to the user. Adaptive removes friction.

View file

@ -0,0 +1,56 @@
# 10. Universal package mapping via project-relative path
**Date:** 2026-07-03
**Status:** Accepted
## Context
Each detected behavioral convention needs a "package" or "module" label so the LLM knows where to place generated code. Options:
- **No package info**: LLM guesses wrong directory, generates code at project root.
- **Per-language directory lookup**: Hardcode `kotlin`/`java`/`python` source root names. Breeds per-language branches — violates the zero-adapters constraint.
- **Full file path**: Too verbose, exposes absolute system paths (/home/user/project/...).
- **Project-relative path**: Pure path arithmetic, no language assumptions.
## Decision
Derive the package from the file's directory relative to the project root:
```python
def _file_to_package(fp, project_root):
rel = os.path.relpath(os.path.dirname(fp), project_root)
if rel == ".":
return "" # file at project root
return rel
```
No source root markers. No per-language directory names. Just `relpath` from the root the user passed to `analyze_directory`.
Examples:
```
/project/src/main/kotlin/org/app/User.kt → src/main/kotlin/org/app
/project/mypackage/module.py → mypackage
/project/lib/core/helper.rb → lib/core
/project/src/main.rs → src
/flat/project/file.py → flat
```
The project root is the directory passed to `analyze_directory(...)` and threaded down through `analyze_clusters``_top_packages`.
## Consequences
**Positive:**
- Zero per-language branches. Works identically for all 10 languages.
- No configuration or convention list to maintain.
- LLM sees the exact directory structure it should mirror in generated code.
**Negative:**
- `relpath` assumes the project root is the scan root. Scanning a subdirectory gives partial paths (still correct, but missing context).
- Files at the root of deeply nested projects get empty package strings. Mitigation: users should scan from the project root.
## Alternatives Considered
- **Per-language source root list (reverted)**: Hardcoded `kotlin`/`java`/`python` directory names. Brittle, violated zero-adapters constraint. Reverted to `feature/kotlin-specific-extras`.
- **Source root markers (`src`/`lib`/`pkg`/`app`)**: Broader than per-language but still assumes project layout conventions. Broke for flat repos, non-standard layouts.
- **No package mapping**: Simpler but useless — LLM can't locate generated code. The package label is essential for file placement.