47 lines
2.8 KiB
Markdown
47 lines
2.8 KiB
Markdown
|
# 3. Method-level n-gram clustering before inference
|
||
|
|
|
||
|
|
**Date:** 2026-07-03
|
||
|
|
|
||
|
|
**Status:** Accepted
|
||
|
|
|
||
|
|
## Context
|
||
|
|
|
||
|
|
The BEX ensemble (CRX, iDRegEx, kORE) infers grammars from sets of symbol sequences. When we run inference on *all methods in a codebase*, the sequences are too diverse — each file has different conventions, and the ensemble produces only a flat vocabulary bag like `(any+assertEquals+assertTrue+every+listOf+verify)+`.
|
||
|
|
|
||
|
|
This doesn't capture the *ordering* of calls or the distinct methodological styles present in the codebase.
|
||
|
|
|
||
|
|
## Decision
|
||
|
|
|
||
|
|
Group methods by shared n-gram (default: 3-gram) call patterns *before* running inference.
|
||
|
|
|
||
|
|
Pipeline: `preprocess_by_method` → `frequency_filter` → `cluster_methods` → per-cluster `infer_ensemble`
|
||
|
|
|
||
|
|
The clustering algorithm:
|
||
|
|
1. Extract call tokens from each method sequence (filter to `function`, `reference.call`, `reference.class` captures).
|
||
|
|
2. Build an n-gram index: for each method, for each sliding window of size N, record the n-gram.
|
||
|
|
3. Sort n-grams by frequency (most shared first).
|
||
|
|
4. **Multi-assignment**: methods can belong to every cluster whose n-gram they match (no greedy `used` subtraction). This avoids the first-pattern-hoards-all problem.
|
||
|
|
5. Capped at 20 clusters (`max_clusters=20`) to prevent output bloat from many single-token n-grams.
|
||
|
|
6. Methods matching NO n-gram (sequences shorter than N, or no peers sharing their n-grams) go to `(other)`.
|
||
|
|
|
||
|
|
Adaptive ngram fallback (`cluster_methods_adaptive`): when `(other)` exceeds 60% of total methods, retry with ngram-1. Repeats down to ngram=1. This prevents a single dominant call token from leaving 95% of methods unclustered.
|
||
|
|
|
||
|
|
## Consequences
|
||
|
|
|
||
|
|
**Positive:**
|
||
|
|
- iDRegEx and kOREInference now produce ordered grammars (e.g. `every+.assertEquals.verify+.any?`) because small, focused clusters have enough signal.
|
||
|
|
- Each cluster reveals a distinct *methodological style* in the codebase (mockist TDD vs data-driven testing vs pure assertion).
|
||
|
|
- Multi-assignment means a method can reveal multiple patterns simultaneously (e.g., both `assertEquals`-heavy and `mockk`-heavy clusters).
|
||
|
|
- Adaptive ngram shrink finds the right granularity automatically.
|
||
|
|
|
||
|
|
**Negative:**
|
||
|
|
- Multi-assignment inflates total `method_count` across clusters (one method counted in N clusters).
|
||
|
|
- Clustering adds a hyperparameter (`ngram_size`, default 3). Adaptive shrink mitigates the tuning burden.
|
||
|
|
- `min_cluster_size` (default 3) filters out tiny but potentially interesting patterns.
|
||
|
|
|
||
|
|
## Alternatives Considered
|
||
|
|
|
||
|
|
- **Infer on all methods (no clustering)**: Produces flat vocabulary only. CRX works at 100% coverage, but iDRegEx and kORE fail on diverse inputs.
|
||
|
|
- **Infer per file**: Too fine-grained — most files have 1-5 methods, not enough for inference.
|
||
|
|
- **Infer per directory**: Better, but directories mix unrelated conventions (setup/teardown vs actual test logic).
|