# 3. Method-level n-gram clustering before inference

**Date:** 2026-07-03

**Status:** Accepted

## Context

The BEX ensemble (CRX, iDRegEx, kORE) infers grammars from sets of symbol sequences. When we run inference on *all methods in a codebase*, the sequences are too diverse — each file has different conventions, and the ensemble produces only a flat vocabulary bag like `(any+assertEquals+assertTrue+every+listOf+verify)+`.

This doesn't capture the *ordering* of calls or the distinct methodological styles present in the codebase.

## Decision

Group methods by shared n-gram (default: 3-gram) call patterns *before* running inference.

Pipeline: `preprocess_by_method` → `frequency_filter` → `cluster_methods` → per-cluster `infer_ensemble`

The clustering algorithm:
1. Extract call tokens from each method sequence (filter to `function`, `reference.call`, `reference.class` captures).
2. Build an n-gram index: for each method, for each sliding window of size N, record the n-gram.
3. Sort n-grams by frequency (most shared first).
4. **Multi-assignment**: methods can belong to every cluster whose n-gram they match (no greedy `used` subtraction). This avoids the first-pattern-hoards-all problem.
5. Capped at 20 clusters (`max_clusters=20`) to prevent output bloat from many single-token n-grams.
6. Methods matching NO n-gram (sequences shorter than N, or no peers sharing their n-grams) go to `(other)`.

Adaptive ngram fallback (`cluster_methods_adaptive`): when `(other)` exceeds 60% of total methods, retry with ngram-1. Repeats down to ngram=1. This prevents a single dominant call token from leaving 95% of methods unclustered.

## Consequences

**Positive:**
- iDRegEx and kOREInference now produce ordered grammars (e.g. `every+.assertEquals.verify+.any?`) because small, focused clusters have enough signal.
- Each cluster reveals a distinct *methodological style* in the codebase (mockist TDD vs data-driven testing vs pure assertion).
- Multi-assignment means a method can reveal multiple patterns simultaneously (e.g., both `assertEquals`-heavy and `mockk`-heavy clusters).
- Adaptive ngram shrink finds the right granularity automatically.

**Negative:**
- Multi-assignment inflates total `method_count` across clusters (one method counted in N clusters).
- Clustering adds a hyperparameter (`ngram_size`, default 3). Adaptive shrink mitigates the tuning burden.
- `min_cluster_size` (default 3) filters out tiny but potentially interesting patterns.

## Alternatives Considered

- **Infer on all methods (no clustering)**: Produces flat vocabulary only. CRX works at 100% coverage, but iDRegEx and kORE fail on diverse inputs.
- **Infer per file**: Too fine-grained — most files have 1-5 methods, not enough for inference.
- **Infer per directory**: Better, but directories mix unrelated conventions (setup/teardown vs actual test logic).