# 12. Remove n-gram clustering from pipeline

**Date:** 2026-07-04

**Status:** Accepted

## Context

The pipeline grouped method sequences by shared n-gram call patterns before
inference. The idea: methods in the same category (test, config, helper) would
cluster together, and each cluster would get a cleaner, more specific grammar.

## What Was Removed

- `cluster_methods()` — built n-gram→method index from call tokens, assigned
  methods to shared-pattern clusters, dumped remainder to `(other)`.
- `cluster_methods_adaptive()` — retried clustering at ngram=2 then ngram=1
  when `(other)` exceeded 60% of methods.
- `_extract_call_tokens()` import from `code.py` (still in `code.py` but no
  longer called by the pipeline).
- `--min-cluster-size` and `--ngram-size` CLI flags.
- `cluster` parameter in `analyze_directory()`.
- Per-cluster loop in `analyze_clusters()` — metadata extraction (files,
  imports, arg patterns) now runs once across all methods.

## Why Removed

1. **No benefit for diverse codebases.** On Kotlin SpringRAG (1581 methods),
   21 named clusters formed (3-13 methods each) but 1254 landed in `(other)`.
   The named clusters were too small to produce better grammars than running
   inference once on the filtered set.

2. **Added complexity for zero signal gain.** The n-gram labels (e.g.
   `locator → click → waitForTimeout`) restated what CRX already outputs as
   `(locator+click+waitForTimeout)+`.per-cluster infer_ensemble call was
   redundant with the single-pass result.

3. **Slower.** 22 extra BEX calls (one per named cluster) for grammars
   that would appear in the single-pass result anyway.

## Pipeline After Removal

```
preprocess_by_method → frequency_filter(0.2) → infer_ensemble(0.8)
```

Single pass. Metadata extracted once.

## How to Reintroduce

The removed code is preserved in the archive branch:

```
git archive/unreverted-25898c2
```

Files:
- `bex/tag_preprocessor/analyze.py` contains `cluster_methods()`,
  `cluster_methods_adaptive()`, and the per-cluster loop body.

To restore:
1. Cherry-pick or copy the two function definitions.
2. Re-add `_extract_call_tokens` to the import from `code.py`.
3. Re-add the `--min-cluster-size` and `--ngram-size` CLI flags.
4. Change `analyze_clusters()` back to: cluster → per-cluster filter → per-cluster infer.
5. Restore the cluster parameter in `analyze_directory()`.

## Performance Impact

| Metric | Before (with clustering) | After (single pass) |
|--------|--------------------------|---------------------|
| Pipeline time (460 .kt files) | ~42s | ~3s |
| Inference time per cluster | ~22s on (other) | ~0.1s total |
| Named clusters | 21 tiny + (other) | 1 group |
| Grammar quality | same `assertEquals+` | same `assertEquals+` |