grammar-inference-engine/docs/adr/0012-remove-ngram-clustering.md

12. Remove n-gram clustering from pipeline

Date: 2026-07-04

Status: Accepted

Context

The pipeline grouped method sequences by shared n-gram call patterns before inference. The idea: methods in the same category (test, config, helper) would cluster together, and each cluster would get a cleaner, more specific grammar.

What Was Removed

• cluster_methods() — built n-gram→method index from call tokens, assigned methods to shared-pattern clusters, dumped remainder to (other).
• cluster_methods_adaptive() — retried clustering at ngram=2 then ngram=1 when (other) exceeded 60% of methods.
• _extract_call_tokens() import from code.py (still in code.py but no longer called by the pipeline).
• --min-cluster-size and --ngram-size CLI flags.
• cluster parameter in analyze_directory().
• Per-cluster loop in analyze_clusters() — metadata extraction (files, imports, arg patterns) now runs once across all methods.

Why Removed

1. No benefit for diverse codebases. On Kotlin SpringRAG (1581 methods), 21 named clusters formed (3-13 methods each) but 1254 landed in (other). The named clusters were too small to produce better grammars than running inference once on the filtered set.

2. Added complexity for zero signal gain. The n-gram labels (e.g. locator → click → waitForTimeout) restated what CRX already outputs as (locator+click+waitForTimeout)+.per-cluster infer_ensemble call was redundant with the single-pass result.

3. Slower. 22 extra BEX calls (one per named cluster) for grammars that would appear in the single-pass result anyway.

Pipeline After Removal

preprocess_by_method → frequency_filter(0.2) → infer_ensemble(0.8)

Single pass. Metadata extracted once.

How to Reintroduce

The removed code is preserved in the archive branch:

git archive/unreverted-25898c2

Files:

• bex/tag_preprocessor/analyze.py contains cluster_methods(), cluster_methods_adaptive(), and the per-cluster loop body.

To restore:

1. Cherry-pick or copy the two function definitions.
2. Re-add _extract_call_tokens to the import from code.py.
3. Re-add the --min-cluster-size and --ngram-size CLI flags.
4. Change analyze_clusters() back to: cluster → per-cluster filter → per-cluster infer.
5. Restore the cluster parameter in analyze_directory().

Performance Impact

Metric	Before (with clustering)	After (single pass)
Pipeline time (460 .kt files)	~42s	~3s
Inference time per cluster	~22s on (other)	~0.1s total
Named clusters	21 tiny + (other)	1 group
Grammar quality	same assertEquals+	same assertEquals+