76 lines
2.7 KiB
Markdown
76 lines
2.7 KiB
Markdown
|
# 12. Remove n-gram clustering from pipeline
|
||
|
|
|
||
|
|
**Date:** 2026-07-04
|
||
|
|
|
||
|
|
**Status:** Accepted
|
||
|
|
|
||
|
|
## Context
|
||
|
|
|
||
|
|
The pipeline grouped method sequences by shared n-gram call patterns before
|
||
|
|
inference. The idea: methods in the same category (test, config, helper) would
|
||
|
|
cluster together, and each cluster would get a cleaner, more specific grammar.
|
||
|
|
|
||
|
|
## What Was Removed
|
||
|
|
|
||
|
|
- `cluster_methods()` — built n-gram→method index from call tokens, assigned
|
||
|
|
methods to shared-pattern clusters, dumped remainder to `(other)`.
|
||
|
|
- `cluster_methods_adaptive()` — retried clustering at ngram=2 then ngram=1
|
||
|
|
when `(other)` exceeded 60% of methods.
|
||
|
|
- `_extract_call_tokens()` import from `code.py` (still in `code.py` but no
|
||
|
|
longer called by the pipeline).
|
||
|
|
- `--min-cluster-size` and `--ngram-size` CLI flags.
|
||
|
|
- `cluster` parameter in `analyze_directory()`.
|
||
|
|
- Per-cluster loop in `analyze_clusters()` — metadata extraction (files,
|
||
|
|
imports, arg patterns) now runs once across all methods.
|
||
|
|
|
||
|
|
## Why Removed
|
||
|
|
|
||
|
|
1. **No benefit for diverse codebases.** On Kotlin SpringRAG (1581 methods),
|
||
|
|
21 named clusters formed (3-13 methods each) but 1254 landed in `(other)`.
|
||
|
|
The named clusters were too small to produce better grammars than running
|
||
|
|
inference once on the filtered set.
|
||
|
|
|
||
|
|
2. **Added complexity for zero signal gain.** The n-gram labels (e.g.
|
||
|
|
`locator → click → waitForTimeout`) restated what CRX already outputs as
|
||
|
|
`(locator+click+waitForTimeout)+`.per-cluster infer_ensemble call was
|
||
|
|
redundant with the single-pass result.
|
||
|
|
|
||
|
|
3. **Slower.** 22 extra BEX calls (one per named cluster) for grammars
|
||
|
|
that would appear in the single-pass result anyway.
|
||
|
|
|
||
|
|
## Pipeline After Removal
|
||
|
|
|
||
|
|
```
|
||
|
|
preprocess_by_method → frequency_filter(0.2) → infer_ensemble(0.8)
|
||
|
|
```
|
||
|
|
|
||
|
|
Single pass. Metadata extracted once.
|
||
|
|
|
||
|
|
## How to Reintroduce
|
||
|
|
|
||
|
|
The removed code is preserved in the archive branch:
|
||
|
|
|
||
|
|
```
|
||
|
|
git archive/unreverted-25898c2
|
||
|
|
```
|
||
|
|
|
||
|
|
Files:
|
||
|
|
- `bex/tag_preprocessor/analyze.py` contains `cluster_methods()`,
|
||
|
|
`cluster_methods_adaptive()`, and the per-cluster loop body.
|
||
|
|
|
||
|
|
To restore:
|
||
|
|
1. Cherry-pick or copy the two function definitions.
|
||
|
|
2. Re-add `_extract_call_tokens` to the import from `code.py`.
|
||
|
|
3. Re-add the `--min-cluster-size` and `--ngram-size` CLI flags.
|
||
|
|
4. Change `analyze_clusters()` back to: cluster → per-cluster filter → per-cluster infer.
|
||
|
|
5. Restore the cluster parameter in `analyze_directory()`.
|
||
|
|
|
||
|
|
## Performance Impact
|
||
|
|
|
||
|
|
| Metric | Before (with clustering) | After (single pass) |
|
||
|
|
|--------|--------------------------|---------------------|
|
||
|
|
| Pipeline time (460 .kt files) | ~42s | ~3s |
|
||
|
|
| Inference time per cluster | ~22s on (other) | ~0.1s total |
|
||
|
|
| Named clusters | 21 tiny + (other) | 1 group |
|
||
|
|
| Grammar quality | same `assertEquals+` | same `assertEquals+` |
|