# 4. Frequency filter with `min_coverage` threshold

**Date:** 2026-07-03

**Status:** Accepted

## Context

A raw method sequence can contain hundreds of unique call tokens, many of which appear in only 1-2 methods. These rare symbols are noise — they inflate grammar size, confuse the inference algorithm, and dilute the signal of common conventions.

We need a principled way to discard rare symbols while keeping the behavioral patterns that define the codebase.

## Decision

Apply a frequency filter *before* clustering: remove any symbol that appears in fewer than `min_coverage` fraction of method sequences.

Default threshold: `0.2` (20% of methods must contain the symbol).

Filtering is done by `frequency_filter()` in `analyze.py`:
```python
n_files = len(sequences)
threshold = max(1, int(n_files * min_coverage))
symbol_file_count = Counter()
for seq in sequences:
    seen = set()
    for _, text, _ in seq:
        symbol_file_count[text] += 1 if text not in seen else 0
        seen.add(text)
keep = {text for text, count in symbol_file_count.items() if count >= threshold}
```

A symbol is counted once per file (not once per occurrence) to avoid skew from files that repeat the same symbol many times.

## Consequences

**Positive:**
- Removes noise before clustering, improving cluster quality.
- Prevents rare one-off function calls from creating spurious n-gram matches.
- The `common` vs `other` cluster distinction is sharper because the filter removes tokens that would appear in neither.

**Negative:**
- With large codebases (600+ methods), 20% threshold may be too aggressive — a symbol needs 120+ occurrences to survive.
  - Mitigation: `--min-coverage` flag lets users tune per codebase.
  - Production code often needs lower values (`0.05`) because methods are more diverse than tests.
- The threshold is relative, not absolute. A 3-file project keeps anything in 1+ files (`max(1, 3*0.2)` = 1).

## Alternatives Considered

- **No filter**: CRX produces `(a+b+c+d+e+f+g+h+i+j+...)+` — the vocabulary is too large to be informative.
- **Absolute threshold**: `min_occurrences=5`. Doesn't scale — works for small projects, wrong for large ones.
- **TF-IDF style weighting**: More sophisticated but adds complexity. The simple coverage filter works well in practice.