grammar-inference-engine/docs/language-size-scoring-analysis.md

# Language Size Scoring: Analysis and Design Notes

## Date: 2026-07-11

## Problem Statement

Our ensemble grammar inference uses a scoring function to select the best grammar
when multiple algorithms (CRX, iDRegEx, kORE) produce candidates. The scoring
function determines which grammar wins.

The concrete problem: on real codebases, the ensemble picked overly generic
grammars like `info+` over specific ones like `info.file.template.shell.service+`.
The generic grammar accepts an astronomically large language (every repetition
of `info` at every length), while the specific grammar accepts exactly one word
of each length ≥ 5. Any human would pick the specific one. Our scorer picked
the generic one.

## Root Cause Analysis

### What MDL Measures

Our old scoring function was MDL (Minimum Description Length):

```python
def mdl_score(expr, sequences):
    model = model_cost(expr)   # number of symbol occurrences in expression
    data = data_cost(expr, sequences)  # Σ log₂(|L(r)| at seq length)
    return model + data
```

`model_cost` counts how many times alphabet symbols appear in the expression.
For `info+`, that's 1 (the symbol `info` appears once). For
`info.file.template.shell.service+`, that's 5. MDL rewards short expressions.

`data_cost` sums `log₂(|L(r)|_len(seq))` over all sequences. Both `info+` and
the specific grammar accept exactly 1 word at length 5, so both get
`data_cost = 5 × log₂(1) = 0`.

Result: `mdl_score('info+') = 1 + 0 = 1.0`, `mdl_score('specific') = 5 + 0 = 5.0`.
MDL picks `info+` because its model cost is tiny.

### Why MDL Fails Here

MDL combines two signals: expression length (model cost) and compression quality
(data cost). When model cost dominates (as it does when data cost is 0 for exact
matches), short generic patterns win. This is the same problem the Bex paper
identified: MDL achieves only 21% success rate vs 98% for Language Size.

## The Bex Paper's Language Size Measure

### Paper: arXiv 1004.2372, Section 4.3.1

Bex, Goethals, Penninckx, Van Gucht, Van den Bussche published
"Learning Deterministic Regular Expressions for the Inference of Schemas
from XML Data" (VLDB 2007, arXiv:1004.2372).

They proposed the iDRegEx algorithm and evaluated two scoring measures:

1. **Language Size** (Section 4.3.1): select the expression that accepts the
   **fewest words** up to length n.
2. **MDL** (Section 4.3.2): model cost + data cost, based on Adriaans & Vitányi
   (2006).

Their result (Section 5, line 1470): on a corpus of synthetic regular expressions
with alphabet size 5, **Language Size achieved 98% success rate while MDL achieved
only 21%**. They explicitly abandoned MDL: *"Therefore in the remainder of this
section we only consider iDRegEx with the language size criterion."*

### The Paper's Formula

The paper defines:

> "We therefore only consider the words up to a length n, where n = 2m + 1
> with m the length of the candidate expression, excluding regular expression
> operators, ∅, and ε."
>
> "Then the best candidate in C is the one with the least value of |L(r)≤n|."

Concretely: for candidate `r`, compute `m = model_cost(r)` (symbol occurrences
only, no operators), then `n = 2m + 1`. Count all words in `L(r)` of length ≤ n.
Pick the candidate with the smallest count. Tie-break: pick the shortest expression.

### Why The Paper's Formula Works In Their Setting

The paper evaluates candidates against a **known target**. The experiment is:

1. Start with a target expression (e.g. `a.b.c`)
2. Generate sample S from the target (words the target accepts)
3. Run iDRegEx on S to produce candidate set C
4. Score each candidate, pick the best
5. Check if best matches the target

The critical detail: **all candidates are derived from the same target**, so they
have similar `model_cost` values and similar `n`. At the same `n`, the correct
grammar accepts far fewer words than generic alternatives:

```
Candidate                      m   n=2m+1   |L≤n|
─────────────────────────────  ──  ───────  ────
a.b.c                          3      7       1
(a+b+c)+                       3      7    3,279
a.a.a                          3      7       1
```

At the same n=7, the correct grammar wins by a landslide (1 vs 3,279).
Tie-breaking by shortest expression handles the `a.a.a` overfit case.

### Why Per-Candidate n Breaks In Our Setting

In our setting, we have **no known target**. Candidates come from different
algorithms (CRX, iDRegEx) and have different `model_cost` values. When we use
per-candidate `n`:

```
Candidate                           m   n=2m+1   |L≤n|
──────────────────────────────────  ──  ───────  ────
info+                               1      3       3
info.file.template.shell.service+   5     11       7
```

`info+` wins (3 < 7) — not because it's better, but because it's evaluated on
a smaller range (lengths 0..3 vs 0..11). The generic grammar gets a free pass
by having a smaller `n`.

This is the same bug as MDL: `model_cost('info+') = 1` is tiny, so MDL also
picks `info+`. Different disguise, same problem.

## Our Adaptation

### What We Changed

Instead of counting at 0..n (per-candidate), we count at **exactly the lengths
present in the data**:

```python
def lang_size_score(expr, sequences):
    if not sequences:
        return lang_size(expr, 2 * model_cost(expr) + 1)  # paper's formula
    total = 0
    for seq in sequences:
        total += _count_words_fast(expr, len(seq))
    return total
```

Both grammars are evaluated on the same lengths (the observed data). The grammar
accepting the fewest words at those lengths genuinely wins.

### Why This Is Correct

The Language Size measure answers: "which grammar adds the fewest spurious words
to the data?" When we count at the data's lengths, we measure exactly this:
how many words does the grammar accept at the lengths we actually observe?

A grammar that accepts many words at each observed length (like `(a+b+c)+`)
adds many spurious words. A grammar that accepts few words (like `a.b.c`) adds
few spurious words. The one adding the fewest is the most specific to the data.

### When The Paper's Formula And Our Adaptation Agree

With diverse sequence lengths, both approaches agree:

```
Sequences: [['a','b','c'], ['a','b'], ['a','c'], ['b','c']]

Candidate              Paper |L≤n|   Our score
─────────────────────  ────────────  ─────────
a.b.c                          1         1
(a+b+c)+                   3,279        54
a.(b+c)?                       3         6
```

Both rank `a.b.c` best. Good.

### When They Disagree

With the `info+` scenario (per-candidate n):

```
Sequences: 5x ['info', 'file', 'template', 'shell', 'service']

Candidate                           Paper |L≤n|   Our score
──────────────────────────────────  ────────────  ─────────
info+                                       3         5
info.file.template.shell.service+           7         5
```

Paper picks `info+` (3 < 7) — WRONG. Our adaptation ties (5 = 5) — HONEST.

### The Remaining Limitation

When all sequences have the same length (e.g. all length 5), both `info+` and
the specific grammar accept exactly 1 word at length 5. They tie — the scoring
metric can't distinguish them. This is **honest**: neither grammar is better
for this data.

The issue is in the **inference step** (CRX producing equivalent grammars for
identical sequences), not the scoring step. In practice, CRX produces
`info.file.template.shell.service` (no `+`) for5 identical sequences, so
`info+` never appears as a candidate.

With diverse sequence lengths, our adaptation correctly differentiates:
`info+` accepts 1 word at each length (total = number of sequences), while
`(a+b+c)+` accepts many words at each length (total = Σ alphabet_size^length).

## Test Matrix

| Scenario | Sequences | Expected winner | Why |
|----------|-----------|----------------|-----|
| Specific vs generic, diverse lengths | `['a','b','c'], ['a','b'], ['a','c']` | `a.b.c` | Accepts 1 word at length 3, 0 at lengths 1-2 |
| Specific vs generic, identical lengths | 5x `['info','file','template','shell','service']` | TIE | Both accept 1 word at length 5 |
| Generic vs more generic | `['a','b','c']` × 5 | `(a+b+c)+` wins over `a+` | `a+` accepts 1 word at each length, `(a+b+c)+` accepts 3^L |
| MDL failure case | `['info','file','template','shell','service']` × 5 | `info+` wins MDL, TIE on langsize | MDL rewards short expressions |
| Empty sequences | `[]` | Falls back to paper formula | No data to evaluate at |
| Single sequence | `[['a','b','c']]` | `a.b.c` wins | Accepts 1 word at length 3 |
| Long sequences | `[['a','b','c','d','e']]` | `a.b.c.d.e` wins | Accepts 1 word at length 5 |

## Files Changed

- `bex/mdl.py`: Added `lang_size_score`, `score_grammar`, `_SCORERS` registry
- `bex/ensemble.py`: `mdl_score_simple` uses `langsize` by default, `infer_ensemble` accepts `method=`
- `bex/mcp_server.py`: Both tools accept `method` parameter
- `bex/tag_preprocessor/analyze.py`: Full call chain threads `method` through
- `docs/adr/0013-language-size-scoring.md`: ADR documenting the decision
- `tests/test_kore.py`: 5 new tests for Language Size scoring