grammar-inference-engine/docs/language-size-scoring-analysis.md

Language Size Scoring: Analysis and Design Notes

Date: 2026-07-11

Problem Statement

Our ensemble grammar inference uses a scoring function to select the best grammar when multiple algorithms (CRX, iDRegEx, kORE) produce candidates. The scoring function determines which grammar wins.

The concrete problem: on real codebases, the ensemble picked overly generic grammars like info+ over specific ones like info.file.template.shell.service+. The generic grammar accepts an astronomically large language (every repetition of info at every length), while the specific grammar accepts exactly one word of each length ≥ 5. Any human would pick the specific one. Our scorer picked the generic one.

Root Cause Analysis

What MDL Measures

Our old scoring function was MDL (Minimum Description Length):

def mdl_score(expr, sequences):
    model = model_cost(expr)   # number of symbol occurrences in expression
    data = data_cost(expr, sequences)  # Σ log₂(|L(r)| at seq length)
    return model + data

model_cost counts how many times alphabet symbols appear in the expression. For info+, that's 1 (the symbol info appears once). For info.file.template.shell.service+, that's 5. MDL rewards short expressions.

data_cost sums log₂(|L(r)|_len(seq)) over all sequences. Both info+ and the specific grammar accept exactly 1 word at length 5, so both get data_cost = 5 × log₂(1) = 0.

Result: mdl_score('info+') = 1 + 0 = 1.0, mdl_score('specific') = 5 + 0 = 5.0. MDL picks info+ because its model cost is tiny.

Why MDL Fails Here

MDL combines two signals: expression length (model cost) and compression quality (data cost). When model cost dominates (as it does when data cost is 0 for exact matches), short generic patterns win. This is the same problem the Bex paper identified: MDL achieves only 21% success rate vs 98% for Language Size.

The Bex Paper's Language Size Measure

Paper: arXiv 1004.2372, Section 4.3.1

Bex, Goethals, Penninckx, Van Gucht, Van den Bussche published "Learning Deterministic Regular Expressions for the Inference of Schemas from XML Data" (VLDB 2007, arXiv:1004.2372).

They proposed the iDRegEx algorithm and evaluated two scoring measures:

1. Language Size (Section 4.3.1): select the expression that accepts the fewest words up to length n.
2. MDL (Section 4.3.2): model cost + data cost, based on Adriaans & Vitányi (2006).

Their result (Section 5, line 1470): on a corpus of synthetic regular expressions with alphabet size 5, Language Size achieved 98% success rate while MDL achieved only 21%. They explicitly abandoned MDL: "Therefore in the remainder of this section we only consider iDRegEx with the language size criterion."

The Paper's Formula

The paper defines:

"We therefore only consider the words up to a length n, where n = 2m + 1 with m the length of the candidate expression, excluding regular expression operators, ∅, and ε."

"Then the best candidate in C is the one with the least value of |L(r)≤n|."

Concretely: for candidate r, compute m = model_cost(r) (symbol occurrences only, no operators), then n = 2m + 1. Count all words in L(r) of length ≤ n. Pick the candidate with the smallest count. Tie-break: pick the shortest expression.

Why The Paper's Formula Works In Their Setting

The paper evaluates candidates against a known target. The experiment is:

1. Start with a target expression (e.g. a.b.c)
2. Generate sample S from the target (words the target accepts)
3. Run iDRegEx on S to produce candidate set C
4. Score each candidate, pick the best
5. Check if best matches the target

The critical detail: all candidates are derived from the same target, so they have similar model_cost values and similar n. At the same n, the correct grammar accepts far fewer words than generic alternatives:

Candidate                      m   n=2m+1   |L≤n|
─────────────────────────────  ──  ───────  ────
a.b.c                          3      7       1
(a+b+c)+                       3      7    3,279
a.a.a                          3      7       1

At the same n=7, the correct grammar wins by a landslide (1 vs 3,279). Tie-breaking by shortest expression handles the a.a.a overfit case.

Why Per-Candidate n Breaks In Our Setting

In our setting, we have no known target. Candidates come from different algorithms (CRX, iDRegEx) and have different model_cost values. When we use per-candidate n:

Candidate                           m   n=2m+1   |L≤n|
──────────────────────────────────  ──  ───────  ────
info+                               1      3       3
info.file.template.shell.service+   5     11       7

info+ wins (3 < 7) — not because it's better, but because it's evaluated on a smaller range (lengths 0..3 vs 0..11). The generic grammar gets a free pass by having a smaller n.

This is the same bug as MDL: model_cost('info+') = 1 is tiny, so MDL also picks info+. Different disguise, same problem.

Our Adaptation

What We Changed

Instead of counting at 0..n (per-candidate), we count at exactly the lengths present in the data:

def lang_size_score(expr, sequences):
    if not sequences:
        return lang_size(expr, 2 * model_cost(expr) + 1)  # paper's formula
    total = 0
    for seq in sequences:
        total += _count_words_fast(expr, len(seq))
    return total

Both grammars are evaluated on the same lengths (the observed data). The grammar accepting the fewest words at those lengths genuinely wins.

Why This Is Correct

The Language Size measure answers: "which grammar adds the fewest spurious words to the data?" When we count at the data's lengths, we measure exactly this: how many words does the grammar accept at the lengths we actually observe?

A grammar that accepts many words at each observed length (like (a+b+c)+) adds many spurious words. A grammar that accepts few words (like a.b.c) adds few spurious words. The one adding the fewest is the most specific to the data.

When The Paper's Formula And Our Adaptation Agree

With diverse sequence lengths, both approaches agree:

Sequences: [['a','b','c'], ['a','b'], ['a','c'], ['b','c']]

Candidate              Paper |L≤n|   Our score
─────────────────────  ────────────  ─────────
a.b.c                          1         1
(a+b+c)+                   3,279        54
a.(b+c)?                       3         6

Both rank a.b.c best. Good.

When They Disagree

With the info+ scenario (per-candidate n):

Sequences: 5x ['info', 'file', 'template', 'shell', 'service']

Candidate                           Paper |L≤n|   Our score
──────────────────────────────────  ────────────  ─────────
info+                                       3         5
info.file.template.shell.service+           7         5

Paper picks info+ (3 < 7) — WRONG. Our adaptation ties (5 = 5) — HONEST.

The Remaining Limitation

When all sequences have the same length (e.g. all length 5), both info+ and the specific grammar accept exactly 1 word at length 5. They tie — the scoring metric can't distinguish them. This is honest: neither grammar is better for this data.

The issue is in the inference step (CRX producing equivalent grammars for identical sequences), not the scoring step. In practice, CRX produces info.file.template.shell.service (no +) for5 identical sequences, so info+ never appears as a candidate.

With diverse sequence lengths, our adaptation correctly differentiates: info+ accepts 1 word at each length (total = number of sequences), while (a+b+c)+ accepts many words at each length (total = Σ alphabet_size^length).

Test Matrix

Scenario	Sequences	Expected winner	Why
Specific vs generic, diverse lengths	['a','b','c'], ['a','b'], ['a','c']	a.b.c	Accepts 1 word at length 3, 0 at lengths 1-2
Specific vs generic, identical lengths	5x ['info','file','template','shell','service']	TIE	Both accept 1 word at length 5
Generic vs more generic	['a','b','c'] × 5	(a+b+c)+ wins over a+	a+ accepts 1 word at each length, (a+b+c)+ accepts 3^L
MDL failure case	['info','file','template','shell','service'] × 5	info+ wins MDL, TIE on langsize	MDL rewards short expressions
Empty sequences	[]	Falls back to paper formula	No data to evaluate at
Single sequence	[['a','b','c']]	a.b.c wins	Accepts 1 word at length 3
Long sequences	[['a','b','c','d','e']]	a.b.c.d.e wins	Accepts 1 word at length 5

Files Changed

• bex/mdl.py: Added lang_size_score, score_grammar, _SCORERS registry
• bex/ensemble.py: mdl_score_simple uses langsize by default, infer_ensemble accepts method=
• bex/mcp_server.py: Both tools accept method parameter
• bex/tag_preprocessor/analyze.py: Full call chain threads method through
• docs/adr/0013-language-size-scoring.md: ADR documenting the decision
• tests/test_kore.py: 5 new tests for Language Size scoring