grammar-inference-engine/blog_post.md

Discovering Unwritten Conventions with Grammar Inference

How we turned 36 Ansible roles into a 200-character grammar — and why it matters for LLM agents.

The problem

Every codebase has unwritten conventions. Your team's Docker Compose files always put image before ports before volumes. Your Ansible deploy roles always start with assert, then file, then template. Your CI pipelines always run lint before test before deploy.

Nobody writes these down. They're emergent — copied from role to role, file to file, until they become a tacit standard.

When an LLM agent needs to generate new content that follows these conventions, you have two options:

1. Stuff every existing file into context — 36 deploy roles = 15,000 tokens. You'll hit the context window on your third example.
2. Give it one or two examples and hope — the LLM will guess the pattern, and it will often guess wrong.

Neither is good. The first is wasteful. The second is unreliable.

What you really want is the compiled convention — the minimal description of what all 36 roles share, expressed in ~200 tokens. An LLM can follow a rule in 200 tokens far more reliably than it can infer a pattern from 36 examples.

This is grammar inference.

The approach

Given a set of example sequences over some alphabet (e.g., Ansible module names, Docker Compose keys, CI job names), learn a regular expression that describes the general pattern.

We implemented two algorithms from Bex et al., a pair of papers from TODS 2010 and arXiv 2010:

• CRX (TODS 2010 §6): A single-pass algorithm that builds a predecessor relation over symbols, computes equivalence classes, and emits a Chain Regular Expression (CHARE) that matches ALL input sequences. Fast, deterministic, captures the full vocabulary.

• iDRegEx (arXiv 2010): A probabilistic algorithm using k-testable Observation Automata (k-OA) trained with Baum-Welch EM. It finds only the minimal common core — the symbols that appear in every example. Robust against noise, but fails (returns ∅) when the examples are too diverse.

Both run in the ensemble: CRX produces a permissive grammar (full vocabulary, many optional parts), iDRegEx produces a strict grammar (minimal core). A Minimum Description Length (MDL) score picks the winner: the grammar that compresses the data best.

The algorithms, briefly

CRX — Chain Regular Expression inference

CRX (Algorithm 7, TODS 2010) works in four steps:

1. Build the immediate-predecessor relation. For every adjacent pair (x, y) across all sequences, record that x precedes y. If symbol assert always appears before file, record assert → file.

2. Compute equivalence classes. Take the reflexive-transitive closure of the predecessor relation. The strongly connected components are equivalence classes — groups of symbols that can appear in the same position. If copy and template both follow file and precede command, they're in the same class.

3. Merge singleton classes. A class with one symbol that shares the same predecessor/successor sets as another singleton class gets merged. This handles symbols that always appear in the same structural position.

4. Topological sort. The equivalence classes are sorted by their position in the Hasse diagram of the predecessor relation. Each class becomes a factor in the output, annotated with a quantifier:

   • + (one or more) if the class forms a cycle
   • +? (zero or more) if the class appears variably
   • ? (optional) if the class can be absent
   • (exact) if the class always appears exactly once

The result is a CHARE: a sequence of factors where each factor is a disjunction of equivalent symbols with a quantifier.

iDRegEx — k-optimal regular expression inference

iDRegEx (Algorithm 4, arXiv 2010) uses a probabilistic automaton:

1. Build a complete k-OA. A k-testable Observation Automaton records all k-grams (subsequences of length k) from the input sequences. The automaton's states represent (k-1)-grams.

2. Train with Baum-Welch. EM iterations assign probabilities to transitions, learning which paths through the automaton are most likely given the data.

3. Disambiguate. Remove nondeterministic transitions — for any state and symbol, keep only the most probable next state.

4. Prune. Remove low-probability edges and unreachable states, leaving only the most likely paths.

5. Extract with rwr². The REWRITE-SQUARED algorithm (rwr², Algorithm 3) collapses the pruned automaton into a k-optimal regular expression — the minimal common core.

MDL scoring — picking the right level of specificity

The Minimum Description Length principle (Rissanen 1978) says: the best grammar is the one that minimizes the sum of its own size and the cost of encoding the data using it.

MDL = model_cost + data_cost

model_cost = the number of alphabet symbol occurrences in the grammar. A grammar with 5 unique symbols used once each has model_cost = 5.

data_cost = Σ log₂(|L(r)|) across all sequences, where |L(r)| is the number of strings of length len(s) that the grammar accepts. A grammar like (a+b+c+...+z)+ accepts 19 possible symbols at each position, so for a sequence of length 120, the data cost is 120 × log₂(19) ≈ 510 bits. A grammar like a.b.c.d.e accepts only 1 string of length 5, so data cost is 0.

The ensemble picks the grammar with the lowest total MDL. This automatically balances specificity against coverage: a grammar that matches only 1 sequence but does so perfectly (low data cost) can beat a grammar that matches all sequences but is extremely permissive (high data cost).

The bugs we found (and fixed)

Implementing the BEX algorithms faithfully required solving several subtle problems.

Bug 1: model_cost counted characters, not symbols

The paper defines model_cost as "the length of r" — the number of symbols in the expression. For the toy alphabet {a, b, c, d, e} used in the paper, characters and symbols are the same. For real-world symbols like community.docker.docker_image, they aren't.

Our model_cost function was counting characters (226 for a typical grammar), when it should count symbol occurrences (19). This massively inflated the MDL score, making CRX appear worse than it actually was.

Fix: Count occurrences of alphabet symbols in the expression using regex word-boundary matching, not string length.

Bug 2: Dispatch order in _count_words_fast

The recursive function _count_words_fast estimates |L(r)| — the number of strings a grammar accepts at a given length. It dispatches on expression structure: first check for concatenation (.), then trailing quantifiers (+?, *, ?, +), then disjunction groups.

Our dispatch checked endswith('+?') before checking '.' in expr. For the expression (All)+.Role?.RoleBinding?.Job+?, the trailing +? on Job+? triggered the quantifier branch first, applying the +? to the entire expression instead of just the Job factor.

Fix: Check concatenation first. Top-level dots can only appear in concatenation, so they should be handled before any quantifier logic.

Bug 3: Greedy matching without backtracking

The _match_tokens function checked whether a sequence matches a grammar. For quantifiers like +? (zero-or-more), it greedily consumed ALL consecutive matching symbols, then moved on. This failed for grammars like a+?.a on input ['a', 'a']: the a+? ate both as, and there was nothing left for the second .a.

Fix: Replace the single-pass greedy matching with _match_possible, a proper backtracking engine that enumerates ALL valid end positions for each token and picks the maximum. This is essentially a tiny regex engine — but limited to the CHARE subset, so it avoids the exponential blowup of general regex matching.

Bug 4: Dot-splitting inside disjunctions

Module names like community.docker.docker_image contain dots. When _parse_parts processed a disjunction child, it recursively called itself — which split the expression on . before treating it as a symbol. The symbol community.docker.docker_image became community then docker then docker_image — three concatenated symbols instead of one.

Fix: Disjunction children are always flat symbols (CRX and iDRegEx don't produce nested disjunctions in practice). Parse them with _parse_flat_symbol, which strips quantifiers but never splits on ..

The results

Ansible deploy roles — 36 roles from companyweb

Your own deploy roles cover everything from AdGuard Home to Woodpecker CI. They have NO schema — each is a free-form script.

Grammar: docker_volume+?.group?.docker_container?.user?.apt?.npm?.
         (assert+...+command+copy+file+template+set_fact+...+wait_for)+?.
         (cron+firewalld)?
Match:   36/36
MDL:     2186.28

Bottleneck analysis: optional docker setup (volume, group, container, user, apt, npm), then a large disjunction of ~25 task modules (one or more), then optional cron/firewalld at the end. This captures the convention precisely.

Compression: 36 roles (15,000 tokens) → 200 tokens (75×)

Geerlingguy Galaxy roles — 15 popular roles

Jeff Geerling's roles are the most popular on Ansible Galaxy. He has never documented their structural pattern. Yet every one of the 15 follows the same arc:

Grammar: fail?.(include_vars+set_fact+package+file+template+service+...)+.
         include+?.(npm+pip)+?.lineinfile?
Match:   15/15
MDL:     596.64

Check prerequisites, OS-specific variables, install packages, configure with templates, start services, optionally run sub-tasks, install npm/pip packages, and optionally tweak config lines.

This is the first explicit description of the geerlingguy role convention. It took 15 roles and a grammar inference algorithm to write it down.

Compression: 15 roles (5,000 tokens) → 60 tokens (83×)

Docker Compose — by project

Docker Compose has a flexible schema, but each project develops its own convention:

mcp-deployment (36 services):

(build+image).command.(environment+volumes)?.ports

files (6 services):

image.environment.volumes.network_mode.privileged?.cap_add?

fresh-ape-base (9 services):

image.ports?.(depends_on+environment+user+volumes)+

Ensemble dynamics

The ensemble (CRX + iDRegEx + MDL) selects different winners depending on the data:

Dataset	Winner	Why
Ansible deploy (36 roles)	CRX	iDRegEx returned ∅ (too diverse)
Ansible galaxy (15 roles)	CRX	iDRegEx returned ∅ (too diverse)
Ansible restore (2 roles)	CRX	Both match all; CRX more compact
Ansible configure (4 roles)	iDRegEx	Finds minimal core include_role
Ansible manage (2 roles)	iDRegEx	Core: assert.authorized_key

iDRegEx wins when the data has a clear common core. CRX wins when there's no single shared subsequence (the roles share the vocabulary but not the order).

The MCP

The engine is exposed as an MCP server:

from bex.mcp_server import infer_best_grammar

# Full coverage
output = infer_best_grammar(
    sequences=role_sequences,
    prefer="crx",
)
# Returns:
#   Best: CRX (MDL 2186.28)
#   Grammar: docker_volume+?.group?...(assert+...+wait_for)+?.(cron+firewalld)?

# Ensemble — let MDL pick
output = infer_best_grammar(sequences=role_sequences)

An agent workflow:

1. Agent needs to write deploy role #37
2. Finds 36 existing deploy roles, extracts their task module sequences
3. Calls infer_best_grammar(sequences=..., prefer='crx')
4. Gets back the grammar in 200 tokens
5. Generates a new role that follows the structural pattern

Without the MCP: 36 role files in context (15,000 tokens), or guesswork. With the MCP: one grammar rule (200 tokens), known to match 36/36 roles.

What it means

Grammar inference turns examples into rules. The rule is a compressed description of the structural convention — and for schema-less content like Ansible roles, this may be the first time the convention has been written down at all.

For LLM agents, this changes the trade-off between context and accuracy. Instead of flooding the context window with examples, the agent can call the MCP, get the rule in ~60 tokens, and follow it. The rule is more reliable than guessing from examples, and it costs less than the first example would have.

The algorithm doesn't need to understand what a deploy role does. It doesn't know that file creates directories and template renders Jinja2. It only needs to see 36 sequences of module names and find the pattern they all share. The structural convention is in the data — you just have to extract it.

References

• Bex, G. J., Gelade, W., Neven, F., & Vansummeren, S. (2010). Learning Deterministic Regular Expressions for the Web. TODS 2010.
• Bex, G. J., Gelade, W., Martens, W., & Neven, F. (2010). Simplifying XML Schema: Single-Type Approximations of Regular Expressions. arXiv:1004.2372.
• Rissanen, J. (1978). Modeling by shortest data description. Automatica 14(5).