263 lines
10 KiB
Markdown
263 lines
10 KiB
Markdown
# Dervish: Discovering Unwritten Conventions with Grammar Inference
|
||
|
||
<p align="center"><img src="dervish.gif" alt="Dervish" width="200"></p>
|
||
|
||
**How we turned 36 Ansible roles into a 200-character grammar — and why
|
||
it matters for LLM agents.**
|
||
|
||
## The problem
|
||
|
||
Every codebase has unwritten conventions. Your team's Docker Compose
|
||
files always put `image` before `ports` before `volumes`. Your Ansible
|
||
deploy roles always start with `assert`, then `file`, then `template`.
|
||
Your CI pipelines always run `lint` before `test` before `deploy`.
|
||
|
||
Nobody writes these down. They're emergent — copied from role to role,
|
||
file to file, until they become a tacit standard.
|
||
|
||
When an LLM agent needs to generate new content that follows these
|
||
conventions, you have two options:
|
||
|
||
1. **Stuff every existing file into context** — 36 deploy roles = 15,000
|
||
tokens. You'll hit the context window on your third example.
|
||
2. **Give it one or two examples and hope** — the LLM will guess the
|
||
pattern, and it will often guess wrong.
|
||
|
||
Neither is good. The first is wasteful. The second is unreliable.
|
||
|
||
What you really want is the **compiled convention** — the minimal
|
||
description of what all 36 roles share, expressed in ~200 tokens. An
|
||
LLM can follow a rule in 200 tokens far more reliably than it can
|
||
infer a pattern from 36 examples.
|
||
|
||
This is grammar inference.
|
||
|
||
## The approach
|
||
|
||
Given a set of example sequences over some alphabet (e.g., Ansible
|
||
module names, Docker Compose keys, CI job names), learn a regular
|
||
expression that describes the general pattern.
|
||
|
||
We implemented two algorithms from Bex et al., a pair of papers from
|
||
TODS 2010 and arXiv 2010:
|
||
|
||
- **CRX** (TODS 2010 §6): A single-pass algorithm that builds a
|
||
predecessor relation over symbols, computes equivalence classes,
|
||
and emits a Chain Regular Expression (CHARE) that matches ALL
|
||
input sequences. Fast, deterministic, captures the full vocabulary.
|
||
|
||
- **iDRegEx** (arXiv 2010): A probabilistic algorithm using k-testable
|
||
Observation Automata (k-OA) trained with Baum-Welch EM. It finds
|
||
only the *minimal common core* — the symbols that appear in every
|
||
example. Robust against noise, but fails (returns ∅) when the
|
||
examples are too diverse.
|
||
|
||
Both run in the **ensemble**: CRX produces a permissive grammar (full
|
||
vocabulary, many optional parts), iDRegEx produces a strict grammar
|
||
(minimal core). A Minimum Description Length (MDL) score picks the
|
||
winner: the grammar that compresses the data best.
|
||
|
||
## The algorithms, briefly
|
||
|
||
### CRX — Chain Regular Expression inference
|
||
|
||
CRX (Algorithm 7, TODS 2010) works in four steps:
|
||
|
||
1. **Build the immediate-predecessor relation.** For every adjacent
|
||
pair (x, y) across all sequences, record that x precedes y. If
|
||
symbol `assert` always appears before `file`, record
|
||
`assert → file`.
|
||
|
||
2. **Compute equivalence classes.** Take the reflexive-transitive
|
||
closure of the predecessor relation. The strongly connected
|
||
components are *equivalence classes* — groups of symbols that can
|
||
appear in the same position. If `copy` and `template` both follow
|
||
`file` and precede `command`, they're in the same class.
|
||
|
||
3. **Merge singleton classes.** A class with one symbol that shares
|
||
the same predecessor/successor sets as another singleton class
|
||
gets merged. This handles symbols that always appear in the
|
||
same structural position.
|
||
|
||
4. **Topological sort.** The equivalence classes are sorted by their
|
||
position in the Hasse diagram of the predecessor relation. Each
|
||
class becomes a factor in the output, annotated with a quantifier:
|
||
- `+` (one or more) if the class forms a cycle
|
||
- `+?` (zero or more) if the class appears variably
|
||
- `?` (optional) if the class can be absent
|
||
- (exact) if the class always appears exactly once
|
||
|
||
The result is a CHARE: a sequence of factors where each factor is a
|
||
disjunction of equivalent symbols with a quantifier.
|
||
|
||
### iDRegEx — k-optimal regular expression inference
|
||
|
||
iDRegEx (Algorithm 4, arXiv 2010) uses a probabilistic automaton:
|
||
|
||
1. **Build a complete k-OA.** A k-testable Observation Automaton
|
||
records all k-grams (subsequences of length k) from the input
|
||
sequences. The automaton's states represent (k-1)-grams.
|
||
|
||
2. **Train with Baum-Welch.** EM iterations assign probabilities to
|
||
transitions, learning which paths through the automaton are most
|
||
likely given the data.
|
||
|
||
3. **Disambiguate.** Remove nondeterministic transitions — for any
|
||
state and symbol, keep only the most probable next state.
|
||
|
||
4. **Prune.** Remove low-probability edges and unreachable states,
|
||
leaving only the most likely paths.
|
||
|
||
5. **Extract with rwr².** The REWRITE-SQUARED algorithm (rwr²,
|
||
Algorithm 3) collapses the pruned automaton into a k-optimal
|
||
regular expression — the minimal common core.
|
||
|
||
### MDL scoring — picking the right level of specificity
|
||
|
||
The Minimum Description Length principle (Rissanen 1978) says: the
|
||
best grammar is the one that minimizes the sum of its own size and
|
||
the cost of encoding the data using it.
|
||
|
||
```
|
||
MDL = model_cost + data_cost
|
||
```
|
||
|
||
**model_cost** = the number of alphabet symbol occurrences in the
|
||
grammar. A grammar with 5 unique symbols used once each has
|
||
model_cost = 5.
|
||
|
||
**data_cost** = Σ log₂(|L(r)|) across all sequences, where |L(r)| is
|
||
the number of strings of length len(s) that the grammar accepts.
|
||
A grammar like `(a+b+c+...+z)+` accepts 19 possible symbols at each
|
||
position, so for a sequence of length 120, the data cost is
|
||
120 × log₂(19) ≈ 510 bits. A grammar like `a.b.c.d.e` accepts only
|
||
1 string of length 5, so data cost is 0.
|
||
|
||
The ensemble picks the grammar with the lowest total MDL. This
|
||
automatically balances specificity against coverage: a grammar that
|
||
matches only 1 sequence but does so perfectly (low data cost) can
|
||
beat a grammar that matches all sequences but is extremely permissive
|
||
(high data cost).
|
||
|
||
## The results
|
||
|
||
### Ansible deploy roles — 36 roles from companyweb
|
||
|
||
Your own deploy roles cover everything from AdGuard Home to
|
||
Woodpecker CI. They have NO schema — each is a free-form script.
|
||
|
||
```
|
||
Grammar: docker_volume+?.group?.docker_container?.user?.apt?.npm?.
|
||
(assert+...+command+copy+file+template+set_fact+...+wait_for)+?.
|
||
(cron+firewalld)?
|
||
Match: 36/36
|
||
MDL: 2186.28
|
||
```
|
||
|
||
Bottleneck analysis: optional docker setup (volume, group, container,
|
||
user, apt, npm), then a large disjunction of ~25 task modules (one or
|
||
more), then optional cron/firewalld at the end. This captures the
|
||
convention precisely.
|
||
|
||
**Compression: 36 roles (15,000 tokens) → 200 tokens (75×)**
|
||
|
||
### Geerlingguy Galaxy roles — 15 popular roles
|
||
|
||
Jeff Geerling's roles are the most popular on Ansible Galaxy. He has
|
||
never documented their structural pattern. Yet every one of the 15
|
||
follows the same arc:
|
||
|
||
```
|
||
Grammar: fail?.(include_vars+set_fact+package+file+template+service+...)+.
|
||
include+?.(npm+pip)+?.lineinfile?
|
||
Match: 15/15
|
||
MDL: 596.64
|
||
```
|
||
|
||
Check prerequisites, OS-specific variables, install packages,
|
||
configure with templates, start services, optionally run sub-tasks,
|
||
install npm/pip packages, and optionally tweak config lines.
|
||
|
||
**This is the first explicit description of the geerlingguy role
|
||
module ordering convention.** It took 15 roles and a grammar inference
|
||
algorithm to write it down.
|
||
|
||
**Compression: 15 roles (5,000 tokens) → 60 tokens (83×)**
|
||
|
||
### Ensemble dynamics
|
||
|
||
The ensemble (CRX + iDRegEx + MDL) selects different winners
|
||
depending on the data:
|
||
|
||
| Dataset | Winner | Why |
|
||
|---------|--------|-----|
|
||
| Ansible galaxy (15 roles) | CRX | iDRegEx returned ∅ (too diverse) |
|
||
| Helm prom-stack (6 configs) | **iDRegEx** | Finds minimal core across all configs |
|
||
| Terraform modules (8) | CRX | iDRegEx returned ∅ (no common core across domains) |
|
||
| Terraform modules (8) | CRX | Every resource type optional across domains |
|
||
| GitHub Actions Go lint (6) | CRX | Tight pattern, all match |
|
||
|
||
iDRegEx wins when the data has a clear common core. CRX wins when
|
||
there's no single shared subsequence (the roles share the *vocabulary*
|
||
but not the *order*).
|
||
|
||
## The MCP
|
||
|
||
The engine is exposed as an MCP server:
|
||
|
||
```python
|
||
from bex.mcp_server import infer_best_grammar
|
||
|
||
# Full coverage
|
||
output = infer_best_grammar(
|
||
sequences=role_sequences,
|
||
prefer="crx",
|
||
)
|
||
# Returns:
|
||
# Best: CRX (MDL 288)
|
||
# Grammar: fail?.(include_vars+set_fact+package+file+template+service+...)+
|
||
# .include+?.(npm+pip)+?.lineinfile?
|
||
|
||
# Ensemble — let MDL pick
|
||
output = infer_best_grammar(sequences=role_sequences)
|
||
```
|
||
|
||
An agent workflow:
|
||
|
||
1. Agent needs to write an Ansible role
|
||
2. Finds 15 existing geerlingguy roles, extracts their task module sequences
|
||
3. Calls `infer_best_grammar(sequences=..., prefer='crx')`
|
||
4. Gets back the grammar in ~60 tokens
|
||
5. Generates a new role that follows the structural pattern
|
||
|
||
Without the MCP: 15 role files in context (5,000 tokens), or guesswork.
|
||
With the MCP: one grammar rule (~60 tokens), known to match 15/15 roles.
|
||
|
||
## What it means
|
||
|
||
Grammar inference turns **examples** into **rules**. The rule is a
|
||
compressed description of the structural convention — and for
|
||
schema-less content like the geerlingguy role module ordering, this is
|
||
the *first time* the convention has been written down at all.
|
||
|
||
For LLM agents, this changes the trade-off between context and
|
||
accuracy. Instead of flooding the context window with examples, the
|
||
agent can call the MCP, get the rule in ~60 tokens, and follow it.
|
||
The rule is more reliable than guessing from examples, and it costs
|
||
less than the first example would have.
|
||
|
||
The algorithm doesn't need to understand what a deploy role does. It
|
||
doesn't know that `file` creates directories and `template` renders
|
||
Jinja2. It only needs to see 36 sequences of module names and find
|
||
the pattern they all share. The structural convention is in the data
|
||
— you just have to extract it.
|
||
|
||
## References
|
||
|
||
- Bex, G. J., Gelade, W., Neven, F., & Vansummeren, S. (2010).
|
||
[*Learning Deterministic Regular Expressions for the Web.*](https://doi.org/10.1145/1806907.1806911) TODS 2010.
|
||
- Bex, G. J., Gelade, W., Martens, W., & Neven, F. (2010).
|
||
[*Simplifying XML Schema: Single-Type Approximations of Regular
|
||
Expressions.*](https://arxiv.org/abs/1004.2372) arXiv:1004.2372.
|
||
- Rissanen, J. (1978). *Modeling by shortest data description.*
|
||
Automatica 14(5).
|