Grammar Inference Engine — Agent Guide

Overview

This repo implements the BEX family of algorithms for inferring regular expression grammars from example sequences. Use it whenever you need to discover the pattern behind a set of strings or structured sequences.

Quick Start for Agents

# Fast pattern inference
from bex.crx import CRX
g = CRX().infer([['a','b','c'], ['a','b'], ['a','c']])  # a.(b+c)?

# Probabilistic k-ORE inference (handles noise better)
from bex.idregex import idregex
g = idregex([['a','b','c'], ['a','b'], ['a','c']], kmax=2, N=3)

Use Cases

Ansible role patterns — extract module sequences from tasks/main.yml, learn per-category grammars
Log analysis — find common patterns in event sequences
API call patterns — learn the typical order of API operations
Configuration structure — discover the schema behind YAML files
Workflow mining — extract the typical task flow from process logs

Architecture

Three inference pipelines:

Pipeline	When to use
CRX (fast)	Many examples, need speed, CHAREs output
iDRegEx (robust)	Few/noisy examples, need probabilistic handling
Tag Preprocessor (`bex.tag_preprocessor`)	Source code analysis — tree-sitter AST → method-level call sequences → per-package grammars

Running Tests

python -m pytest tests/

MCP Server

The primary interface is an MCP server exposing a single tool:

Tool	Parameters	What it does
`infer_best_grammar`	`sequences`, `prefer`, `kmax`, `N`, `min_coverage`	Runs CRX + iDRegEx, picks best by MDL. `prefer='crx'` or `prefer='idregex'` skips ensemble. `min_coverage < 1.0` runs core+outlier analysis.

Start it: python /path/to/bex/mcp_server.py, then connect any MCP client.

Tag Preprocessor CLI

For analyzing source code directories:

python -m bex.tag_preprocessor.analyze /path/to/codebase --verbose
python -m bex.tag_preprocessor.analyze /path/to/codebase --slice package --include '**/src/**'

Key flags: --slice package (per-directory grammars), --verbose (progress), --include/--exclude (glob filters), --kore (enable slow kORE in ensemble).

2.2 KiB Raw Blame History