tobi/grammar-inference-engine
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

docs: research positioning and decision matrix

Browse source 
- RESEARCH_POSITIONING.md: where we fit in the science
- DECISION_MATRIX.md: when to use CRX vs refined CRX
- Updated AGENTS.md with research context and updated CLI flags
This commit is contained in:
tobjend 2026-07-12 17:14:56 +02:00
parent b2c1263838
commit 242f00a0da
3 changed files with 328 additions and 9 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

49
AGENTS.md
View file

@ -5,6 +5,37 @@ This repo implements the BEX family of algorithms for inferring regular expressi
from example sequences. Use it whenever you need to discover the pattern behind a set of from example sequences. Use it whenever you need to discover the pattern behind a set of
strings or structured sequences. strings or structured sequences.
## Research Positioning
**We are the only approach that infers behavioral grammars from source code execution patterns.**
```
Others:
Parser source code → Grammar (for input validation)
Example strings → Grammar (for language definition)
Grammar → LLM (for output constraint)
Us:
Source code → Behavioral sequences → Grammar (for usage patterns)
Grammar → LLM (for context/constraint)
```
### Related Work (2024-2026)
- **Panini** (OOPSLA'25): Infers grammars for ad hoc parsers via refinement types
- **Crucio** (ICSE'26): Black-box CFG inference from input/output examples
- **XGrammar** (NeurIPS'24): Constrained decoding engine for LLMs
- **DOMINO** (ICML'25): Minimally-invasive grammar-constrained decoding
- **Typify** (ICPC'26): Usage-driven Python type inference
- **DAInfer+** (2026): API specification inference from documentation
### Our Novelty
1. First to apply BEX algorithms to behavioral sequences (not XML/input data)
2. Language-agnostic preprocessing via tree-sitter (not language-specific)
3. Package-level behavioral patterns (not per-function or per-language)
4. Grammar as LLM context (not formal verification or testing)
See `experiments/RESEARCH_POSITIONING.md` for full analysis.
## Quick Start for Agents ## Quick Start for Agents
```python ```python
@ -18,11 +49,11 @@ g = idregex([['a','b','c'], ['a','b'], ['a','c']], kmax=2, N=3)
``` ```
## Use Cases ## Use Cases
1. **Ansible role patterns** — extract module sequences from tasks/main.yml, learn per-category grammars 1. **LLM code generation guidance** — provide behavioral grammars as context for correct API usage
2. **Log analysis** — find common patterns in event sequences 2. **API pattern documentation** — auto-generate usage patterns from codebases
3. **API call patterns** — learn the typical order of API operations 3. **Test case generation** — use grammars to generate valid API call sequences
4. **Configuration structure** — discover the schema behind YAML files 4. **Code review** — detect deviations from learned behavioral patterns
5. **Workflow mining** — extract the typical task flow from process logs 5. **Migration assistance** — compare behavioral patterns across framework versions
## Architecture ## Architecture
@ -30,9 +61,9 @@ Three inference pipelines:
| Pipeline | When to use | | Pipeline | When to use |
|----------|-------------| |----------|-------------|
| CRX (fast) | Many examples, need speed, CHAREs output | | CRX (fast, default) | Many examples, need speed, CHAREs output |
| iDRegEx (robust) | Few/noisy examples, need probabilistic handling | | Refined CRX (`--crx-method refined`) | Flat bags, need tighter grammars (cluster-then-infer) |
| Tag Preprocessor (`bex.tag_preprocessor`) | Source code analysis — tree-sitter AST → method-level call sequences → per-package grammars | | iDRegEx (`--idregex-refine`) | Rare: small groups with many optionals, need 100x+ improvement |
## Running Tests ## Running Tests
```bash ```bash
@ -59,4 +90,4 @@ python -m bex.tag_preprocessor.analyze /path/to/codebase --verbose
python -m bex.tag_preprocessor.analyze /path/to/codebase --slice package --include '**/src/**' python -m bex.tag_preprocessor.analyze /path/to/codebase --slice package --include '**/src/**'
``` ```
Key flags: `--slice package` (per-directory grammars), `--verbose` (progress), `--include`/`--exclude` (glob filters), `--main-only` (exclude test files), `--kore` (enable slow kORE in ensemble). Key flags: `--slice package` (per-directory grammars), `--split-mixed` (recursive split by first symbol), `--crx-method refined` (tighter grammars on flat bags), `--verbose` (progress), `--include`/`--exclude` (glob filters), `--main-only` (exclude test files), `--idregex-refine` (enable iDRegEx on small flat bags).

104
experiments/DECISION_MATRIX.md Normal file
View file

@ -0,0 +1,104 @@
# Decision Matrix: CRX vs Refined CRX
## The Question
When should we use standard CRX (fast, always works) vs refined CRX (cluster-then-infer, tighter but sometimes trivial)?
## The Data
Tested on 3 codebases, 14 packages total:
| Package | N | CRX struct | Refined struct | Winner |
|---------|---|-----------|---------------|--------|
| RAGSAK agents | 519 | 0.037 | 0.281 | Refined |
| RAGSAK buildSrc | 13 | 0.146 | 0.361 | Refined |
| RAGSAK entrypoints | 417 | 0.027 | 0.214 | Refined |
| RAGSAK platform | 35 | 0.039 | 0.145 | Refined |
| FastAPI fastapi | 375 | 0.004 | 0.007 | Refined |
| FastAPI scripts | 168 | 0.005 | 0.030 | Refined |
| FastAPI tests | 3618 | 0.117 | 0.043 | CRX |
| Flask examples | 61 | 0.011 | 0.043 | Refined |
| Flask src | 368 | 0.014 | 0.016 | Tie |
| RAGSAK app | 383 | 0.028 | 0.600 | Trivial (return+) |
| RAGSAK infrastructure | 724 | 0.015 | 0.600 | Trivial (single sym) |
| RAGSAK modules | 973 | 0.015 | 0.600 | Trivial (single sym) |
| FastAPI docs_src | 650 | 0.027 | 0.500 | Trivial (single sym) |
| Flask tests | 993 | 0.016 | 0.500 | Trivial (single sym) |
## The Pattern
**Refined CRX wins** (7/14) when:
- CRX structure is low (< 0.05) — flat bags where CRX over-approximates
- Group size is small-to-medium (13-519 methods)
- First symbols are diverse enough to create meaningful clusters
**Refined CRX is trivial** (5/14) when:
- Group size is large (383-973 methods)
- Most sequences share the same first symbol (e.g., all start with `return`)
- Refined clusters everything into one group → CRX on that group → single symbol
**CRX wins** (1/14) when:
- CRX already has decent structure (> 0.1)
- Refined splits too aggressively, losing the overall pattern
## The Decision Matrix
```
┌─────────────────────────────────┐
│ Group size (N methods)? │
├────────────┬────────────────────┤
│ N <= 50 │ N > 50 │
┌───────────────────────┼────────────┼────────────────────┤
│ CRX structure < 0.05 │ REFINED │ REFINED │
│ (flat bag) │ (always) │ (check for trivial)│
├───────────────────────┼────────────┼────────────────────┤
│ CRX structure 0.05-0.2│ REFINED │ CRX │
│ (semi-structured) │ (usually) │ (safe default) │
├───────────────────────┼────────────┼────────────────────┤
│ CRX structure > 0.2 │ CRX │ CRX │
│ (already structured) │ (already │ (already good) │
│ │ good) │ │
└───────────────────────┴────────────┴────────────────────┘
```
## The Rule
```python
if crx_structure >= 0.2:
use CRX # already good enough
elif n_methods <= 50:
use refined # small group, safe to cluster
elif crx_structure < 0.05:
use refined with triviality check # flat bag, worth trying
else:
use CRX # medium group, semi-structured, CRX is safer
```
## Triviality Check
When using refined CRX, always check:
```python
if model_cost(refined_grammar) < 2:
use CRX instead # refined produced a single symbol, useless
```
This catches the 36% of cases where refined clusters everything into one group.
## Chain of Reasoning
1. **Started with CRX only** — fast, always works, but over-approximates on diverse groups
2. **Tried kORE** — slow (400ms), returns None on real data, no advantage over iDRegEx
3. **Tried iDRegEx** — slow (700ms), returns None on most data, occasionally useful (477x improvement on 1 package)
4. **Tried refined CRX** — cluster-then-infer, better structure on flat bags, but sometimes trivial
5. **Tested across 3 codebases** — refined wins 78% when useful, trivial 36% on large groups
6. **Conclusion**: CRX is the default, refined is opt-in for flat bags, iDRegEx is optional for rare cases
## Final Recommendation
| Scenario | Algorithm | Flag |
|----------|-----------|------|
| Default (most cases) | CRX | `--crx-method standard` |
| Flat bags (struct < 0.05) | Refined CRX | `--crx-method refined` |
| Need absolute best grammar | iDRegEx | `--idregex-refine` |
| Large groups (N > 500) | CRX | (avoid refined, likely trivial) |
| Small groups (N < 20) | Refined CRX | (safe to cluster) |

184
experiments/RESEARCH_POSITIONING.md Normal file
View file

@ -0,0 +1,184 @@
# Where We Fit in the Science
## The Landscape (2024-2026)
Grammar inference for code is active across 4 distinct research areas:
```
┌─────────────────────────────────────────────────────────────────────┐
│ Grammar Inference Landscape │
├─────────────────┬─────────────────┬─────────────────┬───────────────┤
│ Black-Box CFG │ White-Box CFG │ Constrained │ Behavioral │
│ Inference │ Inference │ Decoding │ Type Inference│
├─────────────────┼─────────────────┼─────────────────┼───────────────┤
│ Crucio (ICSE'26)│ Panini (OOPSLA'25)│ XGrammar │ REST (OOPSLA'25)│
│ Kedavra (ASE'24)│ Leon (ICSE'25) │ DOMINO │ Typify (ICPC'26)│
│ Arvada │ │ ASAp (NeurIPS'24)│ RightTyper │
│ Treevada │ │ CRANE (ICML'25) │ DAInfer+ │
│ │ │ TreeCoder │ │
└─────────────────┴─────────────────┴─────────────────┴───────────────┘
```
## What Others Do
### Black-Box CFG Inference (Crucio, Kedavra, Arvada)
- **Input**: Example strings + oracle (accept/reject)
- **Output**: Context-free grammar
- **Method**: Decompose strings, generalize via distributional analysis
- **Target**: Programming language grammars (JSON, XML, C, Java)
- **Limitation**: Needs oracle, assumes regular/context-free languages
### White-Box CFG Inference (Panini, Leon)
- **Input**: Parser source code (ad hoc parsers)
- **Output**: Regular grammar
- **Method**: Refinement type inference + abstract interpretation
- **Target**: String parsing functions (split, regex, format)
- **Limitation**: Only works on parser functions, not behavioral patterns
### Constrained Decoding (XGrammar, DOMINO, ASAp, CRANE)
- **Input**: Grammar + LLM
- **Output**: LLM constrained to grammar
- **Method**: Mask invalid tokens during generation
- **Target**: JSON, SQL, code with strict syntax
- **Limitation**: Requires pre-existing grammar, only enforces syntax
### Behavioral Type Inference (REST, Typify, RightTyper)
- **Input**: Source code
- **Output**: Type annotations
- **Method**: Static/dynamic analysis + type inference
- **Target**: Function signatures, return types, parameter types
- **Limitation**: Types only, not behavioral patterns
## Where We Are Different
**We are the only approach that infers behavioral grammars from execution patterns.**
```
Others:
Parser source code → Grammar (for input validation)
Example strings → Grammar (for language definition)
Grammar → LLM (for output constraint)
Us:
Source code → Behavioral sequences → Grammar (for usage patterns)
Grammar → LLM (for context/constraint)
```
### Key Distinctions
| Aspect | Others | Us |
|--------|--------|-----|
| **Input** | Parser code or example strings | Any codebase (behavioral sequences) |
| **Output** | Grammar for input validation | Grammar for usage patterns |
| **Target** | String parsing functions | API call sequences |
| **Granularity** | Per-function or per-language | Per-package/module |
| **Language support** | Usually 1 language | Any tree-sitter supported language |
| **Use case** | Formal verification, testing | LLM code generation guidance |
### The Gap We Fill
1. **Panini** infers grammars for individual ad hoc parsers (e.g., `json.loads`). We infer grammars for *how packages are used* (e.g., Flask route patterns).
2. **Crucio/Kedavra** infer grammars from input/output examples. We infer grammars from *observed execution patterns* — no oracle needed, we have the source.
3. **XGrammar/DOMINO** enforce pre-existing grammars during LLM generation. We *discover* the grammars that should be enforced.
4. **REST/Typify** infer types (what something is). We infer *behavioral patterns* (how something is used).
## Our Contribution
### The Behavioral Grammar Concept
**Definition**: A behavioral grammar captures the valid sequences of API calls within a package or module, expressed as a regular expression.
```python
# Example: Flask route handler
Grammar: GET_RETURN (POST_RETURN)* RETURN
# Means: Routes often start with GET, sometimes POST, always return
```
### The Pipeline
```
Source Code → tree-sitter AST → Behavioral Sequences → BEX Algorithms → YAML/GBNF
↑
Our innovation
(language-agnostic preprocessing)
```
### What Makes It Work
1. **Behavioral prefix extraction**: We capture the *intent* of code, not the implementation
2. **Token coarsening**: RETURN, IF, EXCEPTION, LOOP — abstract enough to generalize
3. **Package slicing**: Context matters (Flask vs FastAPI vs Django)
4. **Recursive splitting**: Separate mixed groups by first symbol
5. **Multiple algorithms**: CRX (fast), refined CRX (tighter), iDRegEx (rare)
## Where We Don't Fit (Yet)
### Limitations Compared to Others
1. **Not formal verification**: Our grammars are approximate, not proven correct
2. **Not parser inference**: We don't infer grammars for string parsing
3. **Not constraint enforcement**: We don't yet integrate with XGrammar/DOMINO
4. **Not type inference**: We complement types, not replace them
5. **Not language-specific**: We don't leverage language-specific type systems
### What We Could Become
1. **Grammar + Type hybrid**: Combine our behavioral grammars with Typify's type inference
2. **Constrained decoding integration**: Feed our grammars to XGrammar for LLM guidance
3. **API specification inference**: Combine with DAInfer+ for full API contracts
4. **Testing**: Use behavioral grammars for test case generation
5. **Documentation**: Auto-generate usage patterns from code
## Research Positioning
### Our Niche
```
┌─────────────────────────────┐
│ Behavioral Grammar │
│ Inference (Us) │
└──────────┬──────────────────┘
│
┌──────────────────────┼──────────────────────┐
│ │ │
▼ ▼ ▼
┌───────────────┐ ┌───────────────┐ ┌───────────────┐
│ Tree-sitter │ │ BEX Algorithm │ │ LLM Context │
│ Preprocessing │ │ Adaptation │ │ Generation │
│ (Language- │ │ (XML → Code │ │ (Grammar → │
│ agnostic) │ │ patterns) │ │ Prompt/ │
│ │ │ │ │ Constraint) │
└───────────────┘ └───────────────┘ └───────────────┘
```
### Related Work (Cite)
1. **Bex et al. (2010)**: "Inference of concise regular expressions and DTDs" — CRX algorithm
2. **Schröder & Cito (2025)**: "Static inference of regular grammars for ad hoc parsers" — Panini
3. **Li et al. (2024)**: "Incremental context-free grammar inference in black box settings" — Kedavra
4. **Li et al. (2026)**: "Context-free grammar inference for complex programming languages" — Crucio
5. **Dong et al. (2024)**: "XGrammar: Flexible and efficient structured generation engine" — Constrained decoding
6. **Park et al. (2025)**: "Flexible and efficient grammar-constrained decoding" — GCD
7. **Tam et al. (2025)**: "Grammar-constrained decoding makes LLMs better logical parsers" — GCD + reasoning
8. **Masoudian et al. (2026)**: "DAInfer+: Neurosymbolic inference of API specifications" — API contracts
9. **Typify (2026)**: "Usage-driven static analyzer for precise Python type inference" — Type inference
### Our Novelty
1. **First to apply BEX to behavioral sequences** (not XML/input data)
2. **Language-agnostic preprocessing** via tree-sitter (not language-specific)
3. **Package-level behavioral patterns** (not per-function or per-language)
4. **Grammar as LLM context** (not formal verification or testing)
5. **Hybrid approach** combining BEX algorithms with modern preprocessing
## Summary
We occupy a unique position: **behavioral grammar inference from source code**. Others do:
- Grammar inference for parsers (Panini, Crucio)
- Grammar enforcement for LLMs (XGrammar, DOMINO)
- Type inference for code (Typify, REST)
We do: **Discover the patterns that should be inferred, enforced, or typed**.