AGENTS.md

@@ -5,6 +5,37 @@ This repo implements the BEX family of algorithms for inferring regular expressi
 from example sequences. Use it whenever you need to discover the pattern behind a set of
 strings or structured sequences.
 
+## Research Positioning
+
+**We are the only approach that infers behavioral grammars from source code execution patterns.**
+
+```
+Others:
+  Parser source code → Grammar (for input validation)
+  Example strings → Grammar (for language definition)
+  Grammar → LLM (for output constraint)
+
+Us:
+  Source code → Behavioral sequences → Grammar (for usage patterns)
+  Grammar → LLM (for context/constraint)
+```
+
+### Related Work (2024-2026)
+- **Panini** (OOPSLA'25): Infers grammars for ad hoc parsers via refinement types
+- **Crucio** (ICSE'26): Black-box CFG inference from input/output examples
+- **XGrammar** (NeurIPS'24): Constrained decoding engine for LLMs
+- **DOMINO** (ICML'25): Minimally-invasive grammar-constrained decoding
+- **Typify** (ICPC'26): Usage-driven Python type inference
+- **DAInfer+** (2026): API specification inference from documentation
+
+### Our Novelty
+1. First to apply BEX algorithms to behavioral sequences (not XML/input data)
+2. Language-agnostic preprocessing via tree-sitter (not language-specific)
+3. Package-level behavioral patterns (not per-function or per-language)
+4. Grammar as LLM context (not formal verification or testing)
+
+See `experiments/RESEARCH_POSITIONING.md` for full analysis.
+
 ## Quick Start for Agents
 
 ```python
@@ -18,11 +49,11 @@ g = idregex([['a','b','c'], ['a','b'], ['a','c']], kmax=2, N=3)
 ```
 
 ## Use Cases
-1. **Ansible role patterns** — extract module sequences from tasks/main.yml, learn per-category grammars
+1. **LLM code generation guidance** — provide behavioral grammars as context for correct API usage
-2. **Log analysis** — find common patterns in event sequences
+2. **API pattern documentation** — auto-generate usage patterns from codebases
-3. **API call patterns** — learn the typical order of API operations
+3. **Test case generation** — use grammars to generate valid API call sequences
-4. **Configuration structure** — discover the schema behind YAML files
+4. **Code review** — detect deviations from learned behavioral patterns
-5. **Workflow mining** — extract the typical task flow from process logs
+5. **Migration assistance** — compare behavioral patterns across framework versions
 
 ## Architecture
 
@@ -30,9 +61,9 @@ Three inference pipelines:
 
 | Pipeline | When to use |
 |----------|-------------|
-| CRX (fast) | Many examples, need speed, CHAREs output |
+| CRX (fast, default) | Many examples, need speed, CHAREs output |
-| iDRegEx (robust) | Few/noisy examples, need probabilistic handling |
+| Refined CRX (`--crx-method refined`) | Flat bags, need tighter grammars (cluster-then-infer) |
-| Tag Preprocessor (`bex.tag_preprocessor`) | Source code analysis — tree-sitter AST → method-level call sequences → per-package grammars |
+| iDRegEx (`--idregex-refine`) | Rare: small groups with many optionals, need 100x+ improvement |
 
 ## Running Tests
 ```bash
@@ -59,4 +90,4 @@ python -m bex.tag_preprocessor.analyze /path/to/codebase --verbose
 python -m bex.tag_preprocessor.analyze /path/to/codebase --slice package --include '**/src/**'
 ```
 
-Key flags: `--slice package` (per-directory grammars), `--verbose` (progress), `--include`/`--exclude` (glob filters), `--main-only` (exclude test files), `--kore` (enable slow kORE in ensemble).
+Key flags: `--slice package` (per-directory grammars), `--split-mixed` (recursive split by first symbol), `--crx-method refined` (tighter grammars on flat bags), `--verbose` (progress), `--include`/`--exclude` (glob filters), `--main-only` (exclude test files), `--idregex-refine` (enable iDRegEx on small flat bags).

experiments/DECISION_MATRIX.md

@@ -0,0 +1,104 @@
+# Decision Matrix: CRX vs Refined CRX
+
+## The Question
+
+When should we use standard CRX (fast, always works) vs refined CRX (cluster-then-infer, tighter but sometimes trivial)?
+
+## The Data
+
+Tested on 3 codebases, 14 packages total:
+
+| Package | N | CRX struct | Refined struct | Winner |
+|---------|---|-----------|---------------|--------|
+| RAGSAK agents | 519 | 0.037 | 0.281 | Refined |
+| RAGSAK buildSrc | 13 | 0.146 | 0.361 | Refined |
+| RAGSAK entrypoints | 417 | 0.027 | 0.214 | Refined |
+| RAGSAK platform | 35 | 0.039 | 0.145 | Refined |
+| FastAPI fastapi | 375 | 0.004 | 0.007 | Refined |
+| FastAPI scripts | 168 | 0.005 | 0.030 | Refined |
+| FastAPI tests | 3618 | 0.117 | 0.043 | CRX |
+| Flask examples | 61 | 0.011 | 0.043 | Refined |
+| Flask src | 368 | 0.014 | 0.016 | Tie |
+| RAGSAK app | 383 | 0.028 | 0.600 | Trivial (return+) |
+| RAGSAK infrastructure | 724 | 0.015 | 0.600 | Trivial (single sym) |
+| RAGSAK modules | 973 | 0.015 | 0.600 | Trivial (single sym) |
+| FastAPI docs_src | 650 | 0.027 | 0.500 | Trivial (single sym) |
+| Flask tests | 993 | 0.016 | 0.500 | Trivial (single sym) |
+
+## The Pattern
+
+**Refined CRX wins** (7/14) when:
+- CRX structure is low (< 0.05) — flat bags where CRX over-approximates
+- Group size is small-to-medium (13-519 methods)
+- First symbols are diverse enough to create meaningful clusters
+
+**Refined CRX is trivial** (5/14) when:
+- Group size is large (383-973 methods)
+- Most sequences share the same first symbol (e.g., all start with `return`)
+- Refined clusters everything into one group → CRX on that group → single symbol
+
+**CRX wins** (1/14) when:
+- CRX already has decent structure (> 0.1)
+- Refined splits too aggressively, losing the overall pattern
+
+## The Decision Matrix
+
+```
+                        ┌─────────────────────────────────┐
+                        │     Group size (N methods)?      │
+                        ├────────────┬────────────────────┤
+                        │   N <= 50  │     N > 50         │
+┌───────────────────────┼────────────┼────────────────────┤
+│ CRX structure < 0.05  │  REFINED   │  REFINED           │
+│ (flat bag)            │  (always)  │  (check for trivial)│
+├───────────────────────┼────────────┼────────────────────┤
+│ CRX structure 0.05-0.2│  REFINED   │  CRX               │
+│ (semi-structured)     │  (usually) │  (safe default)     │
+├───────────────────────┼────────────┼────────────────────┤
+│ CRX structure > 0.2   │  CRX       │  CRX               │
+│ (already structured)  │  (already  │  (already good)     │
+│                       │   good)    │                     │
+└───────────────────────┴────────────┴────────────────────┘
+```
+
+## The Rule
+
+```python
+if crx_structure >= 0.2:
+    use CRX  # already good enough
+elif n_methods <= 50:
+    use refined  # small group, safe to cluster
+elif crx_structure < 0.05:
+    use refined with triviality check  # flat bag, worth trying
+else:
+    use CRX  # medium group, semi-structured, CRX is safer
+```
+
+## Triviality Check
+
+When using refined CRX, always check:
+```python
+if model_cost(refined_grammar) < 2:
+    use CRX instead  # refined produced a single symbol, useless
+```
+
+This catches the 36% of cases where refined clusters everything into one group.
+
+## Chain of Reasoning
+
+1. **Started with CRX only** — fast, always works, but over-approximates on diverse groups
+2. **Tried kORE** — slow (400ms), returns None on real data, no advantage over iDRegEx
+3. **Tried iDRegEx** — slow (700ms), returns None on most data, occasionally useful (477x improvement on 1 package)
+4. **Tried refined CRX** — cluster-then-infer, better structure on flat bags, but sometimes trivial
+5. **Tested across 3 codebases** — refined wins 78% when useful, trivial 36% on large groups
+6. **Conclusion**: CRX is the default, refined is opt-in for flat bags, iDRegEx is optional for rare cases
+
+## Final Recommendation
+
+| Scenario | Algorithm | Flag |
+|----------|-----------|------|
+| Default (most cases) | CRX | `--crx-method standard` |
+| Flat bags (struct < 0.05) | Refined CRX | `--crx-method refined` |
+| Need absolute best grammar | iDRegEx | `--idregex-refine` |
+| Large groups (N > 500) | CRX | (avoid refined, likely trivial) |
+| Small groups (N < 20) | Refined CRX | (safe to cluster) |

experiments/RESEARCH_POSITIONING.md

@@ -0,0 +1,184 @@
+# Where We Fit in the Science
+
+## The Landscape (2024-2026)
+
+Grammar inference for code is active across 4 distinct research areas:
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                    Grammar Inference Landscape                      │
+├─────────────────┬─────────────────┬─────────────────┬───────────────┤
+│ Black-Box CFG   │ White-Box CFG   │ Constrained     │ Behavioral    │
+│ Inference       │ Inference       │ Decoding        │ Type Inference│
+├─────────────────┼─────────────────┼─────────────────┼───────────────┤
+│ Crucio (ICSE'26)│ Panini (OOPSLA'25)│ XGrammar      │ REST (OOPSLA'25)│
+│ Kedavra (ASE'24)│ Leon (ICSE'25)  │ DOMINO         │ Typify (ICPC'26)│
+│ Arvada          │                 │ ASAp (NeurIPS'24)│ RightTyper   │
+│ Treevada        │                 │ CRANE (ICML'25) │ DAInfer+      │
+│                 │                 │ TreeCoder       │               │
+└─────────────────┴─────────────────┴─────────────────┴───────────────┘
+```
+
+## What Others Do
+
+### Black-Box CFG Inference (Crucio, Kedavra, Arvada)
+- **Input**: Example strings + oracle (accept/reject)
+- **Output**: Context-free grammar
+- **Method**: Decompose strings, generalize via distributional analysis
+- **Target**: Programming language grammars (JSON, XML, C, Java)
+- **Limitation**: Needs oracle, assumes regular/context-free languages
+
+### White-Box CFG Inference (Panini, Leon)
+- **Input**: Parser source code (ad hoc parsers)
+- **Output**: Regular grammar
+- **Method**: Refinement type inference + abstract interpretation
+- **Target**: String parsing functions (split, regex, format)
+- **Limitation**: Only works on parser functions, not behavioral patterns
+
+### Constrained Decoding (XGrammar, DOMINO, ASAp, CRANE)
+- **Input**: Grammar + LLM
+- **Output**: LLM constrained to grammar
+- **Method**: Mask invalid tokens during generation
+- **Target**: JSON, SQL, code with strict syntax
+- **Limitation**: Requires pre-existing grammar, only enforces syntax
+
+### Behavioral Type Inference (REST, Typify, RightTyper)
+- **Input**: Source code
+- **Output**: Type annotations
+- **Method**: Static/dynamic analysis + type inference
+- **Target**: Function signatures, return types, parameter types
+- **Limitation**: Types only, not behavioral patterns
+
+## Where We Are Different
+
+**We are the only approach that infers behavioral grammars from execution patterns.**
+
+```
+Others:
+  Parser source code → Grammar (for input validation)
+  Example strings → Grammar (for language definition)
+  Grammar → LLM (for output constraint)
+
+Us:
+  Source code → Behavioral sequences → Grammar (for usage patterns)
+  Grammar → LLM (for context/constraint)
+```
+
+### Key Distinctions
+
+| Aspect | Others | Us |
+|--------|--------|-----|
+| **Input** | Parser code or example strings | Any codebase (behavioral sequences) |
+| **Output** | Grammar for input validation | Grammar for usage patterns |
+| **Target** | String parsing functions | API call sequences |
+| **Granularity** | Per-function or per-language | Per-package/module |
+| **Language support** | Usually 1 language | Any tree-sitter supported language |
+| **Use case** | Formal verification, testing | LLM code generation guidance |
+
+### The Gap We Fill
+
+1. **Panini** infers grammars for individual ad hoc parsers (e.g., `json.loads`). We infer grammars for *how packages are used* (e.g., Flask route patterns).
+
+2. **Crucio/Kedavra** infer grammars from input/output examples. We infer grammars from *observed execution patterns* — no oracle needed, we have the source.
+
+3. **XGrammar/DOMINO** enforce pre-existing grammars during LLM generation. We *discover* the grammars that should be enforced.
+
+4. **REST/Typify** infer types (what something is). We infer *behavioral patterns* (how something is used).
+
+## Our Contribution
+
+### The Behavioral Grammar Concept
+
+**Definition**: A behavioral grammar captures the valid sequences of API calls within a package or module, expressed as a regular expression.
+
+```python
+# Example: Flask route handler
+Grammar: GET_RETURN (POST_RETURN)* RETURN
+# Means: Routes often start with GET, sometimes POST, always return
+```
+
+### The Pipeline
+
+```
+Source Code → tree-sitter AST → Behavioral Sequences → BEX Algorithms → YAML/GBNF
+                                         ↑
+                                    Our innovation
+                              (language-agnostic preprocessing)
+```
+
+### What Makes It Work
+
+1. **Behavioral prefix extraction**: We capture the *intent* of code, not the implementation
+2. **Token coarsening**: RETURN, IF, EXCEPTION, LOOP — abstract enough to generalize
+3. **Package slicing**: Context matters (Flask vs FastAPI vs Django)
+4. **Recursive splitting**: Separate mixed groups by first symbol
+5. **Multiple algorithms**: CRX (fast), refined CRX (tighter), iDRegEx (rare)
+
+## Where We Don't Fit (Yet)
+
+### Limitations Compared to Others
+
+1. **Not formal verification**: Our grammars are approximate, not proven correct
+2. **Not parser inference**: We don't infer grammars for string parsing
+3. **Not constraint enforcement**: We don't yet integrate with XGrammar/DOMINO
+4. **Not type inference**: We complement types, not replace them
+5. **Not language-specific**: We don't leverage language-specific type systems
+
+### What We Could Become
+
+1. **Grammar + Type hybrid**: Combine our behavioral grammars with Typify's type inference
+2. **Constrained decoding integration**: Feed our grammars to XGrammar for LLM guidance
+3. **API specification inference**: Combine with DAInfer+ for full API contracts
+4. **Testing**: Use behavioral grammars for test case generation
+5. **Documentation**: Auto-generate usage patterns from code
+
+## Research Positioning
+
+### Our Niche
+
+```
+                    ┌─────────────────────────────┐
+                    │    Behavioral Grammar       │
+                    │    Inference (Us)           │
+                    └──────────┬──────────────────┘
+                               │
+        ┌──────────────────────┼──────────────────────┐
+        │                      │                      │
+        ▼                      ▼                      ▼
+┌───────────────┐    ┌───────────────┐    ┌───────────────┐
+│ Tree-sitter   │    │ BEX Algorithm │    │ LLM Context   │
+│ Preprocessing │    │ Adaptation    │    │ Generation    │
+│ (Language-    │    │ (XML → Code   │    │ (Grammar →    │
+│  agnostic)    │    │  patterns)    │    │  Prompt/      │
+│               │    │               │    │  Constraint)  │
+└───────────────┘    └───────────────┘    └───────────────┘
+```
+
+### Related Work (Cite)
+
+1. **Bex et al. (2010)**: "Inference of concise regular expressions and DTDs" — CRX algorithm
+2. **Schröder & Cito (2025)**: "Static inference of regular grammars for ad hoc parsers" — Panini
+3. **Li et al. (2024)**: "Incremental context-free grammar inference in black box settings" — Kedavra
+4. **Li et al. (2026)**: "Context-free grammar inference for complex programming languages" — Crucio
+5. **Dong et al. (2024)**: "XGrammar: Flexible and efficient structured generation engine" — Constrained decoding
+6. **Park et al. (2025)**: "Flexible and efficient grammar-constrained decoding" — GCD
+7. **Tam et al. (2025)**: "Grammar-constrained decoding makes LLMs better logical parsers" — GCD + reasoning
+8. **Masoudian et al. (2026)**: "DAInfer+: Neurosymbolic inference of API specifications" — API contracts
+9. **Typify (2026)**: "Usage-driven static analyzer for precise Python type inference" — Type inference
+
+### Our Novelty
+
+1. **First to apply BEX to behavioral sequences** (not XML/input data)
+2. **Language-agnostic preprocessing** via tree-sitter (not language-specific)
+3. **Package-level behavioral patterns** (not per-function or per-language)
+4. **Grammar as LLM context** (not formal verification or testing)
+5. **Hybrid approach** combining BEX algorithms with modern preprocessing
+
+## Summary
+
+We occupy a unique position: **behavioral grammar inference from source code**. Others do:
+- Grammar inference for parsers (Panini, Crucio)
+- Grammar enforcement for LLMs (XGrammar, DOMINO)
+- Type inference for code (Typify, REST)
+
+We do: **Discover the patterns that should be inferred, enforced, or typed**.