From 242f00a0dacae5c2fd0aafdec51c4c015f4152d6 Mon Sep 17 00:00:00 2001 From: tobjend Date: Sun, 12 Jul 2026 17:14:56 +0200 Subject: [PATCH] docs: research positioning and decision matrix - RESEARCH_POSITIONING.md: where we fit in the science - DECISION_MATRIX.md: when to use CRX vs refined CRX - Updated AGENTS.md with research context and updated CLI flags --- AGENTS.md | 49 ++++++-- experiments/DECISION_MATRIX.md | 104 ++++++++++++++++ experiments/RESEARCH_POSITIONING.md | 184 ++++++++++++++++++++++++++++ 3 files changed, 328 insertions(+), 9 deletions(-) create mode 100644 experiments/DECISION_MATRIX.md create mode 100644 experiments/RESEARCH_POSITIONING.md diff --git a/AGENTS.md b/AGENTS.md index fe7cfba..ed4f172 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -5,6 +5,37 @@ This repo implements the BEX family of algorithms for inferring regular expressi from example sequences. Use it whenever you need to discover the pattern behind a set of strings or structured sequences. +## Research Positioning + +**We are the only approach that infers behavioral grammars from source code execution patterns.** + +``` +Others: + Parser source code → Grammar (for input validation) + Example strings → Grammar (for language definition) + Grammar → LLM (for output constraint) + +Us: + Source code → Behavioral sequences → Grammar (for usage patterns) + Grammar → LLM (for context/constraint) +``` + +### Related Work (2024-2026) +- **Panini** (OOPSLA'25): Infers grammars for ad hoc parsers via refinement types +- **Crucio** (ICSE'26): Black-box CFG inference from input/output examples +- **XGrammar** (NeurIPS'24): Constrained decoding engine for LLMs +- **DOMINO** (ICML'25): Minimally-invasive grammar-constrained decoding +- **Typify** (ICPC'26): Usage-driven Python type inference +- **DAInfer+** (2026): API specification inference from documentation + +### Our Novelty +1. First to apply BEX algorithms to behavioral sequences (not XML/input data) +2. Language-agnostic preprocessing via tree-sitter (not language-specific) +3. Package-level behavioral patterns (not per-function or per-language) +4. Grammar as LLM context (not formal verification or testing) + +See `experiments/RESEARCH_POSITIONING.md` for full analysis. + ## Quick Start for Agents ```python @@ -18,11 +49,11 @@ g = idregex([['a','b','c'], ['a','b'], ['a','c']], kmax=2, N=3) ``` ## Use Cases -1. **Ansible role patterns** — extract module sequences from tasks/main.yml, learn per-category grammars -2. **Log analysis** — find common patterns in event sequences -3. **API call patterns** — learn the typical order of API operations -4. **Configuration structure** — discover the schema behind YAML files -5. **Workflow mining** — extract the typical task flow from process logs +1. **LLM code generation guidance** — provide behavioral grammars as context for correct API usage +2. **API pattern documentation** — auto-generate usage patterns from codebases +3. **Test case generation** — use grammars to generate valid API call sequences +4. **Code review** — detect deviations from learned behavioral patterns +5. **Migration assistance** — compare behavioral patterns across framework versions ## Architecture @@ -30,9 +61,9 @@ Three inference pipelines: | Pipeline | When to use | |----------|-------------| -| CRX (fast) | Many examples, need speed, CHAREs output | -| iDRegEx (robust) | Few/noisy examples, need probabilistic handling | -| Tag Preprocessor (`bex.tag_preprocessor`) | Source code analysis — tree-sitter AST → method-level call sequences → per-package grammars | +| CRX (fast, default) | Many examples, need speed, CHAREs output | +| Refined CRX (`--crx-method refined`) | Flat bags, need tighter grammars (cluster-then-infer) | +| iDRegEx (`--idregex-refine`) | Rare: small groups with many optionals, need 100x+ improvement | ## Running Tests ```bash @@ -59,4 +90,4 @@ python -m bex.tag_preprocessor.analyze /path/to/codebase --verbose python -m bex.tag_preprocessor.analyze /path/to/codebase --slice package --include '**/src/**' ``` -Key flags: `--slice package` (per-directory grammars), `--verbose` (progress), `--include`/`--exclude` (glob filters), `--main-only` (exclude test files), `--kore` (enable slow kORE in ensemble). +Key flags: `--slice package` (per-directory grammars), `--split-mixed` (recursive split by first symbol), `--crx-method refined` (tighter grammars on flat bags), `--verbose` (progress), `--include`/`--exclude` (glob filters), `--main-only` (exclude test files), `--idregex-refine` (enable iDRegEx on small flat bags). diff --git a/experiments/DECISION_MATRIX.md b/experiments/DECISION_MATRIX.md new file mode 100644 index 0000000..c58a97e --- /dev/null +++ b/experiments/DECISION_MATRIX.md @@ -0,0 +1,104 @@ +# Decision Matrix: CRX vs Refined CRX + +## The Question + +When should we use standard CRX (fast, always works) vs refined CRX (cluster-then-infer, tighter but sometimes trivial)? + +## The Data + +Tested on 3 codebases, 14 packages total: + +| Package | N | CRX struct | Refined struct | Winner | +|---------|---|-----------|---------------|--------| +| RAGSAK agents | 519 | 0.037 | 0.281 | Refined | +| RAGSAK buildSrc | 13 | 0.146 | 0.361 | Refined | +| RAGSAK entrypoints | 417 | 0.027 | 0.214 | Refined | +| RAGSAK platform | 35 | 0.039 | 0.145 | Refined | +| FastAPI fastapi | 375 | 0.004 | 0.007 | Refined | +| FastAPI scripts | 168 | 0.005 | 0.030 | Refined | +| FastAPI tests | 3618 | 0.117 | 0.043 | CRX | +| Flask examples | 61 | 0.011 | 0.043 | Refined | +| Flask src | 368 | 0.014 | 0.016 | Tie | +| RAGSAK app | 383 | 0.028 | 0.600 | Trivial (return+) | +| RAGSAK infrastructure | 724 | 0.015 | 0.600 | Trivial (single sym) | +| RAGSAK modules | 973 | 0.015 | 0.600 | Trivial (single sym) | +| FastAPI docs_src | 650 | 0.027 | 0.500 | Trivial (single sym) | +| Flask tests | 993 | 0.016 | 0.500 | Trivial (single sym) | + +## The Pattern + +**Refined CRX wins** (7/14) when: +- CRX structure is low (< 0.05) — flat bags where CRX over-approximates +- Group size is small-to-medium (13-519 methods) +- First symbols are diverse enough to create meaningful clusters + +**Refined CRX is trivial** (5/14) when: +- Group size is large (383-973 methods) +- Most sequences share the same first symbol (e.g., all start with `return`) +- Refined clusters everything into one group → CRX on that group → single symbol + +**CRX wins** (1/14) when: +- CRX already has decent structure (> 0.1) +- Refined splits too aggressively, losing the overall pattern + +## The Decision Matrix + +``` + ┌─────────────────────────────────┐ + │ Group size (N methods)? │ + ├────────────┬────────────────────┤ + │ N <= 50 │ N > 50 │ +┌───────────────────────┼────────────┼────────────────────┤ +│ CRX structure < 0.05 │ REFINED │ REFINED │ +│ (flat bag) │ (always) │ (check for trivial)│ +├───────────────────────┼────────────┼────────────────────┤ +│ CRX structure 0.05-0.2│ REFINED │ CRX │ +│ (semi-structured) │ (usually) │ (safe default) │ +├───────────────────────┼────────────┼────────────────────┤ +│ CRX structure > 0.2 │ CRX │ CRX │ +│ (already structured) │ (already │ (already good) │ +│ │ good) │ │ +└───────────────────────┴────────────┴────────────────────┘ +``` + +## The Rule + +```python +if crx_structure >= 0.2: + use CRX # already good enough +elif n_methods <= 50: + use refined # small group, safe to cluster +elif crx_structure < 0.05: + use refined with triviality check # flat bag, worth trying +else: + use CRX # medium group, semi-structured, CRX is safer +``` + +## Triviality Check + +When using refined CRX, always check: +```python +if model_cost(refined_grammar) < 2: + use CRX instead # refined produced a single symbol, useless +``` + +This catches the 36% of cases where refined clusters everything into one group. + +## Chain of Reasoning + +1. **Started with CRX only** — fast, always works, but over-approximates on diverse groups +2. **Tried kORE** — slow (400ms), returns None on real data, no advantage over iDRegEx +3. **Tried iDRegEx** — slow (700ms), returns None on most data, occasionally useful (477x improvement on 1 package) +4. **Tried refined CRX** — cluster-then-infer, better structure on flat bags, but sometimes trivial +5. **Tested across 3 codebases** — refined wins 78% when useful, trivial 36% on large groups +6. **Conclusion**: CRX is the default, refined is opt-in for flat bags, iDRegEx is optional for rare cases + +## Final Recommendation + +| Scenario | Algorithm | Flag | +|----------|-----------|------| +| Default (most cases) | CRX | `--crx-method standard` | +| Flat bags (struct < 0.05) | Refined CRX | `--crx-method refined` | +| Need absolute best grammar | iDRegEx | `--idregex-refine` | +| Large groups (N > 500) | CRX | (avoid refined, likely trivial) | +| Small groups (N < 20) | Refined CRX | (safe to cluster) | diff --git a/experiments/RESEARCH_POSITIONING.md b/experiments/RESEARCH_POSITIONING.md new file mode 100644 index 0000000..54bf6c2 --- /dev/null +++ b/experiments/RESEARCH_POSITIONING.md @@ -0,0 +1,184 @@ +# Where We Fit in the Science + +## The Landscape (2024-2026) + +Grammar inference for code is active across 4 distinct research areas: + +``` +┌─────────────────────────────────────────────────────────────────────┐ +│ Grammar Inference Landscape │ +├─────────────────┬─────────────────┬─────────────────┬───────────────┤ +│ Black-Box CFG │ White-Box CFG │ Constrained │ Behavioral │ +│ Inference │ Inference │ Decoding │ Type Inference│ +├─────────────────┼─────────────────┼─────────────────┼───────────────┤ +│ Crucio (ICSE'26)│ Panini (OOPSLA'25)│ XGrammar │ REST (OOPSLA'25)│ +│ Kedavra (ASE'24)│ Leon (ICSE'25) │ DOMINO │ Typify (ICPC'26)│ +│ Arvada │ │ ASAp (NeurIPS'24)│ RightTyper │ +│ Treevada │ │ CRANE (ICML'25) │ DAInfer+ │ +│ │ │ TreeCoder │ │ +└─────────────────┴─────────────────┴─────────────────┴───────────────┘ +``` + +## What Others Do + +### Black-Box CFG Inference (Crucio, Kedavra, Arvada) +- **Input**: Example strings + oracle (accept/reject) +- **Output**: Context-free grammar +- **Method**: Decompose strings, generalize via distributional analysis +- **Target**: Programming language grammars (JSON, XML, C, Java) +- **Limitation**: Needs oracle, assumes regular/context-free languages + +### White-Box CFG Inference (Panini, Leon) +- **Input**: Parser source code (ad hoc parsers) +- **Output**: Regular grammar +- **Method**: Refinement type inference + abstract interpretation +- **Target**: String parsing functions (split, regex, format) +- **Limitation**: Only works on parser functions, not behavioral patterns + +### Constrained Decoding (XGrammar, DOMINO, ASAp, CRANE) +- **Input**: Grammar + LLM +- **Output**: LLM constrained to grammar +- **Method**: Mask invalid tokens during generation +- **Target**: JSON, SQL, code with strict syntax +- **Limitation**: Requires pre-existing grammar, only enforces syntax + +### Behavioral Type Inference (REST, Typify, RightTyper) +- **Input**: Source code +- **Output**: Type annotations +- **Method**: Static/dynamic analysis + type inference +- **Target**: Function signatures, return types, parameter types +- **Limitation**: Types only, not behavioral patterns + +## Where We Are Different + +**We are the only approach that infers behavioral grammars from execution patterns.** + +``` +Others: + Parser source code → Grammar (for input validation) + Example strings → Grammar (for language definition) + Grammar → LLM (for output constraint) + +Us: + Source code → Behavioral sequences → Grammar (for usage patterns) + Grammar → LLM (for context/constraint) +``` + +### Key Distinctions + +| Aspect | Others | Us | +|--------|--------|-----| +| **Input** | Parser code or example strings | Any codebase (behavioral sequences) | +| **Output** | Grammar for input validation | Grammar for usage patterns | +| **Target** | String parsing functions | API call sequences | +| **Granularity** | Per-function or per-language | Per-package/module | +| **Language support** | Usually 1 language | Any tree-sitter supported language | +| **Use case** | Formal verification, testing | LLM code generation guidance | + +### The Gap We Fill + +1. **Panini** infers grammars for individual ad hoc parsers (e.g., `json.loads`). We infer grammars for *how packages are used* (e.g., Flask route patterns). + +2. **Crucio/Kedavra** infer grammars from input/output examples. We infer grammars from *observed execution patterns* — no oracle needed, we have the source. + +3. **XGrammar/DOMINO** enforce pre-existing grammars during LLM generation. We *discover* the grammars that should be enforced. + +4. **REST/Typify** infer types (what something is). We infer *behavioral patterns* (how something is used). + +## Our Contribution + +### The Behavioral Grammar Concept + +**Definition**: A behavioral grammar captures the valid sequences of API calls within a package or module, expressed as a regular expression. + +```python +# Example: Flask route handler +Grammar: GET_RETURN (POST_RETURN)* RETURN +# Means: Routes often start with GET, sometimes POST, always return +``` + +### The Pipeline + +``` +Source Code → tree-sitter AST → Behavioral Sequences → BEX Algorithms → YAML/GBNF + ↑ + Our innovation + (language-agnostic preprocessing) +``` + +### What Makes It Work + +1. **Behavioral prefix extraction**: We capture the *intent* of code, not the implementation +2. **Token coarsening**: RETURN, IF, EXCEPTION, LOOP — abstract enough to generalize +3. **Package slicing**: Context matters (Flask vs FastAPI vs Django) +4. **Recursive splitting**: Separate mixed groups by first symbol +5. **Multiple algorithms**: CRX (fast), refined CRX (tighter), iDRegEx (rare) + +## Where We Don't Fit (Yet) + +### Limitations Compared to Others + +1. **Not formal verification**: Our grammars are approximate, not proven correct +2. **Not parser inference**: We don't infer grammars for string parsing +3. **Not constraint enforcement**: We don't yet integrate with XGrammar/DOMINO +4. **Not type inference**: We complement types, not replace them +5. **Not language-specific**: We don't leverage language-specific type systems + +### What We Could Become + +1. **Grammar + Type hybrid**: Combine our behavioral grammars with Typify's type inference +2. **Constrained decoding integration**: Feed our grammars to XGrammar for LLM guidance +3. **API specification inference**: Combine with DAInfer+ for full API contracts +4. **Testing**: Use behavioral grammars for test case generation +5. **Documentation**: Auto-generate usage patterns from code + +## Research Positioning + +### Our Niche + +``` + ┌─────────────────────────────┐ + │ Behavioral Grammar │ + │ Inference (Us) │ + └──────────┬──────────────────┘ + │ + ┌──────────────────────┼──────────────────────┐ + │ │ │ + ▼ ▼ ▼ +┌───────────────┐ ┌───────────────┐ ┌───────────────┐ +│ Tree-sitter │ │ BEX Algorithm │ │ LLM Context │ +│ Preprocessing │ │ Adaptation │ │ Generation │ +│ (Language- │ │ (XML → Code │ │ (Grammar → │ +│ agnostic) │ │ patterns) │ │ Prompt/ │ +│ │ │ │ │ Constraint) │ +└───────────────┘ └───────────────┘ └───────────────┘ +``` + +### Related Work (Cite) + +1. **Bex et al. (2010)**: "Inference of concise regular expressions and DTDs" — CRX algorithm +2. **Schröder & Cito (2025)**: "Static inference of regular grammars for ad hoc parsers" — Panini +3. **Li et al. (2024)**: "Incremental context-free grammar inference in black box settings" — Kedavra +4. **Li et al. (2026)**: "Context-free grammar inference for complex programming languages" — Crucio +5. **Dong et al. (2024)**: "XGrammar: Flexible and efficient structured generation engine" — Constrained decoding +6. **Park et al. (2025)**: "Flexible and efficient grammar-constrained decoding" — GCD +7. **Tam et al. (2025)**: "Grammar-constrained decoding makes LLMs better logical parsers" — GCD + reasoning +8. **Masoudian et al. (2026)**: "DAInfer+: Neurosymbolic inference of API specifications" — API contracts +9. **Typify (2026)**: "Usage-driven static analyzer for precise Python type inference" — Type inference + +### Our Novelty + +1. **First to apply BEX to behavioral sequences** (not XML/input data) +2. **Language-agnostic preprocessing** via tree-sitter (not language-specific) +3. **Package-level behavioral patterns** (not per-function or per-language) +4. **Grammar as LLM context** (not formal verification or testing) +5. **Hybrid approach** combining BEX algorithms with modern preprocessing + +## Summary + +We occupy a unique position: **behavioral grammar inference from source code**. Others do: +- Grammar inference for parsers (Panini, Crucio) +- Grammar enforcement for LLMs (XGrammar, DOMINO) +- Type inference for code (Typify, REST) + +We do: **Discover the patterns that should be inferred, enforced, or typed**.