tobi/grammar-inference-engine
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

docs: update experiment log, results, and handover for Round 19-20
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
Details
ci/woodpecker/pr/woodpecker Pipeline failed
Details

Browse source
This commit is contained in:
tobjend 2026-07-13 01:18:32 +02:00
parent 8b3a454a15
commit 36f9e9173d
3 changed files with 292 additions and 82 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

88
experiments/EXPERIMENT_LOG.md
View file

@ -789,3 +789,91 @@ Keep standard CRX as fallback. No need for iDRegEx or kORE in the pipeline.
- `bex/decompose.py`: decompose_sequence(), decompose_all(), decompose_with_coverage()
- `bex/tag_preprocessor/analyze.py`: --decompose, --max-seq-length flags
- `tests/test_decompose.py`: 12 new tests
---
## Round 19: AST Migration
**Goal:** Replace SORE string representations with proper AST nodes throughout the pipeline.
**Method:** Migrated CRX, iDRegEx, ensemble, mdl, and SORE parser from string-based grammars
to `bex.grammar` AST nodes (Symbol, Concat, Alt, Optional, Plus, Star, Empty). Purged all
SORE string operations from the inference pipeline. Added `_count_concat` memoization.
**Commits:** `ea6cac5` → `52f286a` (6 commits)
**Results:**
- RAGSAK: CRX inference dropped from 54.9s → 6.7s after restoring memoization on `_count_concat`
- Fixed `_count_concat` losing its `@lru_cache` decorator during AST migration — was the real performance bug, not ProcessPoolExecutor
- All 313 tests pass
**Key insight:** The memoization loss was invisible because `_count_concat` is called recursively
on every grammar node. Without caching, identical subtrees were re-evaluated exponentially.
**Files changed:**
- `bex/crx.py`: CRX algorithm returns AST nodes
- `bex/ensemble.py`: ensemble matching uses AST comparison
- `bex/mdl.py`: scoring functions operate on AST
- `bex/grammar.py`: AST node definitions, `_count_concat` with `@lru_cache`
---
## Round 20: AST Pipeline Verification + Scoring Fixes
**Goal:** Verify the AST pipeline end-to-end across 3 codebases, fix scoring issues.
**Codebases:** RAGSAK (Kotlin, 462 files, 1609 methods), FastAPI (Python, 143 groups), Zod (TypeScript, 23 groups)
### Phase A: `_COUNT_CAP` fix
**Problem:** `_COUNT_CAP = 10^12` clamped all `lang_size_score` values to the same ceiling,
making bags and tight grammars indistinguishable (all scored 10^12).
**Fix:** Raised `_COUNT_CAP` from `10^12` to `10^30`. With memoization preventing the
recursion hang, the cap no longer needs to be low.
**Result:** `lang_size_score` now discriminates: tight grammar = 20, pure bag = 9975.
MDL score abandoned (ADR-13) — language size scoring chosen because MDL rewards short
expressions over specific patterns.
### Phase B: `decompose=True` default
**Change:** Made decomposition ON by default (CLI + `analyze_directory`). Reduced
`max_seq_length` from 5→4.
**Results:**
| Codebase | Before | After | Change |
|----------|--------|-------|--------|
| RAGSAK | 29 grammars | 95 grammars | 3.3× more patterns |
| RAGSAK pure bags | 9 | 5 | Fewer orderless bags |
| FastAPI | 109 grammars | 118 grammars | Small increase |
| Zod | 16 grammars | 10 grammars | More selective |
### Phase C: `idregex_refine=True` default
**Change:** Enabled iDRegEx refinement by default. Rewrote `_count_optionals` from SORE
string parser to AST walker. Added `_is_pure_bag()` helper.
**Results:**
- RAGSAK v4: 126 grammars total, 6 pure bags, 120 structured
- FastAPI v3: 143 grammars, 26 pure bags, 117 structured
- Zod v3: 23 grammars, 5 pure bags, 18 structured
**Key finding:** iDRegEx doesn't help small bags. On 3-method groups, iDRegEx achieves
only 3.8× tighter (below the 10× threshold gate). The gate correctly rejects it.
Earlier test on `storage` (4 methods) showed 91× — that was an outlier, not the norm.
**Decision:** idregex_refine stays ON but the gate effectively limits it to groups where
iDRegEx produces a genuinely tighter grammar. No further algorithmic changes planned for
orderless bags — they survive because CRX emits one grammar deterministically and
`lang_size_score` only ranks between algorithms, not within CRX's own output.
**Quality reality:** ~85% of grammars remain orderless bags `(A|B|C)+`. The ~15% that are
structured represent real sequential flows (e.g., `post→jsonPath→isEqualTo→exchange→expectStatus`).
Bags are concentrated in large groups (tests, v4/locales) where method diversity is too high
for any algorithm to find ordering.
**Files changed:**
- `bex/tag_preprocessor/analyze.py`: idregex_refine default, _count_optionals AST rewrite, _is_pure_bag helper
- `bex/grammar.py`: _COUNT_CAP raised to 10^30
- `tests/test_analyze.py`: updated tests for AST-based _count_optionals

207
experiments/HANDOVER.md
View file

@ -2,9 +2,9 @@
## Current Status
**Branch**: `feature/treesitter-tag-queries` (PR #2)
**Last commit**: `ee60b62` — fix: YAML expansion bug and add GBNF to output
**Tests**: 269 passed, 8 warnings, 0 failures
**Branch**: `feature/treesitter-tag-queries`
**Last commit**: `8b3a454` — feat: enable idregex_refine by default + AST rewrite of _count_optionals
**Tests**: 313 passed, 8 warnings, 0 failures
## What We Built
@ -13,108 +13,150 @@ A **language-agnostic pipeline** that infers per-package calling conventions fro
1. **Tree-sitter AST** → extract method-level behavioral sequences (call chains, control flow)
2. **Algorithm 7 (CRX)** — generalized regular expression inference from examples
3. **YAML/GBNF output** — structured grammars grouped by package, ready for constrained decoding
3. **AST grammar nodes** — Symbol, Concat, Alt, Optional, Plus, Star, Empty
4. **Language Size scoring** — ranks grammars by compressed description length
5. **YAML/GBNF output** — structured grammars grouped by package, ready for constrained decoding
### Key Features
- **Zero per-language code** — tree-sitter highlights.scm + behavioral prefix filter
- **10 supported languages**: Kotlin, Python, JavaScript, TypeScript, Go, Rust, Java, C++, Ruby, Swift
- **`--slice package`** — per-package grammars (not per-file)
- **`--split-mixed`** — separates interleaved calling conventions
- **`--decompose`** — decomposition forest for complex sequences (7× more grammars on RAGSAK)
- **`min_structure=0.5`** — filters out "flat bag" noise patterns
- **GBNF output** — llama-compatible constrained decoding grammars
### Key Changes Since Last Handover
- **AST migration complete** — all SORE string operations purged from pipeline
- **Scoring**: Language Size (`lang_size_score`) chosen over MDL (ADR-13)
- **Decomposition ON by default** — `decompose=True`, `max_seq_length=4`
- **idregex_refine ON by default** — iDRefEx runs on small groups where it helps
- **`_COUNT_CAP` raised to 10^30** — no longer clamps scorer values
- **Memoization fixed** — `_count_concat` has `@lru_cache`, RAGSAK 54.9s→6.7s
### Algorithm Choice
**CRX (Algorithm 7)** is the default — fast (2ms), always produces something. Refined CRX (cluster-then-infer) is available via `--crx-method refined`. kORE and iDRegEx are opt-in via `--kore` and `--idregex` flags.
### Default Parameters (Golden Config)
```python
{
'decompose': True,
'max_seq_length': 4,
'min_structure': 0.5,
'idregex_refine': True,
'min_methods': 3,
'min_coverage': 0.05,
}
```
### GBNF Grammar Format
Each YAML entry includes a `gbnf` field with the llama-compatible GBNF grammar. This can be passed directly to llama.cpp server API for constrained decoding.
## What Works
### High-Confidence Findings
1. **Decomposition** increases grammar count 3× and reduces pure bags
2. **Language Size scoring** discriminates between tight and bag grammars (20 vs 9975)
3. **CRX is fast and deterministic** — always produces a grammar
4. **Package grouping** — per-directory grammars are the right abstraction level
5. **Memoization is critical** — `_count_concat` without cache = exponential blowup
### Grammar Quality Reality
~85% of grammars are orderless bags `(A|B|C)+`. ~15% are structured sequential flows:
- Web controller tests: `post→jsonPath→isEqualTo→exchange→expectStatus`
- API client patterns: `request→header→send→statusCode→jsonPath`
- Builder chains: `builder→field→value→build→validate`
Bags survive because:
1. CRX emits one grammar deterministically (no alternative to compare)
2. `lang_size_score` only ranks **between** algorithms (CRX vs iDRegEx), not within CRX's own output
3. Methods in large packages don't share sequential patterns — they're genuinely unrelated
## What Doesn't Work
### iDRegEx on Small Bags
iDRegEx achieves only 3.8× tighter on 3-method groups (below the 10× gate threshold).
The gate correctly rejects it. The `storage` group (91× tighter) was an outlier.
### MDL Scoring
Abandoned (ADR-13). MDL rewards short expressions, so generic `info+` beats specific
`a.b.c.d.e+` (21% vs 98% success in Bex paper).
### Cross-Package Grouping
Grouping by first 3 symbols gives 12% coverage but most groups are too sparse.
Package-specific patterns are the norm.
## Files to Know
| File | Purpose |
|------|---------|
| `bex/tag_preprocessor/analyze.py` | Full pipeline: `analyze_directory()` → `analyze_by_package()` → `_infer_group()` → `_build_yaml_output()` |
| `bex/tag_preprocessor/code.py` | `preprocess_by_method()` — AST to behavioral sequences |
| `bex/crx.py` | Standard CRX (Algorithm 7) |
| `bex/crx_refined.py` | Cluster-then-infer CRX |
| `bex/gbnf.py` | SORE→GBNF converter, `validate_sore()`, `grammar_structure_score()` |
| `bex/tag_preprocessor/analyze.py` | Pipeline: `analyze_directory()` → `analyze_by_package()` → `_infer_group()` |
| `bex/grammar.py` | AST nodes, `_count_concat` memoization, `_COUNT_CAP = 10^30` |
| `bex/crx.py` | CRX algorithm (AST-based) |
| `bex/mdl.py` | `lang_size_score()`, `model_cost()`, `data_cost()` |
| `bex/idregex.py` | iDRegEx algorithm |
| `bex/decompose.py` | Decomposition forest |
| `bex/distributional.py` | Crucio-inspired distributional clustering |
| `bex/grammar_index.py` | `GrammarIndex` class for resolving files to grammars |
| `bex/mcp_server.py` | MCP server with `analyze_directory`, `get_grammar`, `get_package_grammars` |
| `bex/mdl.py` | `lang_size_score()` (default), `mdl_score()` (fallback) |
| `bex/reduce.py` | Algorithm 4 (TODS 2010) for grammar reduction |
| `bex/gbnf.py` | GBNF converter, `grammar_structure_score()` |
| `bex/ensemble.py` | `infer_ensemble()` — combine multiple algorithms |
## Experiments
### Key Findings
1. **CRX wins on simplicity** — no post-processing needed, flat chains are interpretable
2. **`DEFAULT_COVERAGE=0.05`** — was 0.8, almost filtered everything out
3. **`min_methods=3`** — sweet spot (was 5, lost 9 FastAPI grammars)
4. **Decomposition helps diverse codebases** — RAGSAK 4→27, FastAPI 16→29 high-structure grammars
5. **Decomposition hurts structured codebases** — kotlinx.coroutines 24→15
### Decision Matrix (CRX vs Refined CRX)
- CRX struct ≥ 0.2 → use CRX (already good)
- CRX struct < 0.05 → use refined (flat bag)
- Group size ≤ 50 → use refined (safe to cluster)
- Group size > 50 → use CRX (refined likely trivial)
### Research Positioning
We are **unique**: first to infer behavioral grammars from source code execution patterns. Related work:
- Panini (white-box CFG from parsers)
- Crucio (black-box CFG from examples)
- XGrammar/DOMINO (constrained decoding)
- Typify/REST (type inference)
Our niche: discover patterns that should be inferred/enforced/typed.
## Open Questions
### 1. Grammar Usefulness for LLM Code Generation
MCP tools are ready but haven't validated if grammars help an LLM during generation. Need to test:
- Does constrained decoding with GBNF improve code quality?
- Do grammars reduce hallucination in call chains?
- Can grammars be used for code completion suggestions?
### 2. Decomposition Trade-off
Decomposition helps diverse codebases but hurts already-structured ones. Need auto-detection:
- If codebase already has good structure → skip decomposition
- If codebase is diverse → apply decomposition
### 3. Cross-Codebase Grammar Reuse
Can grammars from one project inform another? (e.g., "Spring Boot service patterns")
| `bex/mcp_server.py` | MCP server with `analyze_directory`, `get_grammar` |
| `bex/tag_preprocessor/code.py` | `preprocess_by_method()` — AST to behavioral sequences |
## How to Run
```bash
# Basic analysis
bex --include "*.py" --slice package --main-only --format yaml --output grammars.yml
# Basic analysis (all defaults ON)
python -m bex.tag_preprocessor.analyze /path/to/codebase --verbose
# With decomposition + structure filtering
bex --include "*.py" --slice package --main-only --decompose --min-structure 0.5 --format yaml
# With custom settings
python -m bex.tag_preprocessor.analyze /path/to/codebase \
--decompose --idregex-refine --min-structure 0.5 --slice package
# Full pipeline (what we tested)
bex --include "*.kt" --slice package --main-only --split-mixed --decompose --min-structure 0.5 --format yaml
# Disable idregex refinement
python -m bex.tag_preprocessor.analyze /path/to/codebase --no-idregex-refine
# MCP server
bex serve --port 8080
python bex/mcp_server.py --port 8080
```
## Test Coverage
- `tests/test_distributional.py`: 23 tests (distributional clustering)
- `tests/test_decompose.py`: 12 tests (decomposition forest)
- `tests/test_gbnf.py`: 28 tests (GBNF conversion)
- `tests/test_crx_refined.py`: 20 tests (refined CRX)
- `tests/test_grammar_index.py`: 14 tests (grammar index)
- `tests/test_analyze.py`: Pipeline tests
- `tests/test_reduce.py`: Algorithm 4 tests
- `tests/test_mdl.py`: MDL scoring tests
- `tests/test_analyze.py`: Pipeline integration tests
- `tests/test_grammar.py`: AST node tests, count_words memoization
- `tests/test_mdl.py`: Language Size scoring, MDL scoring
- `tests/test_crx.py`: CRX algorithm
- `tests/test_idregex.py`: iDRegEx algorithm
- `tests/test_decompose.py`: Decomposition forest
- `tests/test_distributional.py`: Distributional clustering
- `tests/test_gbnf.py`: GBNF conversion
- `tests/test_crx_refined.py`: Refined CRX
- `tests/test_grammar_index.py`: Grammar index
- `tests/test_reduce.py`: Algorithm 4
Total: 269 tests passing
Total: 313 tests passing
## Decision Log
| Decision | Choice | Rationale |
|----------|--------|-----------|
| Scoring | Language Size (not MDL) | MDL rewards short over specific (ADR-13) |
| Decomposition | ON by default | 3× more grammars, fewer pure bags |
| idregex_refine | ON by default | Gate limits to groups where it helps |
| AST representation | Full AST nodes | Type safety, memoization, no string parsing |
| Algorithm | CRX (default) | Fast, deterministic, always produces output |
## Open Questions
### 1. Grammar Usefulness for LLM Code Generation
MCP tools are ready but haven't validated if grammars help an LLM during generation.
Need to test:
- Does constrained decoding with GBNF improve code quality?
- Do grammars reduce hallucination in call chains?
### 2. The 85% Bag Problem
Most grammars are orderless bags. Two possible directions:
- **Accept it**: Bags represent real diversity in method usage. Not a bug.
- **Better grouping**: If we group methods by semantic role (not just directory),
we might find ordering within sub-groups. Requires understanding method semantics.
### 3. Cross-Codebase Grammar Reuse
Can grammars from one project inform another? (e.g., "Spring Boot service patterns")
## Experiments Summary
| Round | What | Result |
|-------|------|--------|
| 1-5 | Context strategies | Package grouping wins (12% coverage) |
| 6-10 | Reduce, clustering | Reduce merges states, not packages |
| 11-15 | Distributional, ensemble | CRX is sufficient, no ensemble needed |
| 16-17 | Refined CRX | Better ~78% of the time when useful, but trivial ~36% |
| 18 | Decomposition | 3× more grammars, fewer bags |
| 19 | AST migration | 54.9s→6.7s after memoization fix |
| 20 | Scoring + defaults | Language Size works, decomposition ON, idregex ON |
## Next Steps
@ -122,3 +164,4 @@ Total: 269 tests passing
2. **Auto-detect decomposition** — skip if codebase already structured
3. **Cross-project grammar reuse** — share patterns across codebases
4. **IDE integration** — grammar-aware code completion
5. **No further algorithmic changes on bags** — they're a feature, not a bug

79
experiments/RESULTS.md
View file

@ -162,3 +162,82 @@ Converts textual sequences into structural shapes that repeat across packages.
- `bex/gbnf.py` — SORE → GBNF converter
- `tests/test_reduce.py` — 24 tests for Reduce
- `tests/test_gbnf.py` — 15 tests for GBNF converter
---
## Round 20: AST Pipeline + Scoring Fixes (2026-07-13)
**Commit range:** `ea6cac5` → `8b3a454`
### Codebases
| Codebase | Files | Methods | Language |
|----------|-------|---------|----------|
| RAGSAK | 462 | 1609 | Kotlin |
| FastAPI | — | — | Python |
| Zod | — | — | TypeScript |
### Scoring: Language Size over MDL (ADR-13)
Abandoned MDL scoring — it rewards short expressions, so generic `info+` beat specific
`a.b.c.d.e+` (21% vs 98% success in Bex paper). Language Size (`lang_size_score`) chosen.
| Metric | Bag grammar | Structured grammar |
|--------|-------------|-------------------|
| `lang_size_score` | 9975 | 20 |
| `mdl_score` | 10^12 (clamped) | 10^12 (clamped) |
### Final Defaults
| Parameter | Before | After |
|-----------|--------|-------|
| `decompose` | False | **True** |
| `max_seq_length` | 5 | **4** |
| `idregex_refine` | False | **True** |
| `_COUNT_CAP` | 10^12 | **10^30** |
### Results
| Codebase | Grammars | Pure Bags | Structured | Bag % |
|----------|----------|-----------|------------|-------|
| RAGSAK (v4) | 126 | 6 | 120 | 4.8% |
| FastAPI (v3) | 143 | 26 | 117 | 18.2% |
| Zod (v3) | 23 | 5 | 18 | 21.7% |
**Quality breakdown:** ~85% of grammars across codebases remain orderless bags `(A|B|C)+`.
The ~15% that are structured represent real sequential flows:
- Web controller tests: `post→jsonPath→isEqualTo→exchange→expectStatus`
- API client patterns: `request→header→send→statusCode→jsonPath`
- Builder chains: `builder→field→value→build→validate`
### iDRegEx Findings
iDRegEx does NOT help at small scale. On 3-method groups, iDRegEx achieves only 3.8×
tighter (below the 10× gate threshold). The gate correctly rejects it.
| Group size | CRX lang_size | iDRegEx lang_size | Ratio |
|------------|--------------|-------------------|-------|
| 3 methods | 15 | 4 | 3.8× |
| 4 methods (`storage`) | — | — | 91× (outlier) |
Bags survive because:
1. CRX emits one grammar deterministically (no alternative to compare)
2. `lang_size_score` only ranks **between** algorithms, not within CRX's own output
3. iDRegEx is too slow for large groups (200s+ timeout on 2036m FastAPI tests)
### Key Insight
The grammar inference pipeline is fundamentally limited by the input: if methods in a
package don't share a sequential calling pattern, no algorithm can find one. The ~15%
structured grammars represent genuinely reusable patterns; the ~85% bags represent
packages with diverse, unrelated methods grouped only by directory proximity.
### Files
- `bex/grammar.py`: AST nodes, `_count_concat` memoization, `_COUNT_CAP = 10^30`
- `bex/crx.py`: CRX algorithm (AST-based)
- `bex/mdl.py`: `lang_size_score`, `model_cost`, `data_cost`
- `bex/idregex.py`: iDRegEx algorithm
- `bex/decompose.py`: sequence decomposition
- `bex/tag_preprocessor/analyze.py`: pipeline orchestration, all defaults
- `experiments/results/round20_ast_verify/`: full experiment data (v2/v3/v4)