From 36f9e9173d9f88e9446c238f08f90267f7916c0d Mon Sep 17 00:00:00 2001 From: tobjend Date: Mon, 13 Jul 2026 01:18:32 +0200 Subject: [PATCH] docs: update experiment log, results, and handover for Round 19-20 --- experiments/EXPERIMENT_LOG.md | 88 +++++++++++++++ experiments/HANDOVER.md | 207 ++++++++++++++++++++-------------- experiments/RESULTS.md | 79 +++++++++++++ 3 files changed, 292 insertions(+), 82 deletions(-) diff --git a/experiments/EXPERIMENT_LOG.md b/experiments/EXPERIMENT_LOG.md index 49e6b5f..52510a7 100644 --- a/experiments/EXPERIMENT_LOG.md +++ b/experiments/EXPERIMENT_LOG.md @@ -789,3 +789,91 @@ Keep standard CRX as fallback. No need for iDRegEx or kORE in the pipeline. - `bex/decompose.py`: decompose_sequence(), decompose_all(), decompose_with_coverage() - `bex/tag_preprocessor/analyze.py`: --decompose, --max-seq-length flags - `tests/test_decompose.py`: 12 new tests + +--- + +## Round 19: AST Migration + +**Goal:** Replace SORE string representations with proper AST nodes throughout the pipeline. + +**Method:** Migrated CRX, iDRegEx, ensemble, mdl, and SORE parser from string-based grammars +to `bex.grammar` AST nodes (Symbol, Concat, Alt, Optional, Plus, Star, Empty). Purged all +SORE string operations from the inference pipeline. Added `_count_concat` memoization. + +**Commits:** `ea6cac5` → `52f286a` (6 commits) + +**Results:** +- RAGSAK: CRX inference dropped from 54.9s → 6.7s after restoring memoization on `_count_concat` +- Fixed `_count_concat` losing its `@lru_cache` decorator during AST migration — was the real performance bug, not ProcessPoolExecutor +- All 313 tests pass + +**Key insight:** The memoization loss was invisible because `_count_concat` is called recursively +on every grammar node. Without caching, identical subtrees were re-evaluated exponentially. + +**Files changed:** +- `bex/crx.py`: CRX algorithm returns AST nodes +- `bex/ensemble.py`: ensemble matching uses AST comparison +- `bex/mdl.py`: scoring functions operate on AST +- `bex/grammar.py`: AST node definitions, `_count_concat` with `@lru_cache` + +--- + +## Round 20: AST Pipeline Verification + Scoring Fixes + +**Goal:** Verify the AST pipeline end-to-end across 3 codebases, fix scoring issues. + +**Codebases:** RAGSAK (Kotlin, 462 files, 1609 methods), FastAPI (Python, 143 groups), Zod (TypeScript, 23 groups) + +### Phase A: `_COUNT_CAP` fix + +**Problem:** `_COUNT_CAP = 10^12` clamped all `lang_size_score` values to the same ceiling, +making bags and tight grammars indistinguishable (all scored 10^12). + +**Fix:** Raised `_COUNT_CAP` from `10^12` to `10^30`. With memoization preventing the +recursion hang, the cap no longer needs to be low. + +**Result:** `lang_size_score` now discriminates: tight grammar = 20, pure bag = 9975. +MDL score abandoned (ADR-13) — language size scoring chosen because MDL rewards short +expressions over specific patterns. + +### Phase B: `decompose=True` default + +**Change:** Made decomposition ON by default (CLI + `analyze_directory`). Reduced +`max_seq_length` from 5→4. + +**Results:** +| Codebase | Before | After | Change | +|----------|--------|-------|--------| +| RAGSAK | 29 grammars | 95 grammars | 3.3× more patterns | +| RAGSAK pure bags | 9 | 5 | Fewer orderless bags | +| FastAPI | 109 grammars | 118 grammars | Small increase | +| Zod | 16 grammars | 10 grammars | More selective | + +### Phase C: `idregex_refine=True` default + +**Change:** Enabled iDRegEx refinement by default. Rewrote `_count_optionals` from SORE +string parser to AST walker. Added `_is_pure_bag()` helper. + +**Results:** +- RAGSAK v4: 126 grammars total, 6 pure bags, 120 structured +- FastAPI v3: 143 grammars, 26 pure bags, 117 structured +- Zod v3: 23 grammars, 5 pure bags, 18 structured + +**Key finding:** iDRegEx doesn't help small bags. On 3-method groups, iDRegEx achieves +only 3.8× tighter (below the 10× threshold gate). The gate correctly rejects it. +Earlier test on `storage` (4 methods) showed 91× — that was an outlier, not the norm. + +**Decision:** idregex_refine stays ON but the gate effectively limits it to groups where +iDRegEx produces a genuinely tighter grammar. No further algorithmic changes planned for +orderless bags — they survive because CRX emits one grammar deterministically and +`lang_size_score` only ranks between algorithms, not within CRX's own output. + +**Quality reality:** ~85% of grammars remain orderless bags `(A|B|C)+`. The ~15% that are +structured represent real sequential flows (e.g., `post→jsonPath→isEqualTo→exchange→expectStatus`). +Bags are concentrated in large groups (tests, v4/locales) where method diversity is too high +for any algorithm to find ordering. + +**Files changed:** +- `bex/tag_preprocessor/analyze.py`: idregex_refine default, _count_optionals AST rewrite, _is_pure_bag helper +- `bex/grammar.py`: _COUNT_CAP raised to 10^30 +- `tests/test_analyze.py`: updated tests for AST-based _count_optionals diff --git a/experiments/HANDOVER.md b/experiments/HANDOVER.md index 0711579..cf07c6c 100644 --- a/experiments/HANDOVER.md +++ b/experiments/HANDOVER.md @@ -2,9 +2,9 @@ ## Current Status -**Branch**: `feature/treesitter-tag-queries` (PR #2) -**Last commit**: `ee60b62` — fix: YAML expansion bug and add GBNF to output -**Tests**: 269 passed, 8 warnings, 0 failures +**Branch**: `feature/treesitter-tag-queries` +**Last commit**: `8b3a454` — feat: enable idregex_refine by default + AST rewrite of _count_optionals +**Tests**: 313 passed, 8 warnings, 0 failures ## What We Built @@ -13,108 +13,150 @@ A **language-agnostic pipeline** that infers per-package calling conventions fro 1. **Tree-sitter AST** → extract method-level behavioral sequences (call chains, control flow) 2. **Algorithm 7 (CRX)** — generalized regular expression inference from examples -3. **YAML/GBNF output** — structured grammars grouped by package, ready for constrained decoding +3. **AST grammar nodes** — Symbol, Concat, Alt, Optional, Plus, Star, Empty +4. **Language Size scoring** — ranks grammars by compressed description length +5. **YAML/GBNF output** — structured grammars grouped by package, ready for constrained decoding -### Key Features -- **Zero per-language code** — tree-sitter highlights.scm + behavioral prefix filter -- **10 supported languages**: Kotlin, Python, JavaScript, TypeScript, Go, Rust, Java, C++, Ruby, Swift -- **`--slice package`** — per-package grammars (not per-file) -- **`--split-mixed`** — separates interleaved calling conventions -- **`--decompose`** — decomposition forest for complex sequences (7× more grammars on RAGSAK) -- **`min_structure=0.5`** — filters out "flat bag" noise patterns -- **GBNF output** — llama-compatible constrained decoding grammars +### Key Changes Since Last Handover +- **AST migration complete** — all SORE string operations purged from pipeline +- **Scoring**: Language Size (`lang_size_score`) chosen over MDL (ADR-13) +- **Decomposition ON by default** — `decompose=True`, `max_seq_length=4` +- **idregex_refine ON by default** — iDRefEx runs on small groups where it helps +- **`_COUNT_CAP` raised to 10^30** — no longer clamps scorer values +- **Memoization fixed** — `_count_concat` has `@lru_cache`, RAGSAK 54.9s→6.7s -### Algorithm Choice -**CRX (Algorithm 7)** is the default — fast (2ms), always produces something. Refined CRX (cluster-then-infer) is available via `--crx-method refined`. kORE and iDRegEx are opt-in via `--kore` and `--idregex` flags. +### Default Parameters (Golden Config) +```python +{ + 'decompose': True, + 'max_seq_length': 4, + 'min_structure': 0.5, + 'idregex_refine': True, + 'min_methods': 3, + 'min_coverage': 0.05, +} +``` -### GBNF Grammar Format -Each YAML entry includes a `gbnf` field with the llama-compatible GBNF grammar. This can be passed directly to llama.cpp server API for constrained decoding. +## What Works + +### High-Confidence Findings +1. **Decomposition** increases grammar count 3× and reduces pure bags +2. **Language Size scoring** discriminates between tight and bag grammars (20 vs 9975) +3. **CRX is fast and deterministic** — always produces a grammar +4. **Package grouping** — per-directory grammars are the right abstraction level +5. **Memoization is critical** — `_count_concat` without cache = exponential blowup + +### Grammar Quality Reality +~85% of grammars are orderless bags `(A|B|C)+`. ~15% are structured sequential flows: +- Web controller tests: `post→jsonPath→isEqualTo→exchange→expectStatus` +- API client patterns: `request→header→send→statusCode→jsonPath` +- Builder chains: `builder→field→value→build→validate` + +Bags survive because: +1. CRX emits one grammar deterministically (no alternative to compare) +2. `lang_size_score` only ranks **between** algorithms (CRX vs iDRegEx), not within CRX's own output +3. Methods in large packages don't share sequential patterns — they're genuinely unrelated + +## What Doesn't Work + +### iDRegEx on Small Bags +iDRegEx achieves only 3.8× tighter on 3-method groups (below the 10× gate threshold). +The gate correctly rejects it. The `storage` group (91× tighter) was an outlier. + +### MDL Scoring +Abandoned (ADR-13). MDL rewards short expressions, so generic `info+` beats specific +`a.b.c.d.e+` (21% vs 98% success in Bex paper). + +### Cross-Package Grouping +Grouping by first 3 symbols gives 12% coverage but most groups are too sparse. +Package-specific patterns are the norm. ## Files to Know | File | Purpose | |------|---------| -| `bex/tag_preprocessor/analyze.py` | Full pipeline: `analyze_directory()` → `analyze_by_package()` → `_infer_group()` → `_build_yaml_output()` | -| `bex/tag_preprocessor/code.py` | `preprocess_by_method()` — AST to behavioral sequences | -| `bex/crx.py` | Standard CRX (Algorithm 7) | -| `bex/crx_refined.py` | Cluster-then-infer CRX | -| `bex/gbnf.py` | SORE→GBNF converter, `validate_sore()`, `grammar_structure_score()` | +| `bex/tag_preprocessor/analyze.py` | Pipeline: `analyze_directory()` → `analyze_by_package()` → `_infer_group()` | +| `bex/grammar.py` | AST nodes, `_count_concat` memoization, `_COUNT_CAP = 10^30` | +| `bex/crx.py` | CRX algorithm (AST-based) | +| `bex/mdl.py` | `lang_size_score()`, `model_cost()`, `data_cost()` | +| `bex/idregex.py` | iDRegEx algorithm | | `bex/decompose.py` | Decomposition forest | -| `bex/distributional.py` | Crucio-inspired distributional clustering | -| `bex/grammar_index.py` | `GrammarIndex` class for resolving files to grammars | -| `bex/mcp_server.py` | MCP server with `analyze_directory`, `get_grammar`, `get_package_grammars` | -| `bex/mdl.py` | `lang_size_score()` (default), `mdl_score()` (fallback) | -| `bex/reduce.py` | Algorithm 4 (TODS 2010) for grammar reduction | +| `bex/gbnf.py` | GBNF converter, `grammar_structure_score()` | | `bex/ensemble.py` | `infer_ensemble()` — combine multiple algorithms | - -## Experiments - -### Key Findings -1. **CRX wins on simplicity** — no post-processing needed, flat chains are interpretable -2. **`DEFAULT_COVERAGE=0.05`** — was 0.8, almost filtered everything out -3. **`min_methods=3`** — sweet spot (was 5, lost 9 FastAPI grammars) -4. **Decomposition helps diverse codebases** — RAGSAK 4→27, FastAPI 16→29 high-structure grammars -5. **Decomposition hurts structured codebases** — kotlinx.coroutines 24→15 - -### Decision Matrix (CRX vs Refined CRX) -- CRX struct ≥ 0.2 → use CRX (already good) -- CRX struct < 0.05 → use refined (flat bag) -- Group size ≤ 50 → use refined (safe to cluster) -- Group size > 50 → use CRX (refined likely trivial) - -### Research Positioning -We are **unique**: first to infer behavioral grammars from source code execution patterns. Related work: -- Panini (white-box CFG from parsers) -- Crucio (black-box CFG from examples) -- XGrammar/DOMINO (constrained decoding) -- Typify/REST (type inference) - -Our niche: discover patterns that should be inferred/enforced/typed. - -## Open Questions - -### 1. Grammar Usefulness for LLM Code Generation -MCP tools are ready but haven't validated if grammars help an LLM during generation. Need to test: -- Does constrained decoding with GBNF improve code quality? -- Do grammars reduce hallucination in call chains? -- Can grammars be used for code completion suggestions? - -### 2. Decomposition Trade-off -Decomposition helps diverse codebases but hurts already-structured ones. Need auto-detection: -- If codebase already has good structure → skip decomposition -- If codebase is diverse → apply decomposition - -### 3. Cross-Codebase Grammar Reuse -Can grammars from one project inform another? (e.g., "Spring Boot service patterns") +| `bex/mcp_server.py` | MCP server with `analyze_directory`, `get_grammar` | +| `bex/tag_preprocessor/code.py` | `preprocess_by_method()` — AST to behavioral sequences | ## How to Run ```bash -# Basic analysis -bex --include "*.py" --slice package --main-only --format yaml --output grammars.yml +# Basic analysis (all defaults ON) +python -m bex.tag_preprocessor.analyze /path/to/codebase --verbose -# With decomposition + structure filtering -bex --include "*.py" --slice package --main-only --decompose --min-structure 0.5 --format yaml +# With custom settings +python -m bex.tag_preprocessor.analyze /path/to/codebase \ + --decompose --idregex-refine --min-structure 0.5 --slice package -# Full pipeline (what we tested) -bex --include "*.kt" --slice package --main-only --split-mixed --decompose --min-structure 0.5 --format yaml +# Disable idregex refinement +python -m bex.tag_preprocessor.analyze /path/to/codebase --no-idregex-refine # MCP server -bex serve --port 8080 +python bex/mcp_server.py --port 8080 ``` ## Test Coverage -- `tests/test_distributional.py`: 23 tests (distributional clustering) -- `tests/test_decompose.py`: 12 tests (decomposition forest) -- `tests/test_gbnf.py`: 28 tests (GBNF conversion) -- `tests/test_crx_refined.py`: 20 tests (refined CRX) -- `tests/test_grammar_index.py`: 14 tests (grammar index) -- `tests/test_analyze.py`: Pipeline tests -- `tests/test_reduce.py`: Algorithm 4 tests -- `tests/test_mdl.py`: MDL scoring tests +- `tests/test_analyze.py`: Pipeline integration tests +- `tests/test_grammar.py`: AST node tests, count_words memoization +- `tests/test_mdl.py`: Language Size scoring, MDL scoring +- `tests/test_crx.py`: CRX algorithm +- `tests/test_idregex.py`: iDRegEx algorithm +- `tests/test_decompose.py`: Decomposition forest +- `tests/test_distributional.py`: Distributional clustering +- `tests/test_gbnf.py`: GBNF conversion +- `tests/test_crx_refined.py`: Refined CRX +- `tests/test_grammar_index.py`: Grammar index +- `tests/test_reduce.py`: Algorithm 4 -Total: 269 tests passing +Total: 313 tests passing + +## Decision Log + +| Decision | Choice | Rationale | +|----------|--------|-----------| +| Scoring | Language Size (not MDL) | MDL rewards short over specific (ADR-13) | +| Decomposition | ON by default | 3× more grammars, fewer pure bags | +| idregex_refine | ON by default | Gate limits to groups where it helps | +| AST representation | Full AST nodes | Type safety, memoization, no string parsing | +| Algorithm | CRX (default) | Fast, deterministic, always produces output | + +## Open Questions + +### 1. Grammar Usefulness for LLM Code Generation +MCP tools are ready but haven't validated if grammars help an LLM during generation. +Need to test: +- Does constrained decoding with GBNF improve code quality? +- Do grammars reduce hallucination in call chains? + +### 2. The 85% Bag Problem +Most grammars are orderless bags. Two possible directions: +- **Accept it**: Bags represent real diversity in method usage. Not a bug. +- **Better grouping**: If we group methods by semantic role (not just directory), + we might find ordering within sub-groups. Requires understanding method semantics. + +### 3. Cross-Codebase Grammar Reuse +Can grammars from one project inform another? (e.g., "Spring Boot service patterns") + +## Experiments Summary + +| Round | What | Result | +|-------|------|--------| +| 1-5 | Context strategies | Package grouping wins (12% coverage) | +| 6-10 | Reduce, clustering | Reduce merges states, not packages | +| 11-15 | Distributional, ensemble | CRX is sufficient, no ensemble needed | +| 16-17 | Refined CRX | Better ~78% of the time when useful, but trivial ~36% | +| 18 | Decomposition | 3× more grammars, fewer bags | +| 19 | AST migration | 54.9s→6.7s after memoization fix | +| 20 | Scoring + defaults | Language Size works, decomposition ON, idregex ON | ## Next Steps @@ -122,3 +164,4 @@ Total: 269 tests passing 2. **Auto-detect decomposition** — skip if codebase already structured 3. **Cross-project grammar reuse** — share patterns across codebases 4. **IDE integration** — grammar-aware code completion +5. **No further algorithmic changes on bags** — they're a feature, not a bug diff --git a/experiments/RESULTS.md b/experiments/RESULTS.md index 3195ecd..701044a 100644 --- a/experiments/RESULTS.md +++ b/experiments/RESULTS.md @@ -162,3 +162,82 @@ Converts textual sequences into structural shapes that repeat across packages. - `bex/gbnf.py` — SORE → GBNF converter - `tests/test_reduce.py` — 24 tests for Reduce - `tests/test_gbnf.py` — 15 tests for GBNF converter + +--- + +## Round 20: AST Pipeline + Scoring Fixes (2026-07-13) + +**Commit range:** `ea6cac5` → `8b3a454` + +### Codebases + +| Codebase | Files | Methods | Language | +|----------|-------|---------|----------| +| RAGSAK | 462 | 1609 | Kotlin | +| FastAPI | — | — | Python | +| Zod | — | — | TypeScript | + +### Scoring: Language Size over MDL (ADR-13) + +Abandoned MDL scoring — it rewards short expressions, so generic `info+` beat specific +`a.b.c.d.e+` (21% vs 98% success in Bex paper). Language Size (`lang_size_score`) chosen. + +| Metric | Bag grammar | Structured grammar | +|--------|-------------|-------------------| +| `lang_size_score` | 9975 | 20 | +| `mdl_score` | 10^12 (clamped) | 10^12 (clamped) | + +### Final Defaults + +| Parameter | Before | After | +|-----------|--------|-------| +| `decompose` | False | **True** | +| `max_seq_length` | 5 | **4** | +| `idregex_refine` | False | **True** | +| `_COUNT_CAP` | 10^12 | **10^30** | + +### Results + +| Codebase | Grammars | Pure Bags | Structured | Bag % | +|----------|----------|-----------|------------|-------| +| RAGSAK (v4) | 126 | 6 | 120 | 4.8% | +| FastAPI (v3) | 143 | 26 | 117 | 18.2% | +| Zod (v3) | 23 | 5 | 18 | 21.7% | + +**Quality breakdown:** ~85% of grammars across codebases remain orderless bags `(A|B|C)+`. +The ~15% that are structured represent real sequential flows: +- Web controller tests: `post→jsonPath→isEqualTo→exchange→expectStatus` +- API client patterns: `request→header→send→statusCode→jsonPath` +- Builder chains: `builder→field→value→build→validate` + +### iDRegEx Findings + +iDRegEx does NOT help at small scale. On 3-method groups, iDRegEx achieves only 3.8× +tighter (below the 10× gate threshold). The gate correctly rejects it. + +| Group size | CRX lang_size | iDRegEx lang_size | Ratio | +|------------|--------------|-------------------|-------| +| 3 methods | 15 | 4 | 3.8× | +| 4 methods (`storage`) | — | — | 91× (outlier) | + +Bags survive because: +1. CRX emits one grammar deterministically (no alternative to compare) +2. `lang_size_score` only ranks **between** algorithms, not within CRX's own output +3. iDRegEx is too slow for large groups (200s+ timeout on 2036m FastAPI tests) + +### Key Insight + +The grammar inference pipeline is fundamentally limited by the input: if methods in a +package don't share a sequential calling pattern, no algorithm can find one. The ~15% +structured grammars represent genuinely reusable patterns; the ~85% bags represent +packages with diverse, unrelated methods grouped only by directory proximity. + +### Files + +- `bex/grammar.py`: AST nodes, `_count_concat` memoization, `_COUNT_CAP = 10^30` +- `bex/crx.py`: CRX algorithm (AST-based) +- `bex/mdl.py`: `lang_size_score`, `model_cost`, `data_cost` +- `bex/idregex.py`: iDRegEx algorithm +- `bex/decompose.py`: sequence decomposition +- `bex/tag_preprocessor/analyze.py`: pipeline orchestration, all defaults +- `experiments/results/round20_ast_verify/`: full experiment data (v2/v3/v4)