From 92af932e9d64091275cd1a10532bb77697ad5658 Mon Sep 17 00:00:00 2001
From: tobjend <tobend85@gmail.com>
Date: Sun, 12 Jul 2026 02:57:40 +0200
Subject: [PATCH] =?UTF-8?q?docs:=20Round=2011=20results=20=E2=80=94=20pipe?=
 =?UTF-8?q?line=20speed=20+=20GBNF=20conversion?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

- iDRegEx opt-in: Flask 55s→2.8s, RAGSAK 74s→13s, FastAPI 30s→30s
- GBNF validation: 130/130 OK, 0 FAIL (17 malformed skipped)
- Grammar quality: 43% structured, rest flat/trivial
- Logged to experiments/EXPERIMENT_LOG.md
---
 experiments/EXPERIMENT_LOG.md | 50 +++++++++++++++++++++++++++++++++++
 1 file changed, 50 insertions(+)

diff --git a/experiments/EXPERIMENT_LOG.md b/experiments/EXPERIMENT_LOG.md
index 6c95970..63bd07d 100644
--- a/experiments/EXPERIMENT_LOG.md
+++ b/experiments/EXPERIMENT_LOG.md
@@ -427,3 +427,53 @@ CRX method. Keep standard CRX available for comparison.
 - Test clustering on Flask, Coroutines, FastAPI
 - Try better clustering features (k-mer, edit distance, prefix sharing)
 - Evaluate: does tighter grammar → better code completion / convention docs?
+
+---
+
+## Round 11: Pipeline Speed + GBNF Conversion (commits `bc7d3b6`, `3468813`)
+
+**Hypothesis:** iDRegEx in the ensemble is the bottleneck. GBNF conversion needs error handling.
+
+**Method:**
+- Made iDRegEx opt-in via `--idregex` flag (was running on every group)
+- Added `validate_sore()` — skip malformed SOREs gracefully
+- Fixed OverflowError: `lang_size_score` produces huge ints for large disjunctions
+- Fixed GBNF tokenizer: strip newlines from literals
+
+**Results — Pipeline Speed:**
+| Codebase | Before (with iDRegEx) | After (CRX only) | Speedup |
+|----------|----------------------|-------------------|---------|
+| Flask    | 55s+                 | 2.8s              | 20×     |
+| RAGSAK   | 74s                  | 13s               | 5.7×    |
+| FastAPI  | hung at 300s         | 30s               | >10×    |
+
+Root cause: `src/flask/json` (50 methods) alone took 55s in iDRegEx. Flask's `tests` group (962 methods) would have been worse.
+
+**Results — GBNF Conversion:**
+| Codebase | Grammars | GBNF OK | GBNF FAIL | Malformed (skipped) |
+|----------|----------|---------|-----------|---------------------|
+| Flask    | 5        | 5       | 0         | 0                   |
+| RAGSAK   | 19       | 19      | 0         | 11                  |
+| FastAPI  | 106      | 106     | 0         | 6                   |
+| **Total**| **130**  | **130** | **0**     | **17**              |
+
+17 malformed SOREs contain raw code (e.g. `w_body=`, `(+,+:N+...)`) — preprocessing bug, not parser issue.
+
+**Grammar Quality Analysis:**
+- **Structured** (has ordering via `.`, `?`): RAGSAK 18, FastAPI 94, Flask 3
+- **Flat disjunction** (bag of symbols): RAGSAK 1, FastAPI 10, Flask 2
+- **Trivial** (single symbol): RAGSAK 0, FastAPI 2, Flask 0
+
+Best structured examples:
+- `return.render_template+` — clear: return, then render_template one or more times
+- `assertNull?.parseS3Location.error+?.(assertEquals+bucket)+?.key?` — test flow
+- `buildObservationContext?.shouldRetrieve?.(ASK+ChatResponse+...)?...` — agent flow
+- `if.(img+item_id).(FileResponse+else+media_type+return)+.JSONResponse+?.status_code?.content?` — if/else structure
+
+**Decision:** iDRegEx stays opt-in. GBNF validation catches malformed SOREs early.
+The structured grammars (43% of all grammars) capture real calling conventions.
+
+**Open questions:**
+- 1129-file FastAPI has 30 diverse groups — need better grouping for large codebases
+- Malformed SOREs from raw code in symbol names — need upstream fix in code.py
+- Flat disjunctions are noisy — should we filter by grammar complexity?