diff --git a/experiments/EXPERIMENT_LOG.md b/experiments/EXPERIMENT_LOG.md index 6c95970..63bd07d 100644 --- a/experiments/EXPERIMENT_LOG.md +++ b/experiments/EXPERIMENT_LOG.md @@ -427,3 +427,53 @@ CRX method. Keep standard CRX available for comparison. - Test clustering on Flask, Coroutines, FastAPI - Try better clustering features (k-mer, edit distance, prefix sharing) - Evaluate: does tighter grammar → better code completion / convention docs? + +--- + +## Round 11: Pipeline Speed + GBNF Conversion (commits `bc7d3b6`, `3468813`) + +**Hypothesis:** iDRegEx in the ensemble is the bottleneck. GBNF conversion needs error handling. + +**Method:** +- Made iDRegEx opt-in via `--idregex` flag (was running on every group) +- Added `validate_sore()` — skip malformed SOREs gracefully +- Fixed OverflowError: `lang_size_score` produces huge ints for large disjunctions +- Fixed GBNF tokenizer: strip newlines from literals + +**Results — Pipeline Speed:** +| Codebase | Before (with iDRegEx) | After (CRX only) | Speedup | +|----------|----------------------|-------------------|---------| +| Flask | 55s+ | 2.8s | 20× | +| RAGSAK | 74s | 13s | 5.7× | +| FastAPI | hung at 300s | 30s | >10× | + +Root cause: `src/flask/json` (50 methods) alone took 55s in iDRegEx. Flask's `tests` group (962 methods) would have been worse. + +**Results — GBNF Conversion:** +| Codebase | Grammars | GBNF OK | GBNF FAIL | Malformed (skipped) | +|----------|----------|---------|-----------|---------------------| +| Flask | 5 | 5 | 0 | 0 | +| RAGSAK | 19 | 19 | 0 | 11 | +| FastAPI | 106 | 106 | 0 | 6 | +| **Total**| **130** | **130** | **0** | **17** | + +17 malformed SOREs contain raw code (e.g. `w_body=`, `(+,+:N+...)`) — preprocessing bug, not parser issue. + +**Grammar Quality Analysis:** +- **Structured** (has ordering via `.`, `?`): RAGSAK 18, FastAPI 94, Flask 3 +- **Flat disjunction** (bag of symbols): RAGSAK 1, FastAPI 10, Flask 2 +- **Trivial** (single symbol): RAGSAK 0, FastAPI 2, Flask 0 + +Best structured examples: +- `return.render_template+` — clear: return, then render_template one or more times +- `assertNull?.parseS3Location.error+?.(assertEquals+bucket)+?.key?` — test flow +- `buildObservationContext?.shouldRetrieve?.(ASK+ChatResponse+...)?...` — agent flow +- `if.(img+item_id).(FileResponse+else+media_type+return)+.JSONResponse+?.status_code?.content?` — if/else structure + +**Decision:** iDRegEx stays opt-in. GBNF validation catches malformed SOREs early. +The structured grammars (43% of all grammars) capture real calling conventions. + +**Open questions:** +- 1129-file FastAPI has 30 diverse groups — need better grouping for large codebases +- Malformed SOREs from raw code in symbol names — need upstream fix in code.py +- Flat disjunctions are noisy — should we filter by grammar complexity?