tobi/grammar-inference-engine
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

docs: Round 11 results — pipeline speed + GBNF conversion

Browse source 
- iDRegEx opt-in: Flask 55s→2.8s, RAGSAK 74s→13s, FastAPI 30s→30s
- GBNF validation: 130/130 OK, 0 FAIL (17 malformed skipped)
- Grammar quality: 43% structured, rest flat/trivial
- Logged to experiments/EXPERIMENT_LOG.md
This commit is contained in:
tobjend 2026-07-12 02:57:40 +02:00
parent 3468813ec8
commit 92af932e9d
1 changed files with 50 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

50
experiments/EXPERIMENT_LOG.md
View file

@ -427,3 +427,53 @@ CRX method. Keep standard CRX available for comparison.
- Test clustering on Flask, Coroutines, FastAPI - Test clustering on Flask, Coroutines, FastAPI
- Try better clustering features (k-mer, edit distance, prefix sharing) - Try better clustering features (k-mer, edit distance, prefix sharing)
- Evaluate: does tighter grammar → better code completion / convention docs? - Evaluate: does tighter grammar → better code completion / convention docs?
---
## Round 11: Pipeline Speed + GBNF Conversion (commits `bc7d3b6`, `3468813`)
**Hypothesis:** iDRegEx in the ensemble is the bottleneck. GBNF conversion needs error handling.
**Method:**
- Made iDRegEx opt-in via `--idregex` flag (was running on every group)
- Added `validate_sore()` — skip malformed SOREs gracefully
- Fixed OverflowError: `lang_size_score` produces huge ints for large disjunctions
- Fixed GBNF tokenizer: strip newlines from literals
**Results — Pipeline Speed:**
| Codebase | Before (with iDRegEx) | After (CRX only) | Speedup |
|----------|----------------------|-------------------|---------|
| Flask | 55s+ | 2.8s | 20× |
| RAGSAK | 74s | 13s | 5.7× |
| FastAPI | hung at 300s | 30s | >10× |
Root cause: `src/flask/json` (50 methods) alone took 55s in iDRegEx. Flask's `tests` group (962 methods) would have been worse.
**Results — GBNF Conversion:**
| Codebase | Grammars | GBNF OK | GBNF FAIL | Malformed (skipped) |
|----------|----------|---------|-----------|---------------------|
| Flask | 5 | 5 | 0 | 0 |
| RAGSAK | 19 | 19 | 0 | 11 |
| FastAPI | 106 | 106 | 0 | 6 |
| **Total**| **130** | **130** | **0** | **17** |
17 malformed SOREs contain raw code (e.g. `w_body=`, `(+,+:N+...)`) — preprocessing bug, not parser issue.
**Grammar Quality Analysis:**
- **Structured** (has ordering via `.`, `?`): RAGSAK 18, FastAPI 94, Flask 3
- **Flat disjunction** (bag of symbols): RAGSAK 1, FastAPI 10, Flask 2
- **Trivial** (single symbol): RAGSAK 0, FastAPI 2, Flask 0
Best structured examples:
- `return.render_template+` — clear: return, then render_template one or more times
- `assertNull?.parseS3Location.error+?.(assertEquals+bucket)+?.key?` — test flow
- `buildObservationContext?.shouldRetrieve?.(ASK+ChatResponse+...)?...` — agent flow
- `if.(img+item_id).(FileResponse+else+media_type+return)+.JSONResponse+?.status_code?.content?` — if/else structure
**Decision:** iDRegEx stays opt-in. GBNF validation catches malformed SOREs early.
The structured grammars (43% of all grammars) capture real calling conventions.
**Open questions:**
- 1129-file FastAPI has 30 diverse groups — need better grouping for large codebases
- Malformed SOREs from raw code in symbol names — need upstream fix in code.py
- Flat disjunctions are noisy — should we filter by grammar complexity?