diff --git a/experiments/EXPERIMENT_LOG.md b/experiments/EXPERIMENT_LOG.md index 63bd07d..79bb9df 100644 --- a/experiments/EXPERIMENT_LOG.md +++ b/experiments/EXPERIMENT_LOG.md @@ -477,3 +477,41 @@ The structured grammars (43% of all grammars) capture real calling conventions. - 1129-file FastAPI has 30 diverse groups — need better grouping for large codebases - Malformed SOREs from raw code in symbol names — need upstream fix in code.py - Flat disjunctions are noisy — should we filter by grammar complexity? + +--- + +## Round 12: Grammar Structure Scoring (commit `e62fffc`) + +**Hypothesis:** Flat disjunctions `(a+b+c+...+z)+` are CRX over-approximations — they list symbols without ordering and aren't useful for code completion or convention docs. We can filter them. + +**Method:** `grammar_structure_score(sore)` — measures structural richness: +- Count dots (`.`), optional (`?`), repetition outside parens (`+`, `*`) = ordering ops +- Count disjunction parts inside parens = noise +- Score = `min(1.0, struct_ratio * 3)`, penalized if high disjunction ratio with no concatenation + +**Thresholds:** +- **Structured** (≥0.5): Has ordering — tells you the SEQUENCE things happen +- **Semi** (0.2-0.5): Partial structure +- **Flat** (0.05-0.2): Bag of symbols — CRX over-approximation +- **Trivial** (<0.05): Single symbol or empty + +**Results with `min_structure=0.2`:** +| Codebase | Before | After | Kept | Dropped | +|----------|--------|-------|------|---------| +| Flask | 5 | 2 | 2 | 3 flat + 6 diverse | +| RAGSAK | 19 | 10 | 10 | 9 flat + 114 diverse/malformed | +| FastAPI | 106 | 47 | 47 | 59 flat + 36 diverse/malformed | +| **Total**| **130**| **59**| **59**| **218** | + +**Example kept grammars (score ≥ 0.2):** +- `return.render_template+` (score=0.29) — Flask test convention +- `assertNull?.parseS3Location.error+?.(assertEquals+bucket)+?.key?` (0.59) — RAGSAK test flow +- `if?.return?.(token+x_token)?.raise?.HTTPException+?.status_code?.detail?` (0.76) — FastAPI auth pattern +- `return.request?.scope?.get+?` (1.00) — maximally structured + +**Example dropped grammars (score < 0.2):** +- `(@+Blueprint+__name__+app+append+class+client+...)+` (0.02) — Flask test bag-of-words +- `(Any+BlueprintSetupState+ValueError+...)+` (0.01) — Flask sansio bag-of-words +- `(@+FastAPI+TestClient+app+client+data+...)+` (0.10) — FastAPI test bag-of-words + +**Decision:** `min_structure=0.2` is the right default. Flat bags are noise — they tell you what symbols exist but not how they're used. The 59 structured grammars capture real calling conventions with ordering information.