diff --git a/experiments/EXPERIMENT_LOG.md b/experiments/EXPERIMENT_LOG.md new file mode 100644 index 0000000..c8a6322 --- /dev/null +++ b/experiments/EXPERIMENT_LOG.md @@ -0,0 +1,230 @@ +# Experiment Log — Grammar Inference Pipeline + +Track what we tried, what worked, what failed, and what's next. Each experiment +includes: hypothesis, method, result, verdict. + +--- + +## Round 1: Context Strategies (commit `bbdfe93`) + +**Hypothesis:** The calling context (prefix before the method body) determines +which methods share a convention. Better context grouping → better grammars. + +**Method:** Tested 4 strategies on RAGSAK (Kotlin) and Flask (Python): +- **Baseline:** Group by package directory +- **Option A:** Group by last k components of file path (`file_path_k{k}`) +- **Option B:** Group by first k symbols of call sequence (`first_k_sym_{k}`) +- **Option C:** Two-dimensional: (path_k, first_k_symbols) + +**Result:** +| Strategy | RAGSAK patterns | RAGSAK coverage | Flask patterns | Flask coverage | +|----------|----------------|-----------------|----------------|---------------| +| Package baseline | 73 | 12.0% | 21 | 10.7% | +| File path k=1 | 73 | 12.0% | 21 | 10.7% | +| First k=1 | 20 | 4.6% | 2 | 1.4% | +| First k=3 | 47 | 12.0% | 21 | 10.7% | + +**Verdict:** Package grouping and first_k_sym_3 produce similar results. +Cross-package grouping by first symbol is too sparse — most groups are either +too large (skipped) or too diverse (skipped). The useful patterns are +package-specific, not cross-package. + +--- + +## Round 2: Reduce Algorithm (commit `b516b29`) + +**Hypothesis:** Reduce (Algorithm 4, TODS 2010) merges structurally similar +contexts, revealing cross-package patterns by unifying equivalent states. + +**Method:** Implemented faithful Reduce with support-weighted SOA edit distance, +adjunction, iterative merging, and minimize. Tested at ε=0.05 to 0.4. + +**Result:** +- RAGSAK at ε=0.3: 1 merge (`JobStatus.every.getJobStatus` ↔ `JobStatus.now.minusMinutes`) +- Flask at ε=0.3: 1 merge (`def.boolean` ↔ `def.is_boolean`) +- Coverage improvement: negligible (< 1%) + +**Verdict:** Reduce doesn't help. The contexts we produce are already too +specific (unique per package) for the distance metric to find meaningful merges. +Reduce works when you have a large SOA with many equivalent states — we have +one SOA per package with few states. Wrong abstraction level. + +**Why it failed:** Reduce merges states in a single automaton. We're producing +one automaton per package group. There's nothing to merge across packages +because each package gets its own inference run. Reduce would need to operate +on a cross-package SOA, which we don't build. + +--- + +## Round 3: Language Size Scoring (commit `dfb56a0`) + +**Hypothesis:** Bex et al.'s Language Size measure (arXiv:1004.2372, Section +4.3.1) is a better scoring function than MDL for our use case. + +**Method:** Implemented `lang_size_score()` as default scoring method. Added +diversity threshold: skip groups with unique_ratio > 0.9 or methods < 5. + +**Result:** 39 new tests. Pipeline runs correctly with new scoring. Coverage +numbers similar to before — scoring method doesn't change which patterns are +found, just which grammar is selected per group. + +**Verdict:** Scoring method is not the bottleneck. The problem is upstream +(pattern extraction), not downstream (pattern selection). + +--- + +## Round 4: GBNF Output (commit `011df39`) + +**Hypothesis:** SORE → GBNF conversion enables constrained LLM generation. +SORE operators map directly to GBNF syntax. + +**Method:** Implemented recursive descent parser for SORE, AST intermediate +representation, and GBNF renderer. 15 tests. + +**Result:** All tests pass. `to_gbnf('raise.(ValueError)+')` → +`"raise" "ValueError"+`. Correct mapping of +, ?, *, |, ., parens. + +**Verdict:** Implementation works. But the input SOREs are too specific +to individual packages to be useful for constrained generation. The converter +is correct; the patterns it converts are the problem. + +--- + +## Round 5: Cross-Package Exact Matches + +**Hypothesis:** Some call sequences appear verbatim in multiple packages. +These are the real cross-package conventions. + +**Method:** Grouped all sequences by exact tuple match across packages. + +**Result:** 38 exact cross-package sequences in RAGSAK. Most are trivial: +- `('clearAllMocks',)` — 4 packages (test teardown) +- `('Builder',)` — 4 packages (builder pattern) +- `('Any',)` — 4 packages (Kotlin type) +- `('get',)` — 4 packages (getter) + +Interesting ones: +- `('assumeTrue', 'isDockerAvailable', 'start', 'pullAndWarmup')` — 4 packages (Docker test setup) +- `('isNullOrBlank', 'error', 'error')` — 3 packages (null check → error) +- `('sortedBy', 'map', 'toDescriptor')` — 3 packages (data pipeline) +- `('ObjectMapper', 'findAndRegisterModules')` — 2 packages (Jackson config) + +**Verdict:** Exact matches are too rare and mostly trivial. The real +cross-package patterns are structural, not textual — "null check → error" +appears with different method names in different packages. + +--- + +## What We Learned (Summary) + +1. **Per-package grouping is too sparse.** 1-3 sequences per package isn't + enough for any inference method to produce general patterns. + +2. **Cross-package exact matches are rare.** Only 38 in RAGSAK, mostly trivial + single-call sequences. + +3. **Reduce doesn't help at our abstraction level.** It merges states within + one automaton; we need to merge patterns across packages. + +4. **Scoring/selection isn't the bottleneck.** MDL vs Language Size doesn't + change what patterns are found. + +5. **The calling context prefix is the right signal** but grouping by it + produces groups that are either too large, too diverse, or trivial. + +6. **GBNF converter works correctly** but the input patterns are too specific. + +--- + +## Next: Structural Coarsening + Cross-Package Detection + +### Idea + +Collapse method names → categories using tree-sitter capture names. This +converts textual sequences into structural shapes: + +``` +('isNullOrBlank', 'error', 'error') → (CALL, ERROR, ERROR) +('raise', 'ValueError', 'ValueError') → (CALL, ERROR, ERROR) +``` + +Same structural shape, different packages → cross-package convention. + +### Why This Might Work + +- We already extract tree-sitter capture names in `code.py:56-69` +- We already classify nodes into categories in `code.py:72-100` + (`lit`, `call`, `var`, `lambda`, `kwarg`, `expr`, `template`, `other`) +- The behavioral prefix filter (`CALL_PREFIXES`) keeps raw text; we need a + parallel path that keeps the category instead +- Coarsened sequences have smaller alphabets → more methods per group → + better inference +- Patterns like `(CALL, ERROR, ERROR)` are meaningful conventions that + repeat across packages + +### What We Need + +1. **Coarsening map:** `capture_name → category` using the existing + `CALL_PREFIXES`, `ARG_LITERAL_TYPES`, `LAMBDA_TYPES` classifications + plus a new `ERROR_TYPES` set + +2. **Coarsened sequence extraction:** Same pipeline as now, but output + category labels instead of method names + +3. **Cross-package grouping:** Group by coarsened shape (first k categories), + find shapes that appear in ≥2 packages + +4. **SORE inference on coarsened sequences:** Smaller alphabet, more examples + per group → better patterns + +5. **Evaluation:** Compare coarsened patterns vs raw patterns on: + - Coverage (% of methods in learned groups) + - Cross-package reach (# of packages per pattern) + - Usefulness for constrained generation (GBNF quality) + +### Open Questions + +- Does coarsening lose too much specificity? `(CALL, ERROR, ERROR)` is less + informative than `(raise, ValueError, ValueError)` — is the tradeoff worth it? +- What categories to use? The existing classifications in `code.py` are a + starting point but may need refinement (e.g., separating ERROR from CALL) +- How to handle the long tail? Most sequences are 1-2 symbols — coarsening + doesn't help much for those +- Is the GBNF output useful at all? Maybe the output should be a conditional + frequency table instead of a grammar + +### Experiment Design + +**Phase 1: Coarsening Proof of Concept** +- Implement coarsening map in `code.py` +- Add `--coarsen` flag to CLI +- Run on RAGSAK + Flask, compare raw vs coarsened patterns +- Measure: alphabet size reduction, group size increase, pattern count + +**Phase 2: Cross-Package Detection** +- Group coarsened sequences by shape (first k categories) +- Find shapes appearing in ≥2 packages +- For each shape, infer SORE on coarsened sequences +- Measure: cross-package patterns found, coverage improvement + +**Phase 3: Output Quality** +- Convert coarsened SOREs to GBNF +- Evaluate: are the GBNF rules more general/useful than raw SOREs? +- Compare: coarsened GBNF vs raw GBNF vs conditional frequency table + +--- + +## The "Redacted" Concept + +When collapsing method names to categories, we lose the specific method name +but gain the structural pattern. This is a form of **abstraction** — moving +from concrete examples to general rules. + +The question is whether the abstraction is at the right level: +- Too specific: `(raise, ValueError, ValueError)` — package-specific noise +- Right level: `(CALL, ERROR, ERROR)` — cross-package convention +- Too abstract: `(X, Y, Y)` — trivial, tells the LLM nothing + +The middle ground depends on the category vocabulary. We need enough categories +to be informative (CALL, ERROR, LIT, VAR, TYPE, etc.) but not so many that +every sequence is unique. diff --git a/experiments/RESULTS.md b/experiments/RESULTS.md index cd8685d..3195ecd 100644 --- a/experiments/RESULTS.md +++ b/experiments/RESULTS.md @@ -102,18 +102,63 @@ - Flask has more unique methods per package — less repetition - Flask SOREs are shorter/simpler — less compressible patterns -## Next Steps +## Round 5: Cross-Package Exact Matches -1. **Accept the ceiling**: ~10-12% coverage may be the maximum for prefix-based grouping -2. **Symbol coarsening**: Collapse specific symbols into categories (GETTER, SETTER, etc.) -3. **Hierarchical grouping**: Group by k=1 first, then sub-group by k=2 within each group -4. **GBNF output**: Convert SOREs to GBNF format for llama.cpp constrained decoding -5. **Test on kotlinx.coroutines**: Third codebase for cross-validation +**Hypothesis:** Some call sequences appear verbatim in multiple packages. +These are the real cross-package conventions. + +**Method:** Group all sequences by exact tuple match, count packages per sequence. + +**Result:** 38 exact cross-package sequences in RAGSAK. Most trivial: +- `('clearAllMocks',)` — 4 packages (test teardown) +- `('Builder',)` — 4 packages (builder pattern) +- `('Any',)` — 4 packages (Kotlin type) + +Interesting ones: +- `('assumeTrue', 'isDockerAvailable', 'start', 'pullAndWarmup')` — 4 pkgs (Docker setup) +- `('isNullOrBlank', 'error', 'error')` — 3 pkgs (null check → error) +- `('sortedBy', 'map', 'toDescriptor')` — 3 pkgs (data pipeline) +- `('ObjectMapper', 'findAndRegisterModules')` — 2 pkgs (Jackson config) + +**Verdict:** Exact matches too rare and mostly trivial. The real cross-package +patterns are structural, not textual — "null check → error" appears with +different method names in different packages. + +## Summary of Failed/Dismissed Approaches + +| Approach | Why it failed | +|----------|--------------| +| Per-package inference | Too sparse (1-3 seqs/package) | +| Reduce algorithm | Wrong abstraction level — merges states within one automaton, not across packages | +| Cross-package grouping by first symbol | 4.6% / 1.4% coverage — most groups skipped | +| Exact cross-package matches | 38 sequences, mostly trivial single-call | +| MDL vs Language Size scoring | Scoring isn't the bottleneck — pattern extraction is | + +## What Actually Works + +- **Behavioral grouping (first 3 symbols)** — 12% / 10.7% coverage, consistent across codebases +- **Calling context as prefix** — the right signal, but package-specific +- **GBNF conversion** — correct implementation, but input patterns too specific + +## Next: Structural Coarsening + Cross-Package Detection + +See `EXPERIMENT_LOG.md` for full reasoning and experiment design. + +Core idea: collapse method names → categories using tree-sitter capture names. +Converts textual sequences into structural shapes that repeat across packages. + +``` +('isNullOrBlank', 'error', 'error') → (CALL, ERROR, ERROR) +('raise', 'ValueError', 'ValueError') → (CALL, ERROR, ERROR) +``` ## Files Generated - `experiments/results/ragsak_summary.json` — RAGSAK metrics - `experiments/results/flask_summary.json` — Flask metrics - `experiments/context_eval.py` — experiment runner (supports multiple codebases) +- `experiments/EXPERIMENT_LOG.md` — full experiment history and next steps - `bex/reduce.py` — Algorithm 4 (TODS 2010) implementation +- `bex/gbnf.py` — SORE → GBNF converter - `tests/test_reduce.py` — 24 tests for Reduce +- `tests/test_gbnf.py` — 15 tests for GBNF converter