experiments/EXPERIMENT_LOG.md

@@ -0,0 +1,230 @@
+# Experiment Log — Grammar Inference Pipeline
+
+Track what we tried, what worked, what failed, and what's next. Each experiment
+includes: hypothesis, method, result, verdict.
+
+---
+
+## Round 1: Context Strategies (commit `bbdfe93`)
+
+**Hypothesis:** The calling context (prefix before the method body) determines
+which methods share a convention. Better context grouping → better grammars.
+
+**Method:** Tested 4 strategies on RAGSAK (Kotlin) and Flask (Python):
+- **Baseline:** Group by package directory
+- **Option A:** Group by last k components of file path (`file_path_k{k}`)
+- **Option B:** Group by first k symbols of call sequence (`first_k_sym_{k}`)
+- **Option C:** Two-dimensional: (path_k, first_k_symbols)
+
+**Result:**
+| Strategy | RAGSAK patterns | RAGSAK coverage | Flask patterns | Flask coverage |
+|----------|----------------|-----------------|----------------|---------------|
+| Package baseline | 73 | 12.0% | 21 | 10.7% |
+| File path k=1 | 73 | 12.0% | 21 | 10.7% |
+| First k=1 | 20 | 4.6% | 2 | 1.4% |
+| First k=3 | 47 | 12.0% | 21 | 10.7% |
+
+**Verdict:** Package grouping and first_k_sym_3 produce similar results.
+Cross-package grouping by first symbol is too sparse — most groups are either
+too large (skipped) or too diverse (skipped). The useful patterns are
+package-specific, not cross-package.
+
+---
+
+## Round 2: Reduce Algorithm (commit `b516b29`)
+
+**Hypothesis:** Reduce (Algorithm 4, TODS 2010) merges structurally similar
+contexts, revealing cross-package patterns by unifying equivalent states.
+
+**Method:** Implemented faithful Reduce with support-weighted SOA edit distance,
+adjunction, iterative merging, and minimize. Tested at ε=0.05 to 0.4.
+
+**Result:**
+- RAGSAK at ε=0.3: 1 merge (`JobStatus.every.getJobStatus` ↔ `JobStatus.now.minusMinutes`)
+- Flask at ε=0.3: 1 merge (`def.boolean` ↔ `def.is_boolean`)
+- Coverage improvement: negligible (< 1%)
+
+**Verdict:** Reduce doesn't help. The contexts we produce are already too
+specific (unique per package) for the distance metric to find meaningful merges.
+Reduce works when you have a large SOA with many equivalent states — we have
+one SOA per package with few states. Wrong abstraction level.
+
+**Why it failed:** Reduce merges states in a single automaton. We're producing
+one automaton per package group. There's nothing to merge across packages
+because each package gets its own inference run. Reduce would need to operate
+on a cross-package SOA, which we don't build.
+
+---
+
+## Round 3: Language Size Scoring (commit `dfb56a0`)
+
+**Hypothesis:** Bex et al.'s Language Size measure (arXiv:1004.2372, Section
+4.3.1) is a better scoring function than MDL for our use case.
+
+**Method:** Implemented `lang_size_score()` as default scoring method. Added
+diversity threshold: skip groups with unique_ratio > 0.9 or methods < 5.
+
+**Result:** 39 new tests. Pipeline runs correctly with new scoring. Coverage
+numbers similar to before — scoring method doesn't change which patterns are
+found, just which grammar is selected per group.
+
+**Verdict:** Scoring method is not the bottleneck. The problem is upstream
+(pattern extraction), not downstream (pattern selection).
+
+---
+
+## Round 4: GBNF Output (commit `011df39`)
+
+**Hypothesis:** SORE → GBNF conversion enables constrained LLM generation.
+SORE operators map directly to GBNF syntax.
+
+**Method:** Implemented recursive descent parser for SORE, AST intermediate
+representation, and GBNF renderer. 15 tests.
+
+**Result:** All tests pass. `to_gbnf('raise.(ValueError)+')` →
+`"raise" "ValueError"+`. Correct mapping of +, ?, *, |, ., parens.
+
+**Verdict:** Implementation works. But the input SOREs are too specific
+to individual packages to be useful for constrained generation. The converter
+is correct; the patterns it converts are the problem.
+
+---
+
+## Round 5: Cross-Package Exact Matches
+
+**Hypothesis:** Some call sequences appear verbatim in multiple packages.
+These are the real cross-package conventions.
+
+**Method:** Grouped all sequences by exact tuple match across packages.
+
+**Result:** 38 exact cross-package sequences in RAGSAK. Most are trivial:
+- `('clearAllMocks',)` — 4 packages (test teardown)
+- `('Builder',)` — 4 packages (builder pattern)
+- `('Any',)` — 4 packages (Kotlin type)
+- `('get',)` — 4 packages (getter)
+
+Interesting ones:
+- `('assumeTrue', 'isDockerAvailable', 'start', 'pullAndWarmup')` — 4 packages (Docker test setup)
+- `('isNullOrBlank', 'error', 'error')` — 3 packages (null check → error)
+- `('sortedBy', 'map', 'toDescriptor')` — 3 packages (data pipeline)
+- `('ObjectMapper', 'findAndRegisterModules')` — 2 packages (Jackson config)
+
+**Verdict:** Exact matches are too rare and mostly trivial. The real
+cross-package patterns are structural, not textual — "null check → error"
+appears with different method names in different packages.
+
+---
+
+## What We Learned (Summary)
+
+1. **Per-package grouping is too sparse.** 1-3 sequences per package isn't
+   enough for any inference method to produce general patterns.
+
+2. **Cross-package exact matches are rare.** Only 38 in RAGSAK, mostly trivial
+   single-call sequences.
+
+3. **Reduce doesn't help at our abstraction level.** It merges states within
+   one automaton; we need to merge patterns across packages.
+
+4. **Scoring/selection isn't the bottleneck.** MDL vs Language Size doesn't
+   change what patterns are found.
+
+5. **The calling context prefix is the right signal** but grouping by it
+   produces groups that are either too large, too diverse, or trivial.
+
+6. **GBNF converter works correctly** but the input patterns are too specific.
+
+---
+
+## Next: Structural Coarsening + Cross-Package Detection
+
+### Idea
+
+Collapse method names → categories using tree-sitter capture names. This
+converts textual sequences into structural shapes:
+
+```
+('isNullOrBlank', 'error', 'error')     →  (CALL, ERROR, ERROR)
+('raise', 'ValueError', 'ValueError')   →  (CALL, ERROR, ERROR)
+```
+
+Same structural shape, different packages → cross-package convention.
+
+### Why This Might Work
+
+- We already extract tree-sitter capture names in `code.py:56-69`
+- We already classify nodes into categories in `code.py:72-100`
+  (`lit`, `call`, `var`, `lambda`, `kwarg`, `expr`, `template`, `other`)
+- The behavioral prefix filter (`CALL_PREFIXES`) keeps raw text; we need a
+  parallel path that keeps the category instead
+- Coarsened sequences have smaller alphabets → more methods per group →
+  better inference
+- Patterns like `(CALL, ERROR, ERROR)` are meaningful conventions that
+  repeat across packages
+
+### What We Need
+
+1. **Coarsening map:** `capture_name → category` using the existing
+   `CALL_PREFIXES`, `ARG_LITERAL_TYPES`, `LAMBDA_TYPES` classifications
+   plus a new `ERROR_TYPES` set
+
+2. **Coarsened sequence extraction:** Same pipeline as now, but output
+   category labels instead of method names
+
+3. **Cross-package grouping:** Group by coarsened shape (first k categories),
+   find shapes that appear in ≥2 packages
+
+4. **SORE inference on coarsened sequences:** Smaller alphabet, more examples
+   per group → better patterns
+
+5. **Evaluation:** Compare coarsened patterns vs raw patterns on:
+   - Coverage (% of methods in learned groups)
+   - Cross-package reach (# of packages per pattern)
+   - Usefulness for constrained generation (GBNF quality)
+
+### Open Questions
+
+- Does coarsening lose too much specificity? `(CALL, ERROR, ERROR)` is less
+  informative than `(raise, ValueError, ValueError)` — is the tradeoff worth it?
+- What categories to use? The existing classifications in `code.py` are a
+  starting point but may need refinement (e.g., separating ERROR from CALL)
+- How to handle the long tail? Most sequences are 1-2 symbols — coarsening
+  doesn't help much for those
+- Is the GBNF output useful at all? Maybe the output should be a conditional
+  frequency table instead of a grammar
+
+### Experiment Design
+
+**Phase 1: Coarsening Proof of Concept**
+- Implement coarsening map in `code.py`
+- Add `--coarsen` flag to CLI
+- Run on RAGSAK + Flask, compare raw vs coarsened patterns
+- Measure: alphabet size reduction, group size increase, pattern count
+
+**Phase 2: Cross-Package Detection**
+- Group coarsened sequences by shape (first k categories)
+- Find shapes appearing in ≥2 packages
+- For each shape, infer SORE on coarsened sequences
+- Measure: cross-package patterns found, coverage improvement
+
+**Phase 3: Output Quality**
+- Convert coarsened SOREs to GBNF
+- Evaluate: are the GBNF rules more general/useful than raw SOREs?
+- Compare: coarsened GBNF vs raw GBNF vs conditional frequency table
+
+---
+
+## The "Redacted" Concept
+
+When collapsing method names to categories, we lose the specific method name
+but gain the structural pattern. This is a form of **abstraction** — moving
+from concrete examples to general rules.
+
+The question is whether the abstraction is at the right level:
+- Too specific: `(raise, ValueError, ValueError)` — package-specific noise
+- Right level: `(CALL, ERROR, ERROR)` — cross-package convention
+- Too abstract: `(X, Y, Y)` — trivial, tells the LLM nothing
+
+The middle ground depends on the category vocabulary. We need enough categories
+to be informative (CALL, ERROR, LIT, VAR, TYPE, etc.) but not so many that
+every sequence is unique.

experiments/RESULTS.md

@@ -102,18 +102,63 @@
 - Flask has more unique methods per package — less repetition
 - Flask SOREs are shorter/simpler — less compressible patterns
 
-## Next Steps
+## Round 5: Cross-Package Exact Matches
 
-1. **Accept the ceiling**: ~10-12% coverage may be the maximum for prefix-based grouping
-2. **Symbol coarsening**: Collapse specific symbols into categories (GETTER, SETTER, etc.)
-3. **Hierarchical grouping**: Group by k=1 first, then sub-group by k=2 within each group
-4. **GBNF output**: Convert SOREs to GBNF format for llama.cpp constrained decoding
-5. **Test on kotlinx.coroutines**: Third codebase for cross-validation
+**Hypothesis:** Some call sequences appear verbatim in multiple packages.
+These are the real cross-package conventions.
+
+**Method:** Group all sequences by exact tuple match, count packages per sequence.
+
+**Result:** 38 exact cross-package sequences in RAGSAK. Most trivial:
+- `('clearAllMocks',)` — 4 packages (test teardown)
+- `('Builder',)` — 4 packages (builder pattern)
+- `('Any',)` — 4 packages (Kotlin type)
+
+Interesting ones:
+- `('assumeTrue', 'isDockerAvailable', 'start', 'pullAndWarmup')` — 4 pkgs (Docker setup)
+- `('isNullOrBlank', 'error', 'error')` — 3 pkgs (null check → error)
+- `('sortedBy', 'map', 'toDescriptor')` — 3 pkgs (data pipeline)
+- `('ObjectMapper', 'findAndRegisterModules')` — 2 pkgs (Jackson config)
+
+**Verdict:** Exact matches too rare and mostly trivial. The real cross-package
+patterns are structural, not textual — "null check → error" appears with
+different method names in different packages.
+
+## Summary of Failed/Dismissed Approaches
+
+| Approach | Why it failed |
+|----------|--------------|
+| Per-package inference | Too sparse (1-3 seqs/package) |
+| Reduce algorithm | Wrong abstraction level — merges states within one automaton, not across packages |
+| Cross-package grouping by first symbol | 4.6% / 1.4% coverage — most groups skipped |
+| Exact cross-package matches | 38 sequences, mostly trivial single-call |
+| MDL vs Language Size scoring | Scoring isn't the bottleneck — pattern extraction is |
+
+## What Actually Works
+
+- **Behavioral grouping (first 3 symbols)** — 12% / 10.7% coverage, consistent across codebases
+- **Calling context as prefix** — the right signal, but package-specific
+- **GBNF conversion** — correct implementation, but input patterns too specific
+
+## Next: Structural Coarsening + Cross-Package Detection
+
+See `EXPERIMENT_LOG.md` for full reasoning and experiment design.
+
+Core idea: collapse method names → categories using tree-sitter capture names.
+Converts textual sequences into structural shapes that repeat across packages.
+
+```
+('isNullOrBlank', 'error', 'error')     →  (CALL, ERROR, ERROR)
+('raise', 'ValueError', 'ValueError')   →  (CALL, ERROR, ERROR)
+```
 
 ## Files Generated
 
 - `experiments/results/ragsak_summary.json` — RAGSAK metrics
 - `experiments/results/flask_summary.json` — Flask metrics
 - `experiments/context_eval.py` — experiment runner (supports multiple codebases)
+- `experiments/EXPERIMENT_LOG.md` — full experiment history and next steps
 - `bex/reduce.py` — Algorithm 4 (TODS 2010) implementation
+- `bex/gbnf.py` — SORE → GBNF converter
 - `tests/test_reduce.py` — 24 tests for Reduce
+- `tests/test_gbnf.py` — 15 tests for GBNF converter