docs: experiment log + cross-package analysis
- Full experiment history: context strategies, Reduce, scoring, GBNF, cross-package - Documented failures: per-package sparsity, Reduce at wrong level, exact match rarity - Designed next experiment: structural coarsening via tree-sitter categories - Updated RESULTS.md with Round 5 findings and summary table
This commit is contained in:
tobjend
parent
011df391c2
commit
8e1c7f3767
2 changed files with 281 additions and 6 deletions
230
experiments/EXPERIMENT_LOG.md
Normal file
230
experiments/EXPERIMENT_LOG.md
Normal file
|
|
@ -0,0 +1,230 @@
|
|||
# Experiment Log — Grammar Inference Pipeline
|
||||
|
||||
Track what we tried, what worked, what failed, and what's next. Each experiment
|
||||
includes: hypothesis, method, result, verdict.
|
||||
|
||||
---
|
||||
|
||||
## Round 1: Context Strategies (commit `bbdfe93`)
|
||||
|
||||
**Hypothesis:** The calling context (prefix before the method body) determines
|
||||
which methods share a convention. Better context grouping → better grammars.
|
||||
|
||||
**Method:** Tested 4 strategies on RAGSAK (Kotlin) and Flask (Python):
|
||||
- **Baseline:** Group by package directory
|
||||
- **Option A:** Group by last k components of file path (`file_path_k{k}`)
|
||||
- **Option B:** Group by first k symbols of call sequence (`first_k_sym_{k}`)
|
||||
- **Option C:** Two-dimensional: (path_k, first_k_symbols)
|
||||
|
||||
**Result:**
|
||||
| Strategy | RAGSAK patterns | RAGSAK coverage | Flask patterns | Flask coverage |
|
||||
|----------|----------------|-----------------|----------------|---------------|
|
||||
| Package baseline | 73 | 12.0% | 21 | 10.7% |
|
||||
| File path k=1 | 73 | 12.0% | 21 | 10.7% |
|
||||
| First k=1 | 20 | 4.6% | 2 | 1.4% |
|
||||
| First k=3 | 47 | 12.0% | 21 | 10.7% |
|
||||
|
||||
**Verdict:** Package grouping and first_k_sym_3 produce similar results.
|
||||
Cross-package grouping by first symbol is too sparse — most groups are either
|
||||
too large (skipped) or too diverse (skipped). The useful patterns are
|
||||
package-specific, not cross-package.
|
||||
|
||||
---
|
||||
|
||||
## Round 2: Reduce Algorithm (commit `b516b29`)
|
||||
|
||||
**Hypothesis:** Reduce (Algorithm 4, TODS 2010) merges structurally similar
|
||||
contexts, revealing cross-package patterns by unifying equivalent states.
|
||||
|
||||
**Method:** Implemented faithful Reduce with support-weighted SOA edit distance,
|
||||
adjunction, iterative merging, and minimize. Tested at ε=0.05 to 0.4.
|
||||
|
||||
**Result:**
|
||||
- RAGSAK at ε=0.3: 1 merge (`JobStatus.every.getJobStatus` ↔ `JobStatus.now.minusMinutes`)
|
||||
- Flask at ε=0.3: 1 merge (`def.boolean` ↔ `def.is_boolean`)
|
||||
- Coverage improvement: negligible (< 1%)
|
||||
|
||||
**Verdict:** Reduce doesn't help. The contexts we produce are already too
|
||||
specific (unique per package) for the distance metric to find meaningful merges.
|
||||
Reduce works when you have a large SOA with many equivalent states — we have
|
||||
one SOA per package with few states. Wrong abstraction level.
|
||||
|
||||
**Why it failed:** Reduce merges states in a single automaton. We're producing
|
||||
one automaton per package group. There's nothing to merge across packages
|
||||
because each package gets its own inference run. Reduce would need to operate
|
||||
on a cross-package SOA, which we don't build.
|
||||
|
||||
---
|
||||
|
||||
## Round 3: Language Size Scoring (commit `dfb56a0`)
|
||||
|
||||
**Hypothesis:** Bex et al.'s Language Size measure (arXiv:1004.2372, Section
|
||||
4.3.1) is a better scoring function than MDL for our use case.
|
||||
|
||||
**Method:** Implemented `lang_size_score()` as default scoring method. Added
|
||||
diversity threshold: skip groups with unique_ratio > 0.9 or methods < 5.
|
||||
|
||||
**Result:** 39 new tests. Pipeline runs correctly with new scoring. Coverage
|
||||
numbers similar to before — scoring method doesn't change which patterns are
|
||||
found, just which grammar is selected per group.
|
||||
|
||||
**Verdict:** Scoring method is not the bottleneck. The problem is upstream
|
||||
(pattern extraction), not downstream (pattern selection).
|
||||
|
||||
---
|
||||
|
||||
## Round 4: GBNF Output (commit `011df39`)
|
||||
|
||||
**Hypothesis:** SORE → GBNF conversion enables constrained LLM generation.
|
||||
SORE operators map directly to GBNF syntax.
|
||||
|
||||
**Method:** Implemented recursive descent parser for SORE, AST intermediate
|
||||
representation, and GBNF renderer. 15 tests.
|
||||
|
||||
**Result:** All tests pass. `to_gbnf('raise.(ValueError)+')` →
|
||||
`"raise" "ValueError"+`. Correct mapping of +, ?, *, |, ., parens.
|
||||
|
||||
**Verdict:** Implementation works. But the input SOREs are too specific
|
||||
to individual packages to be useful for constrained generation. The converter
|
||||
is correct; the patterns it converts are the problem.
|
||||
|
||||
---
|
||||
|
||||
## Round 5: Cross-Package Exact Matches
|
||||
|
||||
**Hypothesis:** Some call sequences appear verbatim in multiple packages.
|
||||
These are the real cross-package conventions.
|
||||
|
||||
**Method:** Grouped all sequences by exact tuple match across packages.
|
||||
|
||||
**Result:** 38 exact cross-package sequences in RAGSAK. Most are trivial:
|
||||
- `('clearAllMocks',)` — 4 packages (test teardown)
|
||||
- `('Builder',)` — 4 packages (builder pattern)
|
||||
- `('Any',)` — 4 packages (Kotlin type)
|
||||
- `('get',)` — 4 packages (getter)
|
||||
|
||||
Interesting ones:
|
||||
- `('assumeTrue', 'isDockerAvailable', 'start', 'pullAndWarmup')` — 4 packages (Docker test setup)
|
||||
- `('isNullOrBlank', 'error', 'error')` — 3 packages (null check → error)
|
||||
- `('sortedBy', 'map', 'toDescriptor')` — 3 packages (data pipeline)
|
||||
- `('ObjectMapper', 'findAndRegisterModules')` — 2 packages (Jackson config)
|
||||
|
||||
**Verdict:** Exact matches are too rare and mostly trivial. The real
|
||||
cross-package patterns are structural, not textual — "null check → error"
|
||||
appears with different method names in different packages.
|
||||
|
||||
---
|
||||
|
||||
## What We Learned (Summary)
|
||||
|
||||
1. **Per-package grouping is too sparse.** 1-3 sequences per package isn't
|
||||
enough for any inference method to produce general patterns.
|
||||
|
||||
2. **Cross-package exact matches are rare.** Only 38 in RAGSAK, mostly trivial
|
||||
single-call sequences.
|
||||
|
||||
3. **Reduce doesn't help at our abstraction level.** It merges states within
|
||||
one automaton; we need to merge patterns across packages.
|
||||
|
||||
4. **Scoring/selection isn't the bottleneck.** MDL vs Language Size doesn't
|
||||
change what patterns are found.
|
||||
|
||||
5. **The calling context prefix is the right signal** but grouping by it
|
||||
produces groups that are either too large, too diverse, or trivial.
|
||||
|
||||
6. **GBNF converter works correctly** but the input patterns are too specific.
|
||||
|
||||
---
|
||||
|
||||
## Next: Structural Coarsening + Cross-Package Detection
|
||||
|
||||
### Idea
|
||||
|
||||
Collapse method names → categories using tree-sitter capture names. This
|
||||
converts textual sequences into structural shapes:
|
||||
|
||||
```
|
||||
('isNullOrBlank', 'error', 'error') → (CALL, ERROR, ERROR)
|
||||
('raise', 'ValueError', 'ValueError') → (CALL, ERROR, ERROR)
|
||||
```
|
||||
|
||||
Same structural shape, different packages → cross-package convention.
|
||||
|
||||
### Why This Might Work
|
||||
|
||||
- We already extract tree-sitter capture names in `code.py:56-69`
|
||||
- We already classify nodes into categories in `code.py:72-100`
|
||||
(`lit`, `call`, `var`, `lambda`, `kwarg`, `expr`, `template`, `other`)
|
||||
- The behavioral prefix filter (`CALL_PREFIXES`) keeps raw text; we need a
|
||||
parallel path that keeps the category instead
|
||||
- Coarsened sequences have smaller alphabets → more methods per group →
|
||||
better inference
|
||||
- Patterns like `(CALL, ERROR, ERROR)` are meaningful conventions that
|
||||
repeat across packages
|
||||
|
||||
### What We Need
|
||||
|
||||
1. **Coarsening map:** `capture_name → category` using the existing
|
||||
`CALL_PREFIXES`, `ARG_LITERAL_TYPES`, `LAMBDA_TYPES` classifications
|
||||
plus a new `ERROR_TYPES` set
|
||||
|
||||
2. **Coarsened sequence extraction:** Same pipeline as now, but output
|
||||
category labels instead of method names
|
||||
|
||||
3. **Cross-package grouping:** Group by coarsened shape (first k categories),
|
||||
find shapes that appear in ≥2 packages
|
||||
|
||||
4. **SORE inference on coarsened sequences:** Smaller alphabet, more examples
|
||||
per group → better patterns
|
||||
|
||||
5. **Evaluation:** Compare coarsened patterns vs raw patterns on:
|
||||
- Coverage (% of methods in learned groups)
|
||||
- Cross-package reach (# of packages per pattern)
|
||||
- Usefulness for constrained generation (GBNF quality)
|
||||
|
||||
### Open Questions
|
||||
|
||||
- Does coarsening lose too much specificity? `(CALL, ERROR, ERROR)` is less
|
||||
informative than `(raise, ValueError, ValueError)` — is the tradeoff worth it?
|
||||
- What categories to use? The existing classifications in `code.py` are a
|
||||
starting point but may need refinement (e.g., separating ERROR from CALL)
|
||||
- How to handle the long tail? Most sequences are 1-2 symbols — coarsening
|
||||
doesn't help much for those
|
||||
- Is the GBNF output useful at all? Maybe the output should be a conditional
|
||||
frequency table instead of a grammar
|
||||
|
||||
### Experiment Design
|
||||
|
||||
**Phase 1: Coarsening Proof of Concept**
|
||||
- Implement coarsening map in `code.py`
|
||||
- Add `--coarsen` flag to CLI
|
||||
- Run on RAGSAK + Flask, compare raw vs coarsened patterns
|
||||
- Measure: alphabet size reduction, group size increase, pattern count
|
||||
|
||||
**Phase 2: Cross-Package Detection**
|
||||
- Group coarsened sequences by shape (first k categories)
|
||||
- Find shapes appearing in ≥2 packages
|
||||
- For each shape, infer SORE on coarsened sequences
|
||||
- Measure: cross-package patterns found, coverage improvement
|
||||
|
||||
**Phase 3: Output Quality**
|
||||
- Convert coarsened SOREs to GBNF
|
||||
- Evaluate: are the GBNF rules more general/useful than raw SOREs?
|
||||
- Compare: coarsened GBNF vs raw GBNF vs conditional frequency table
|
||||
|
||||
---
|
||||
|
||||
## The "Redacted" Concept
|
||||
|
||||
When collapsing method names to categories, we lose the specific method name
|
||||
but gain the structural pattern. This is a form of **abstraction** — moving
|
||||
from concrete examples to general rules.
|
||||
|
||||
The question is whether the abstraction is at the right level:
|
||||
- Too specific: `(raise, ValueError, ValueError)` — package-specific noise
|
||||
- Right level: `(CALL, ERROR, ERROR)` — cross-package convention
|
||||
- Too abstract: `(X, Y, Y)` — trivial, tells the LLM nothing
|
||||
|
||||
The middle ground depends on the category vocabulary. We need enough categories
|
||||
to be informative (CALL, ERROR, LIT, VAR, TYPE, etc.) but not so many that
|
||||
every sequence is unique.
|
||||
|
|
@ -102,18 +102,63 @@
|
|||
- Flask has more unique methods per package — less repetition
|
||||
- Flask SOREs are shorter/simpler — less compressible patterns
|
||||
|
||||
## Next Steps
|
||||
## Round 5: Cross-Package Exact Matches
|
||||
|
||||
1. **Accept the ceiling**: ~10-12% coverage may be the maximum for prefix-based grouping
|
||||
2. **Symbol coarsening**: Collapse specific symbols into categories (GETTER, SETTER, etc.)
|
||||
3. **Hierarchical grouping**: Group by k=1 first, then sub-group by k=2 within each group
|
||||
4. **GBNF output**: Convert SOREs to GBNF format for llama.cpp constrained decoding
|
||||
5. **Test on kotlinx.coroutines**: Third codebase for cross-validation
|
||||
**Hypothesis:** Some call sequences appear verbatim in multiple packages.
|
||||
These are the real cross-package conventions.
|
||||
|
||||
**Method:** Group all sequences by exact tuple match, count packages per sequence.
|
||||
|
||||
**Result:** 38 exact cross-package sequences in RAGSAK. Most trivial:
|
||||
- `('clearAllMocks',)` — 4 packages (test teardown)
|
||||
- `('Builder',)` — 4 packages (builder pattern)
|
||||
- `('Any',)` — 4 packages (Kotlin type)
|
||||
|
||||
Interesting ones:
|
||||
- `('assumeTrue', 'isDockerAvailable', 'start', 'pullAndWarmup')` — 4 pkgs (Docker setup)
|
||||
- `('isNullOrBlank', 'error', 'error')` — 3 pkgs (null check → error)
|
||||
- `('sortedBy', 'map', 'toDescriptor')` — 3 pkgs (data pipeline)
|
||||
- `('ObjectMapper', 'findAndRegisterModules')` — 2 pkgs (Jackson config)
|
||||
|
||||
**Verdict:** Exact matches too rare and mostly trivial. The real cross-package
|
||||
patterns are structural, not textual — "null check → error" appears with
|
||||
different method names in different packages.
|
||||
|
||||
## Summary of Failed/Dismissed Approaches
|
||||
|
||||
| Approach | Why it failed |
|
||||
|----------|--------------|
|
||||
| Per-package inference | Too sparse (1-3 seqs/package) |
|
||||
| Reduce algorithm | Wrong abstraction level — merges states within one automaton, not across packages |
|
||||
| Cross-package grouping by first symbol | 4.6% / 1.4% coverage — most groups skipped |
|
||||
| Exact cross-package matches | 38 sequences, mostly trivial single-call |
|
||||
| MDL vs Language Size scoring | Scoring isn't the bottleneck — pattern extraction is |
|
||||
|
||||
## What Actually Works
|
||||
|
||||
- **Behavioral grouping (first 3 symbols)** — 12% / 10.7% coverage, consistent across codebases
|
||||
- **Calling context as prefix** — the right signal, but package-specific
|
||||
- **GBNF conversion** — correct implementation, but input patterns too specific
|
||||
|
||||
## Next: Structural Coarsening + Cross-Package Detection
|
||||
|
||||
See `EXPERIMENT_LOG.md` for full reasoning and experiment design.
|
||||
|
||||
Core idea: collapse method names → categories using tree-sitter capture names.
|
||||
Converts textual sequences into structural shapes that repeat across packages.
|
||||
|
||||
```
|
||||
('isNullOrBlank', 'error', 'error') → (CALL, ERROR, ERROR)
|
||||
('raise', 'ValueError', 'ValueError') → (CALL, ERROR, ERROR)
|
||||
```
|
||||
|
||||
## Files Generated
|
||||
|
||||
- `experiments/results/ragsak_summary.json` — RAGSAK metrics
|
||||
- `experiments/results/flask_summary.json` — Flask metrics
|
||||
- `experiments/context_eval.py` — experiment runner (supports multiple codebases)
|
||||
- `experiments/EXPERIMENT_LOG.md` — full experiment history and next steps
|
||||
- `bex/reduce.py` — Algorithm 4 (TODS 2010) implementation
|
||||
- `bex/gbnf.py` — SORE → GBNF converter
|
||||
- `tests/test_reduce.py` — 24 tests for Reduce
|
||||
- `tests/test_gbnf.py` — 15 tests for GBNF converter
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue