feature/treesitter-tag-queries #2

Open

tobi wants to merge 78 commits from feature/treesitter-tag-queries into main

pull from: feature/treesitter-tag-queries

merge into: tobi:main

tobi:main

tobi:feature/kotlin-specific-extras

tobi:feature/kore-inference

Conversation 0 Commits 78 Files changed 246 +324585 -1472

tobi commented 2026-07-03 21:00:42 +00:00

Owner

Copy link

No description provided.

tobi added 12 commits 2026-07-03 21:00:42 +00:00

feat: inspect TreeSitter tag queries for Dervish universal AST extraction ... 069c63f2c8

- Fetch TAGS_QUERY and HIGHLIGHTS_QUERY from 8 official TreeSitter language packages
- Save all curated queries as .scm files
- Analysis confirms universal query approach is viable
- TreeSitter silently ignores non-existent node types per language
- Add full conversation references (gemini chat markdown files)
- Add comprehensive conversation summary with roadmap

chore: add downloaded community queries for kotlin + typescript ... 8d06ac2d52

Kotlin (fwcd/tree-sitter-kotlin) and TypeScript packages
don't bundle queries. These are downloaded from upstream repos.

feat: add community tags findings audit and kotlin tags query 986eaa7f83

feat: universal AST preprocessor with nvim-treesitter queries ... ee5ebc9eb4

- Replace old per-language query files with unified tag-preprocessor
- Universal preprocessor uses nvim-treesitter highlights.scm for 10 langs
- Resolves ; inherits: directives (ecma, jsx) for JS/TS
- ts-kotlin bundled query for Kotlin compatibility
- Strips unsupported #set! 3-arg predicates for jsx
- archive old community queries

chore: add Gemini session chat logs as reference 70300ff917

docs: document Kotlin and JS/TS special cases in analysis 8bda174293

feat: universal tag-preprocessor orchestrator with frequency filter ... 0c7703f63b

- Rename bex/tag-preprocessor/ to bex/tag_preprocessor/ (valid Python package)
- Add analyze.py: scan_directory, frequency_filter, infer, analyze_directory
- Frequency filter removes symbols below min_coverage threshold (Phase 1.0 step 4)
- infer() wires preprocess → frequency_filter → ensemble (Phase 1.0 step 5)
- --dir CLI mode for running full pipeline on directories (Phase 1.0 step 6)
- 9 new tests (test_analyze.py), all 94 tests pass

feat: method-level sequence extraction via child_by_field_name('body') ... 2620b6e49f

- _find_method_bodies uses tree-sitter's universal body field (9/10 grammars)
- Kotlin fallback: scan children for body-like types
- preprocess_by_method groups highlight captures by enclosing method body
- Returns per-method sequences for k-ORE ordering analysis
- analyze.py infer() now uses preprocess_by_method
- Document findings in ANALYSIS.md
- 97 tests pass

feat: cluster methods by n-gram patterns before inference ... 73b94af959

- _extract_call_tokens filters to call-like captures only
- cluster_methods groups sequences by shared 3-gram call patterns
- analyze_clusters runs per-cluster CRX/iDRegEx/kORE inference
- Lists 17 convention clusters for RAGSAK test code
- iDRegEx and kOREInference now produce ordered grammars per cluster
- (other) cluster captures diverse conventions as CRX vocabulary
- min-cluster-size (default 3) and ngram-size (default 3) CLI flags

feat: language-agnostic import/arg/package extraction ... fd574da53d

- Add _classify_arg_node, _find_arglist_node, _iterate_arg_nodes,
  extract_arg_info, _summarize_arg_info (no per-language branches)
- _find_arglist_node: drop call_suffix from fallback list
- _iterate_arg_nodes: generic named-child iterator only
- Add IMPORT_PATTERNS + _extract_imports (regex-based, all languages)
- Add _file_to_package / _top_packages (pure relpath from project root)
- Add --json output format for prompt injection
- analyze_clusters returns 4-tuple with meta (files, imports, arg_patterns, packages)

feat: adaptive multi-assignment clustering; add ADRs 1-10 ... e23922a1b7

- Multi-assignment clustering (no greedy 'used' set)
- Adaptive ngram fallback (shrink when (other) > 60%)
- Add docs/adr/ with 10 architecture decision records
- Fix ADR 1 (query modification description)
- Fix ADR 3 (multi-assignment + adaptive shrink)
- Fix ADR 5 (import sort order clarification)
- Fix ADR 6 (remove Kotlin call_suffix references)
- New ADR 9 (adaptive clustering rationale)
- New ADR 10 (universal package mapping via relpath)

docs: pipeline overview ASCII diagram 55d21af48e

tobi added 8 commits 2026-07-11 18:51:19 +00:00

fix: restore frequency_filter at 0.2, add dual 0.8 BEX coverage ... eb2173442b

- frequency_filter(min_coverage=0.2) strips rare symbols before clustering/inference
- DEFAULT_COVERAGE=0.8 passed to infer_ensemble for sequence-level core/outlier detection
- Both filters active: symbol-level (0.2) then BEX sequence-level (0.8)
- Fix seq_of_file tracking broken by filter creating new list objects

fix: add bare function fallback to CALL_PREFIXES, fix IMPORT_PATTERNS order, update pipeline diagram ... 4a506cd39b

- CALL_PREFIXES: add bare "function" as last fallback — Kotlin uses
  @function for both calls and definitions (no function.call capture)
- _find_arglist_node: remove template_string from arglist detection
- IMPORT_PATTERNS: require_relative before require (Ruby fix)
- pipeline-overview.txt: remove .gitignore, add min_coverage=0.8 + core/outlier
- DEFAULT_COVERAGE=0.8, frequency_filter at fixed 0.2

feat: add --slice package (per-directory grouping), --include/--exclude, --verbose, drop root group inference, run full ensemble on all groups e3ad256321

drop kORE from default ensemble, add --kore flag to opt in ... 710da56916

- Remove kORE from top-level imports, _ALGORITHMS, and infer_ensemble body
- Add include_kore=False parameter, runs kORE only when opted in
- Add --kore CLI flag threaded through all pipeline layers
- Remove koreinference from --prefer choices
- Keep kore.py module in repo for reference/future use
- Update tests: remove test_prefer_koreinference, update algorithm assertions

parallelize preprocessing across files via ProcessPoolExecutor ... 5906adfc95

Replace serial file-read + tree-sitter parse loop with _preprocess_files()
using ProcessPoolExecutor. Module-level _preprocess_file() for pickling.

Covers all three callers: analyze_clusters, analyze_by_package, infer.

parallelize kORE outer (k, n) trials via ProcessPoolExecutor ... 0b5b0e623b

Add _kore_trial() module-level worker for pickling. infer() accepts
n_workers param — >1 runs all (k x N) trials concurrently instead
of serial. Default 1 preserves existing behavior.

Thread n_workers through _run_kore() and infer_ensemble() to support
--kore flag with parallel kORE inference.

Revert "parallelize kORE outer (k, n) trials via ProcessPoolExecutor" ... ce6521ad5e

This reverts commit 0b5b0e623b.

docs: update stale docs — remove kORE from default ensemble, add tag preprocessor CLI e94c52b71a

tobi added 1 commit 2026-07-11 19:28:42 +00:00

feat: add analyze_directory MCP tool — scan codebase, infer conventions, persist to .dervish/ 830104b399

tobi added 53 commits 2026-07-13 00:09:57 +00:00

WIP: language size scoring + diversity threshold (step 1 pending) dfb56a083a

feat: add diversity threshold — skip groups with unique_ratio > 0.9 or methods < min_methods ... ab20b03256

Adds --min-methods CLI flag (default 5). Groups that are too diverse or
too small are skipped with skip_reason in meta. Prevents noisy/meaningless
grammars from diverse packages.

experiments: test all context strategies on RAGSAK — behavioral grouping wins ... bbdfe93679

Key findings:
- File path grouping (structural) = baseline (0.6% coverage) — no improvement
- First-k-symbols grouping (behavioral) = 20× improvement (12% coverage at k=3)
- Two-dimensional (path + symbols) = worse than behavioral alone
- Winner: Option B k=3 — 47 SOREs, 12% coverage

Smart filters make experiments fast (<1s vs minutes):
- max_unique_ratio=0.85
- max_alphabet=20
- max_soa_edges=100

Preserved in experiments/results/ for future reference.

feat: implement Reduce algorithm (Algorithm 4, TODS 2010) ... b516b2985d

- bex/reduce.py: Faithful implementation of Reduce with support-weighted
  SOA edit distance, adjunction, iterative merging, and Minimize
- experiments/context_eval.py: Multi-codebase support (RAGSAK + Flask),
  Reduce experiments with thresholds 0.05-0.4
- tests/test_reduce.py: 24 tests covering all Reduce components
- Flask cloned to external_refs/flask for cross-validation

Results:
- RAGSAK: 12.0% coverage (First 3 symbols)
- Flask: 10.7% coverage (First 3 symbols)
- Reduce has minimal impact (1-2 merges per codebase at ε=0.3)
- Coverage ceiling appears to be ~10-12% for prefix-based grouping

feat: implement SORE → GBNF converter ... 011df391c2

- Recursive descent parser for SORE syntax (+, ?, *, |, ., parens)
- AST intermediate representation (_Literal, _Concat, _Alt, _Plus, _Optional, _Star)
- to_gbnf(sore) → full GBNF rule string
- to_gbnf_with_rules(sore, name) → named rule for composition
- 15 tests covering all SORE operators and nesting patterns

docs: experiment log + cross-package analysis ... 8e1c7f3767

- Full experiment history: context strategies, Reduce, scoring, GBNF, cross-package
- Documented failures: per-package sparsity, Reduce at wrong level, exact match rarity
- Designed next experiment: structural coarsening via tree-sitter categories
- Updated RESULTS.md with Round 5 findings and summary table

feat: structural coarsening experiment — keeps function names raw, coarsens keywords ... 592974f039

- coarsen_token() maps tree-sitter captures to categories (RETURN, IF, LOOP, etc.)
- Function calls kept as raw text (they ARE the behavioral content)
- Only ~5% structural tokens coarsened
- Results: Flask coverage 17.5% -> 28.8% (+11.3%), RAGSAK unchanged (95% calls)
- Cross-package shapes emerge: ('IF', 'KW', 'RETURN') in 4 Flask packages

fix: bare Kotlin captures + minimal coarsening (RETURN/IF/EXCEPTION/LOOP only) ... d65b78bdf1

- Added bare Kotlin captures to BEHAVIORAL_PREFIXES: conditional, exception, repeat, property, type
- Dropped variable (too noisy), kept only 4 high-signal categories in coarsen_token
- RAGSAK: 87 cross-package contexts (up from 69), real null-check patterns
- Flask: k=2 coverage 4.5% -> 15.8%, cross-package rendering conventions

docs: Round 6b results — Kotlin capture fix + minimal coarsening 8540947eec

feat: 4-codebase evaluation — conventions vs completions tradeoff ... dd183f6241

Added kotlinx.coroutines (1040 .kt files) and FastAPI (1129 .py files).

Key findings across 4 codebases:
- Coarsening trades per-package coverage for cross-package reach
- FastAPI: coverage IMPROVES (12.4% -> 20.7%), cross-pkg +15
- Coroutines: cross-pkg +15, real exception-handling patterns
- RAGSAK: cross-pkg +18, null-check conventions across 8 packages
- Flask: slight cross-pkg loss, rendering conventions still visible

The conventions vs completions tradeoff is real and measurable.

chore: add external_refs to .gitignore 1b98b1d24e

feat: frequency threshold sweep — 0.01-0.20 across 4 codebases ... 8028570ceb

Sweet spot: 0.01-0.05. At 0.01 RAGSAK gets 61 SOREs (26.1% cov) with
real conventions like warn.status.body.ErrorResponse. At 0.05 coverage
jumps to 45.4% but that pattern disappears. Flask dies at 0.15+.

Current default min_coverage=0.2 is too aggressive for most codebases.

feat: CRX refined — cluster-then-infer for tighter grammars ... 739000e8c6

Standard CRX over-approximates when Hasse diagram is non-linear (24% of
RAGSAK packages). Cluster-then-infer groups sequences by (first, last,
length), infers per-cluster, picks largest cluster's grammar.

Results on RAGSAK:
  Avg max disjunction: 2.8 → 1.7 (39% tighter)
  Packages improved: 6/10

Tradeoff: cluster granularity (too coarse = over-approximation,
too fine = no generalization). Current: (first, last, length_bucket).

Exports crx_refined() and crx_with_confidence() from bex package.
20 new tests. All 199 tests pass.

fix: wire crx_refined into pipeline, fix hardcoded min_coverage ... 6912841b9e

- Add --crx-method (standard|refined) to CLI and analyze_directory
- Pass crx_method through analyze_by_package → _infer_group
- Fix hardcoded min_coverage=0.2 in 3 places (was ignoring parameter)
- Refined CRX wraps crx_with_confidence into standard result format

fix(gbnf): handle disjunction inside parens and compound repetition ... f57c302c91

The GBNF parser now correctly handles SORE's overloaded + operator:
- + inside (a+b+c) → alternation (not repetition)
- + outside parens → repetition
- +? and +* compound operators → normalized to Star

Also adds implicit concatenation when LPAREN follows a repetition,
so a+(b+c) parses as a+ followed by (b|c).

28 tests pass (13 new disjunction/compound tests). Full suite: 212 passed.

perf: iDRegEx opt-in, GBNF newline fix, OverflowError fix ... bc7d3b6ca1

- iDRegEx now opt-in via --idregex flag (was running on every group,
  causing 55s+ on Flask alone — src/flask/json took 55s in iDRegEx)
- GBNF tokenizer strips newlines from literals (multi-line symbols)
- Fix OverflowError: lang_size_score produces huge ints for large
  disjunctions, format as string not float
- Flask: 2.7s (was 55s+), RAGSAK: 13s (was 74s)

fix: validate SORE before returning, skip malformed grammars ... 3468813ec8

- Add validate_sore() to gbnf.py — checks parseability without converting
- _infer_group now validates grammar and returns skip_reason='malformed_grammar'
  for SOREs containing raw code (e.g. w_body=, sult=, (+,+:N+...)
- Results: 130 grammars, 130 GBNF OK, 0 GBNF FAIL
  - Flask: 5 OK, 0 FAIL, 3.4s
  - RAGSAK: 19 OK, 0 FAIL, 11 malformed, 12.9s
  - FastAPI: 106 OK, 0 FAIL, 6 malformed, 30.5s

docs: Round 11 results — pipeline speed + GBNF conversion ... 92af932e9d

- iDRegEx opt-in: Flask 55s→2.8s, RAGSAK 74s→13s, FastAPI 30s→30s
- GBNF validation: 130/130 OK, 0 FAIL (17 malformed skipped)
- Grammar quality: 43% structured, rest flat/trivial
- Logged to experiments/EXPERIMENT_LOG.md

feat: grammar_structure_score + min_structure filter ... e62fffc6e0

Quantifies how structured a SORE is (0.0=flat bag, 1.0=fully structured).
Flat bags are CRX over-approximations — they list symbols without ordering.

With min_structure=0.2:
  Flask:   2 kept (was 5),  9 dropped
  RAGSAK: 10 kept (was 19), 114 dropped
  FastAPI: 47 kept (was 106), 95 dropped
  Total: 59 useful grammars, 218 noise removed

CLI: --min-structure 0.2 (default: 0, keep all)

docs: Round 12 results — grammar structure scoring 483526cb6a

fix: set DEFAULT_COVERAGE to 0.05 (experimentally determined sweet spot) ... 11364151a6

The default min_coverage of 0.8 was filtering out almost all symbols
before CRX could see them, making every other parameter tuning moot.
At 0.8, only symbols present in 80%+ of methods survive — which means
0-2 symbols per group. At 0.05, 46 symbols survive for fastapi/src
and CRX can actually find structure.

This was the root cause of why max_unique_symbols gate never triggered
and why CRX kept producing flat bags.

tune: min_methods 5→3, fix DEFAULT_COVERAGE help text ... 52c90bfdd8

min_methods=3 gains 9 extra FastAPI grammars (3-4 method groups with
clear sequential structure) without quality loss. No change for Flask
or RAGSAK (their small groups fail other gates anyway).

Multi-codebase validation (Round 13): tested Flask, FastAPI, httpx,
Pydantic, SQLAlchemy. Key finding: CRX works on codebases with small
focused sequential modules (FastAPI docs_src=22 grammars) but not on
large utility libraries (Flask=0, Pydantic=0, SQLAlchemy=1).

fix: sanitize symbol extraction, fix malformed grammars ... 17561a2f37

- Add sanitize_symbol() to extract clean identifiers from AST nodes
- Strip SORE special characters ((),+?*[]{}|\^$.) from symbols
- Extract just function name from multi-line expressions
- Skip single-char fragments (truncated identifiers)
- Filter empty symbols in preprocessing
- Wire iLocal context extraction into pipeline
- Result: RAGSAK malformed grammars 6→0, FastAPI clean
- 212 tests pass

feat: add reduce slicing, fix rwr0 timeout, fix sequence format ... 2733840358

- Add --slice reduce (Algorithm 4, TODS 2010) for merging similar directories
- Fix rwr0 infinite loop: add max_iterations=1000 limit
- Fix reduce.py: handle tuple sequences (capture_name, text, line)
- Fix minimize_contexts: skip rwr0 on complex SOAs (too slow)
- Result: Reduce finds 0 merges on RAGSAK (SOA distances too high)
- Reduce is not suitable for source code grouping (designed for XML schemas)

feat: split mixed-pattern groups before CRX inference ... ca328b5402

Add --split-mixed flag that detects groups with diverse first symbols
and splits them before CRX. This separates distinct calling patterns
that CRX would otherwise merge into flat bags.

Results:
- FastAPI: 22→33 grammars, avg score 0.47→0.57
- RAGSAK: 5→8 grammars, avg score 0.45→0.71
- dependency_testing: bag→clean 'return.q?.commons?.skip?.limit?' (1.00)

212 tests pass

refactor: recursive split-by-first-symbol (max_depth=3) ... ca8a13b549

Replace single-level split with recursive splitting that drills deeper
into mixed-pattern groups. Each leaf group produces its own grammar,
and the best leaf is returned per parent group.

Results:
  FastAPI:  22 → 32 grammars (+45%), 8 → 17 high-quality (>=0.5) (+113%)
  RAGSAK:   5 → 8 grammars (+60%), 2 → 6 high-quality (+200%)
  Flask:    0 → 2 grammars (was zero, now produces output)

The improvement comes from capturing grammars in groups that previously
couldn't produce one at all — sub-groups of 3-5 methods that are too
small for single-split but contain clear patterns (e.g., all 'return'
or all 'if' sequences).

docs: add Round 13 to experiment log (recursive split results) fe3261b9de

feat: grammar index for runtime lookup ... 7b54a5ef77

Add GrammarIndex class that loads grammars.yml and provides fast
lookup by (package, context_symbol). Enables agents to get the right
GBNF grammar at code generation time.

API:
  load_grammar_index(project_root) → GrammarIndex
  idx.get(file_path, context_symbol=None) → gbnf_string
  idx.get_package(file_path) → [(symbol, grammar, score, methods)]

14 tests, all passing.

feat: runtime grammar lookup via MCP + fix YAML output ... 44415c5b42

MCP server:
- analyze_directory: updated signature to match CLI (split_mixed,
  min_structure, min_methods, min_coverage=0.05)
- get_grammar(directory, file_path, context_symbol): returns the right
  GBNF for constrained generation at code generation time
- get_package_grammars(directory, file_path): lists all grammars for
  a file's package, ranked by quality

YAML output:
- _build_yaml_output now includes leaf grammars from recursive split
  (all_grammars in meta), so each calling context gets its own entry

Workflow for agents:
  1. analyze_directory → grammars.yml persisted
  2. get_grammar(file, 'return') → GBNF for constrained generation
  3. LLM generates code following the package's convention

wip: call_only filter for JS/TS + checkpoint before GBNF rewrite 617b7ad578

fix: byte/char offset mismatch in tree-sitter text extraction ... 93d53f164a

Root cause: parser.parse(code.encode()) returns byte offsets, but
code[node.start_byte:node.end_byte] indexed into a Python string
(character offsets). Non-ASCII chars caused cumulative drift → truncated symbols.

Fix: store code_bytes = code.encode(), index into that, decode only final text.

Impact:
- Zod: 653 truncated symbols → 0
- RAGSAK: 653 truncated symbols → 0
- RAGSAK grammars: 8 → 27 (3.4×)
- FastAPI grammars: 33 → 111 (3.4×)
- Malformed: 6 → 2 (RAGSAK), 0 (FastAPI/Flask)

Also removed sanitize_symbol() (dead code after fix) and call_only parameter.

experiment: kORE/iDRegEx vs CRX on flat bags (Round 15) ... e9f672cba4

Key finding: iDRegEx/kORE return None on genuinely diverse groups (flat bags).
CRX is the only algorithm that produces anything for these groups.
For structured groups, CRX already captures ordering well.

Conclusion: kORE/iDRegEx are not better fallbacks for flat bags.
The pipeline's existing filtering (min_structure, split_mixed) is the right approach.

feat: iDRegEx refinement for CRX flat bags (Round 16) ... b92b7653e2

Heuristic: only run iDRegEx when n_methods ≤ 10 AND CRX grammar has
>50% top-level optional parts (flat chain signal). If iDRegEx grammar
is >10x tighter by lang_size, use it. Otherwise keep CRX.

RAGSAK result: agents/capability (5 methods) refined from
slot?.(defaultCapabilityId+summarize)?... (lang_size=1432) to
(defaultCapabilityId|summarize) (lang_size=3) — 477x tighter.

Speed cost: ~0.7s per candidate, negligible on 74s pipeline.
CLI: --idregex-refine flag (default off).

Also adds _count_optionals() and _should_try_idregex() helpers
with 8 pytest tests. 234 tests pass.

experiment: CRX vs refined CRX across 3 codebases (Round 17) ... 1d94c09344

Refined wins 7/9 when producing useful grammars (model_cost >= 2).
Trivial output (single symbol) in 5/13 cases on large groups.
CRX wins only once (FastAPI tests, 3618 methods).

Key finding: refined CRX is better ~78% of the time but needs
triviality check (model_cost >= 2) to avoid single-symbol grammars.

docs: ASCII diagrams for pipeline, parameters, and decision matrix b2c1263838

docs: research positioning and decision matrix ... 242f00a0da

- RESEARCH_POSITIONING.md: where we fit in the science
- DECISION_MATRIX.md: when to use CRX vs refined CRX
- Updated AGENTS.md with research context and updated CLI flags

docs: achievement summary and experiment outputs (Round 14-15) ... 2da3a7161f

- ACHIEVEMENT_SUMMARY.md: what we built and learned
- Round 14: Full pipeline run on 4 codebases (clean symbols)
- Round 15: kORE/iDRegEx vs CRX comparison

Next: Crucio-inspired clustering for better grouping

feat: distributional clustering for better grouping (Phase 1) ... 841f5efcf5

Inspired by Crucio's distributional matrix (ICSE 2026).
Groups symbols by context similarity instead of first-symbol.

New module: bex/distributional.py
- extract_contexts(): get (left, right) context pairs
- build_distribution_matrix(): symbol × context counts
- cluster_symbols(): agglomerative clustering by similarity
- split_by_cluster(): split sequences by cluster membership
- distributional_split(): drop-in replacement for _split_by_first_symbol

CLI: --cluster-method distributional (opt-in, first-symbol is default)

Example: 'if' and 'while' both appear before 'return'
→ first-symbol: 3 groups (if, while, return)
→ distributional: 2 groups (if/while merged, return)

23 tests pass. Full suite: 257 tests pass.

Phase 2: decomposition forest for complex sequences ... 8b2899d16e

Decomposition breaks long sequences into shorter fragments before
inference. This helps when sequences are too long for CRX to handle
(>5 symbols → flat bags).

Results:
- RAGSAK: 21 → 80 grammars (3.8× increase)
- FastAPI: 111 → 118 grammars (small increase)

Changes:
- bex/decompose.py: decompose_sequence(), decompose_all(), decompose_with_coverage()
- bex/tag_preprocessor/analyze.py: --decompose, --max-seq-length flags
- Skip diversity check when decomposing (decomposition creates diverse fragments)
- 12 new tests in tests/test_decompose.py

Co-authored-by: OpenCode <opencode@corentic.eu>

MCP server: add decomposition + quality filtering params ... 9ca56e2c69

- analyze_directory tool: add decompose, max_seq_length, cluster_method,
  crx_method params for Crucio-inspired improvements
- min_structure default bumped to 0.5 for MCP (only high-structure
  grammars returned to agents)
- _build_yaml_output: filter entries below min_structure threshold
- README: update source code analysis section, fix kORE/iDRegEx mentions
- 269 tests pass

Co-authored-by: OpenCode <opencode@corentic.eu>

fix: YAML expansion bug and add GBNF to output ... ee60b62854

- Removed all_grammars expansion in _build_yaml_output (226→54 RAGSAK, 384→87 FastAPI)
- Added gbnf field to each YAML entry via to_gbnf()
- Updated test_roundtrip_persisted_ragsak to use correct file path

Closes #66

docs: handover document for project continuity 44dd34a46f

docs: Round 19 evaluation plan + golden config ... 9d206f586f

- Plan: match rate audit, decomposition A/B, distributional A/B, fragment quality
- Golden config: best-known values for all heuristic parameters
- Single source of truth in bex/golden_config.py

wip: AST foundation — grammar.py, expr.py→AST, soa.py→AST labels ea6cac53e3

feat: BEX_VERBOSE env var + real-time parallel progress logging ... dac33db616

_vprint now honors the BEX_VERBOSE env var in addition to the CLI
--verbose flag, so non-CLI callers (MCP, dervish integration) and
worker processes (which start with a fresh module) emit logs too.
--verbose exports BEX_VERBOSE=1 so ProcessPoolExecutor workers inherit it.

analyze_by_package now logs real-time per-group completion with
[i/N] label (count methods) done (Xs), making it obvious the parallel
pipeline is progressing and not hung.

test: update tests for AST representation ... 19a1db48ef

Assertions now use isinstance() on AST nodes (Concat, Alt, Plus,
Optional, Star, Symbol, Empty) instead of comparing SORE strings.

chore: update experiment scripts for AST representation f8dda557d2

fix: restore memoization on _count_concat to stop count_words explosion ... 52f286a073

During the AST migration, _count_concat lost its @lru_cache (it was
replaced by an uncached recursive version). Distributing a length L
across concat parts then revisited the same (remaining_parts, length)
states exponentially -> 16M function calls for a single 4-method group,
and several RAGSAK groups took 18-52s (looked hung at the tail).

Restore memoization: _count_concat is now @lru_cache-keyed on
(tuple(parts), length). Also fix a stray blank line between the
@lru_cache decorator and count_words.

Impact (RAGSAK, --slice package --min-structure 0.5):
  straggler groups 18-52s -> <0.5s; full run 54.9s -> 6.7s.
Adds a regression test asserting a length-20 (a|b)* string counts in <2s.

experiments: round 20 AST-migration verification across 3 languages ... 1810a695d0

Run RAGSAK (Kotlin), fastapi (Python), zod (TypeScript) with --slice
package --min-structure 0.5, verbose+timeout, results saved for later.

After _count_concat memoization fix: RAGSAK full run 54.9s -> 6.7s,
no tail stalls. RAGSAK: 121 packages (28 with grammar). fastapi/zod
complete in ~25s (dominated by arg-extraction on 2000+ method test
groups, not grammar explosion). Includes SUMMARY.md with before/after.

feat: default decomposition ON + restore Language Size scoring discrimination ... 8f412905c5

B (decompose default):
- analyze_directory + CLI now default decompose=True (--no-decompose to
  disable); default max_seq_length 5 -> 4 (matches golden_config).
- Decomposing long sequences into <=4-symbol fragments yields tighter
  grammars and disables the too_diverse skip, so far more packages get a
  grammar. RAGSAK: 29 -> 95 grammars, pure bags 9 -> 5; fastapi 74 -> 26
  pure bags; zod 15 -> 5. All runs exit 0, no stalls.

A (repair Language Size scorer, not a new one):
- _COUNT_CAP was applied to count_words' RETURN value, silently clamping
  lang_size_score / model_cost / data_cost at 10^12 for every real codebase
  grammar. That broke ADR-13 (Language Size picks most-specific grammar):
  bags and tight grammars tied at 10^12, so the scorer could not prefer
  specific grammars over generic ones.
- _COUNT_CAP raised to 10**30. Memoization (_count_concat, earlier commit)
  already prevents the recursion hang the cap was guarding against, so the
  cap no longer needs to clamp scores. lang_size_score now discriminates
  (verified: tight=20 vs bag=9975).

experiments: round 20 v3 results (decompose default + cap fix) ... 8460c3c862

v3 runs for RAGSAK/fastapi/zod with decompose default-on and _COUNT_CAP
raised. Updated SUMMARY.md with before/after quality table.

feat: enable idregex_refine by default + AST rewrite of _count_optionals ... 8b3a454a15

- idregex_refine=True default across all entry points + CLI
- Added --no-idregex-refine flag to CLI
- Rewrote _count_optionals from SORE string parser to AST walker
- Added _is_pure_bag() helper for bag detection
- Updated _should_try_idregex() to trigger on pure Plus(Alt) bags
- Updated tests for AST-based _count_optionals
- Added RAGSAK v4 experiment results

docs: update experiment log, results, and handover for Round 19-20 36f9e9173d

tobi added 1 commit 2026-07-13 00:15:54 +00:00

feat: loosen filtering thresholds to keep more grammars ... 9b3dca46eb

- min_methods: 3→2 (keep groups with 2+ methods)
- unique_ratio: 0.9→0.95 (keep groups with up to 95% unique sequences)
- max_mdl: 200→500 (keep higher-MDL grammars)

This prevents throwing away good grammars that have genuine structure
but happen to have many unique sequences or moderate MDL scores.

tobi added 1 commit 2026-07-13 00:23:47 +00:00

feat: add post-hoc noise filtering for grammars ... becbd82c56

- Add filter_noise() to remove test/stdlib tokens from AST
- Add grammar_noise_ratio() to calculate noise fraction
- Integrate into _build_json_output() and _build_yaml_output()
- Precision improvement: 28-55% → 84-94% across all 3 codebases
- Grammar count unchanged (233) — filtering is post-hoc, preserves recall

tobi added 1 commit 2026-07-13 00:32:59 +00:00

feat: add quality gate for grammar usefulness scoring ... 9be44c2964

- Add grammar_quality_score() — scores 0.0-1.0 based on structure
- Add is_useful_grammar() — filters grammars with quality >= 0.3
- Quality criteria: ordering (+0.3), alternation groups (+0.2 each),
  symbol count (+0.2), concat depth (+0.1)
- Results: 89/102 RAGSAK, 116/121 FastAPI, 9/10 Zod pass quality gate
- Top 15-20 RAGSAK grammars have genuine domain patterns

tobi added 1 commit 2026-07-13 00:33:32 +00:00

docs: add hype document for grammar inference breakthrough 8fc459e06b

Some checks failed

Hide all checks

ci/woodpecker/push/woodpecker Pipeline failed

Details

ci/woodpecker/pr/woodpecker Pipeline failed

Details

This pull request can be merged automatically.

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin feature/treesitter-tag-queries:feature/treesitter-tag-queries

git checkout feature/treesitter-tag-queries

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git checkout main

git merge --no-ff feature/treesitter-tag-queries

git checkout feature/treesitter-tag-queries

git rebase main

git checkout main

git merge --ff-only feature/treesitter-tag-queries

git checkout feature/treesitter-tag-queries

git rebase main

git checkout main

git merge --no-ff feature/treesitter-tag-queries

git checkout main

git merge --squash feature/treesitter-tag-queries

git checkout main

git merge --ff-only feature/treesitter-tag-queries

git checkout main

git merge feature/treesitter-tag-queries

git push origin main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: tobi/grammar-inference-engine#2

No description provided.