tobi/grammar-inference-engine
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

feature/treesitter-tag-queries #2

Open
tobi wants to merge 78 commits from feature/treesitter-tag-queries into main
pull from: feature/treesitter-tag-queries
merge into: tobi:main
Conversation 0 Commits 78 Files changed 246 +24 -7
1 changed files with 24 additions and 7 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files
Showing only changes of commit 8bda174293 - Show all commits

31
references/tags-queries/ANALYSIS.md
View file

@ -200,13 +200,30 @@ Checked July 3, 2026 via `raw.githubusercontent.com/nvim-treesitter/nvim-treesit
All tags.scm returned 404 — nvim-treesitter may have restructured or moved to a different branch/tag.
## Resolution
This analysis was performed against community **tags.scm** queries. The actual implementation uses **nvim-treesitter `highlights.scm`** instead, which resolves the critical gaps:
| Gap in tags.scm | highlights.scm outcome |
|---|---|
| Missing control flow (`if`/`else`/`when`) | `@keyword.conditional` — all languages |
| Missing error handling (`try`/`catch`/`throw`) | `@keyword.exception` — all languages |
| Missing `@return` | `@keyword.return` — all languages |
| Missing annotations/decorators | Java/Kotlin `@attribute`, Rust `@attribute`, Python `@function` (decorator) |
| Naming inconsistency across langs | Mitigated by **prefix-based filter** (`definition.*`, `keyword.*`, `reference.*`, etc.) — exact capture names don't need to match, only prefixes |
Highlights.scm proved richer than tags.scm for behavioral sequence extraction, because highlight queries deliberately distinguish keywords, functions, calls, and attributes — exactly what Dervish needs.
### Special Cases
Two languages required deviations from the plain nvim-treesitter approach:
**Kotlin** — nvim-treesitter `queries/kotlin/highlights.scm` fails at row 296 due to a conflict between the grammar's duplicate `annotation` node type (id 92 named=False, id 304 named=True). Resolution: use `ts-kotlin`'s bundled `queries/highlights.scm` instead (`pip install ts-kotlin`). This is the fwcd grammar fork, which also includes the `simple_identifier` node needed by highlight queries.
**JavaScript/TypeScript** — JS query uses `; inherits: ecma, jsx` and TS uses `; inherits: ecma`. The `code.py` preprocessor resolves inheritance by concatenating parent query content before child content. Additionally, `jsx.scm` contains two `#set!` predicates with 3 arguments (`#set! @_capture property value`) which py-tree-sitter 0.26 rejects (expects 1-2 args). Resolution: those two `bo.commentstring` patterns were stripped from `jsx.scm`.
## Verdict
**Community queries alone are insufficient for Dervish sequence extraction.** They:
1. Purpose-built for navigation, not behavioral sequence extraction
2. Lack critical behavioral captures (annotations, control flow, error handling)
3. Inconsistent across languages
4. No sequencing/traversal mechanism
5. tags.scm no longer available on nvim-treesitter master
**Community tags.scm alone are insufficient. But nvim-treesitter highlights.scm — used as-is with two caveats — covers 4/5 gaps.** The remaining gap (no sequencing/traversal mechanism) is handled by tree-sitter's built-in ordered cursor iteration over captures, which naturally preserves source order.
A **universal query file** (like the deleted `universal.scm`) using actual tree-sitter node type names is the correct approach, supplemented by a custom AST traversal to produce ordered sequences.
A universal query file is no longer needed. The `code.py` preprocessor loads highlights.scm per-language, resolves `; inherits:` directives, handles the two known deviations (Kotlin bundled query, jsx `#set!` stripping), and filters by behavioral capture prefix — zero adapters, zero per-language branches.