tobi/grammar-inference-engine
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
grammar-inference-engine/docs/adr/0001-use-nvim-treesitter-highlights-scm.md
tobjend e23922a1b7
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
Details
feat: adaptive multi-assignment clustering; add ADRs 1-10 
- Multi-assignment clustering (no greedy 'used' set)
- Adaptive ngram fallback (shrink when (other) > 60%)
- Add docs/adr/ with 10 architecture decision records
- Fix ADR 1 (query modification description)
- Fix ADR 3 (multi-assignment + adaptive shrink)
- Fix ADR 5 (import sort order clarification)
- Fix ADR 6 (remove Kotlin call_suffix references)
- New ADR 9 (adaptive clustering rationale)
- New ADR 10 (universal package mapping via relpath)
2026-07-03 22:58:09 +02:00

2.2 KiB
Raw Blame History

1. Use nvim-treesitter highlights.scm as behavioral capture source

Date: 2026-07-03

Status: Accepted

Context

We need a universal source of behavioral code tokens (function calls, references, definitions) across multiple programming languages. Options:

  • tags.scm (nvim-treesitter): Purpose-built for symbol tagging. Covers definitions and references.
  • highlights.scm (nvim-treesitter): Built for syntax highlighting. Covers a wider range of tokens including keywords, operators, and built-ins.
  • Custom per-language queries: Write and maintain our own query files for each language.

We need tokens that represent what the code does at runtime — not just structure.

Decision

Use nvim-treesitter highlights.scm as the capture source for all 10 languages.

We filter captures to a BEHAVIORAL_PREFIXES set: definition., reference., keyword., function, attribute, constructor, label, type.definition, module.

For Kotlin, use the ts-kotlin (fwcd fork) bundled highlights.scm instead of nvim-treesitter's, because nvim-treesitter's Kotlin query references a duplicate annotation node type that doesn't exist in the grammar.

Consequences

Positive:

  • highlights.scm covers 4 out of 5 behavioral capture types that tags.scm misses, across all 10 languages.
  • No per-language custom code or adapters needed.
  • Community-maintained queries stay fresh with language evolution.
  • Same query files work for both parsing and tokenizing.

Negative:

  • highlights.scm includes non-behavioral captures (comments, punctuation, operators) — we filter these out.
  • Some nvim-treesitter queries use #set! directives (#set! priority, #eq?) that py-tree-sitter doesn't support. These patterns are removed in the bundled copies under queries/.
  • Kotlin requires a separate grammar package (ts-kotlin) because the nvim-treesitter Kotlin grammar is incompatible.

Alternatives Considered

  • tags.scm: Cleaner signal-to-noise ratio, but misses function, attribute, constructor, module captures that are essential for behavioral understanding.
  • Custom queries: Would give full control but require per-language maintenance — violates our universal-preprocessor constraint.