tobi/grammar-inference-engine
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
grammar-inference-engine/docs/adr/0001-use-nvim-treesitter-highlights-scm.md
tobjend ca7ccb36ff
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
Details
docs: add 8 architecture decision records 
ADR 0001: nvim-treesitter highlights.scm as capture source
ADR 0002: language-agnostic method extraction via child_by_field_name
ADR 0003: method-level n-gram clustering before inference
ADR 0004: frequency filter with min_coverage threshold
ADR 0005: import extraction per cluster
ADR 0006: argument pattern extraction via AST node classification
ADR 0007: JSON output for LLM prompt injection
ADR 0008: BEX ensemble for grammar inference
2026-07-03 22:01:35 +02:00

2.1 KiB
Raw Blame History

1. Use nvim-treesitter highlights.scm as behavioral capture source

Date: 2026-07-03

Status: Accepted

Context

We need a universal source of behavioral code tokens (function calls, references, definitions) across multiple programming languages. Options:

  • tags.scm (nvim-treesitter): Purpose-built for symbol tagging. Covers definitions and references.
  • highlights.scm (nvim-treesitter): Built for syntax highlighting. Covers a wider range of tokens including keywords, operators, and built-ins.
  • Custom per-language queries: Write and maintain our own query files for each language.

We need tokens that represent what the code does at runtime — not just structure.

Decision

Use nvim-treesitter highlights.scm as the capture source for all 10 languages.

We filter captures to a BEHAVIORAL_PREFIXES set: definition., reference., keyword., function, attribute, constructor, label, type.definition, module.

For Kotlin, use the ts-kotlin (fwcd fork) bundled highlights.scm instead of nvim-treesitter's, because nvim-treesitter's Kotlin query references a duplicate annotation node type that doesn't exist in the grammar.

Consequences

Positive:

  • highlights.scm covers 4 out of 5 behavioral capture types that tags.scm misses, across all 10 languages.
  • No per-language custom code or adapters needed.
  • Community-maintained queries stay fresh with language evolution.
  • Same query files work for both parsing and tokenizing.

Negative:

  • highlights.scm includes non-behavioral captures (comments, punctuation, operators) — we filter these out.
  • Two jsx captures use #set! with 3 arguments, which py-tree-sitter 0.26 rejects. Strip these 2 patterns.
  • Kotlin requires a separate grammar package (ts-kotlin) because the nvim-treesitter Kotlin grammar is incompatible.

Alternatives Considered

  • tags.scm: Cleaner signal-to-noise ratio, but misses function, attribute, constructor, module captures that are essential for behavioral understanding.
  • Custom queries: Would give full control but require per-language maintenance — violates our universal-preprocessor constraint.