grammar-inference-engine/experiments/round15/PIPELINE.md

Pipeline Architecture — Grammar Inference Engine

Full Pipeline: Source Code → GBNF Grammar

  Source Code (directory of .py/.kt/.ts files)
  │
  ▼
┌─────────────────────────────────────────────────────┐
│  1. SCAN  (scan_directory)                          │
│     Glob *.py/*.kt/*.ts, filter --include/--exclude │
│     Output: {ext: [file_paths]}                     │
└─────────────────────┬───────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────┐
│  2. PREPROCESS  (preprocess_by_method)              │
│     Tree-sitter parse → extract method-level call    │
│     sequences. Coarsen tokens (RETURN/IF/LOOP/EXC). │
│     Output: [(capture, text, line)] per method      │
│                                                     │
│     Key: code_bytes = code.encode() for correct     │
│     byte/char offset handling (Round 14 fix)        │
└─────────────────────┬───────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────┐
│  3. GROUP  (--slice mode)                           │
│                                                     │
│   flat      │ One group per language (no split)     │
│   package   │ Group by directory path, merge small   │
│   reduce    │ Group by dir + Reduce similarity       │
│   ilocal    │ Group by iLocal context extraction     │
│                                                     │
│   Output: {group_label: [sequences]}                │
└─────────────────────┬───────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────┐
│  4. FILTER  (frequency_filter)                      │
│     Remove symbols in < min_coverage fraction of    │
│     methods. Default min_coverage = 0.05 (5%).      │
└─────────────────────┬───────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────┐
│  5. SPLIT (--split-mixed)                           │
│     _recursive_split(): group by first symbol,      │
│     split until all subgroups have one first-symbol. │
│     max_depth=3. Returns leaf subgroups.             │
└─────────────────────┬───────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────┐
│  6. INFER  (infer_ensemble)                         │
│                                                     │
│   CRX (always)  → 2ms, deterministic                │
│   iDRegEx (opt) → 100-700ms, probabilistic          │
│   kORE (opt)    → 400-700ms, Baum-Welch             │
│                                                     │
│   Pick best by lang_size score (lower = tighter).    │
│   Output: SORE grammar string                       │
│                                                     │
│   ┌───────────────────────────────────────────────┐  │
│   │  6b. iDRegEx REFINEMENT (Round 16, opt-in)   │  │
│   │                                               │  │
│   │  Only triggers when ALL:                      │  │
│   │    - n_methods ≤ 10                           │  │
│   │    - CRX grammar has >50% top-level optionals │  │
│   │                                               │  │
│   │  If iDRegEx grammar is >10x tighter by        │  │
│   │  lang_size: use iDRegEx. Else keep CRX.       │  │
│   └───────────────────────────────────────────────┘  │
└─────────────────────┬───────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────┐
│  7. VALIDATE  (validate_sore + grammar_structure)   │
│     - Parse SORE → check syntax                     │
│     - Compute structure score (0-1)                  │
│     - Filter by min_structure (default 0.0)          │
│     - Reject malformed grammars                      │
└─────────────────────┬───────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────┐
│  8. OUTPUT                                          │
│     Text:  Package → Grammar + Score + Args         │
│     YAML:  Persisted to .dervish/grammars.yml       │
│     JSON:  Machine-readable export                  │
│                                                     │
│   GrammarIndex (runtime): for MCP tool queries      │
└─────────────────────────────────────────────────────┘

CLI Parameters

python -m bex.tag_preprocessor.analyze <directory> [options]

REQUIRED:
  directory              Path to scan

GROUPING:
  --slice flat|package|reduce|ilocal   (default: flat)
  --split-mixed                        Split mixed first-symbol groups
  --reduce-threshold FLOAT             Reduce similarity (default: 0.15)
  --context-strategy STRATEGY          iLocal strategy (default: dir)

FILTERING:
  --include GLOB          Include files matching pattern
  --exclude GLOB          Exclude files matching pattern
  --main-only             Exclude test files
  --min-coverage FLOAT    Symbol frequency threshold (default: 0.05)
  --min-methods INT       Min methods per group (default: 3)
  --min-structure FLOAT   Min structure score to keep (default: 0.0)

ALGORITHMS:
  --prefer crx|idregex    Force single algorithm
  --kore                  Include kORE in ensemble
  --idregex               Include iDRegEx in ensemble
  --idregex-refine        Refine CRX flat bags with iDRegEx (Round 16)
  --kmax INT              Max k for k-ORE (default: 2)
  --crx-method standard|refined  (default: standard)
  --scoring-method langsize|mdl  (default: langsize)

OUTPUT:
  --format text|json      Output format
  --json                  Shortcut for --format json
  --verbose               Print progress

Decision Matrix: Which Algorithm When?

┌─────────────────────────┬──────────────┬───────────────┬──────────────┐
│ Scenario                │ CRX          │ iDRegEx       │ kORE         │
│                         │ (0-9ms)      │ (100-700ms)   │ (400-700ms)  │
├─────────────────────────┼──────────────┼───────────────┼──────────────┤
│ Speed                   │ 1x           │ 100-350x      │ 200-350x     │
│                         │              │ slower        │ slower       │
├─────────────────────────┼──────────────┼───────────────┼──────────────┤
│ Small structured        │ ✅ Best       │ ✅ Better      │ ✅ Same as    │
│ (3-5 methods,           │ a.b?.c?.d?   │ a.(b|c).(d|e) │ iDRegEx      │
│ clear branching)        │ score=1.0    │ score=0.86    │              │
├─────────────────────────┼──────────────┼───────────────┼──────────────┤
│ Flat CRX bag            │ ✅ Always     │ ⚠️ Sometimes  │ ❌ Often      │
│ (many optional parts,   │ produces     │ None. When it  │ None.        │
│ small group ≤10)        │ something    │ works: >10x   │ Adds nothing │
│                         │              │ tighter!      │ over iDRegEx │
├─────────────────────────┼──────────────┼───────────────┼──────────────┤
│ Medium/Large diverse    │ ✅ Default    │ ❌ None or     │ ❌ None or    │
│ (>10 methods,           │ always works │ trivial        │ trivial      │
│ diverse patterns)       │              │               │              │
├─────────────────────────┼──────────────┼───────────────┼──────────────┤
│ Very diverse            │ ✅ Only       │ ✅ But trivial │ ✅ But trivial│
│ (no shared structure)   │ reliable     │ (a.b.c|x.y.z) │ (same)       │
│                         │ option       │ score=0.0     │              │
└─────────────────────────┴──────────────┴───────────────┴──────────────┘

DECISION LOGIC (--idregex-refine enabled):

  1. Run CRX                          → always, fast
  2. Count top-level optional parts
  3. IF n_methods ≤ 10
     AND optionals/parts > 50%        → CRX produced flat optional chain
     THEN run iDRegEx
       IF iDRegEx returns grammar
          AND lang_size improvement > 10x
          → use iDRegEx               (477x tighter on RAGSAK example!)
          ELSE keep CRX
  4. ELSE                             → keep CRX

SCORING:

  lang_size_score (Bex et al.):   Lower = better
    Counts how many words the grammar accepts at each input length.
    CRX flat chains:     9432 words  (a?.b?.c?.d?.e?.f? → many combos)
    iDRegEx disjunctions:   60 words  (a.(b|c).(d|e) → exact paths)

  grammar_structure_score:        Higher = more ordering info
    Measures dots, optionals, repetition vs disjunction ratio.
    Used for filtering (min_structure), NOT for algorithm selection.

  mdl_score:                      Lower = better (model + data cost)
    Fallback scoring method. lang_size is preferred.

Speed Profile (empirical, RAGSAK .kt 462 files)

Phase                     │ Time    │ Notes
──────────────────────────┼─────────┼──────────────────────────────────
Scan + Preprocess         │ 2.8s    │ ProcessPoolExecutor parallel
Group + Filter            │ <0.1s   │ In-memory
Split (recursive)         │ <0.1s   │ Pure Python
CRX Inference (27 groups) │ ~54s    │ 27 groups × ~2s each
iDRefinement (1 group)    │ ~0.7s   │ 1 eligible candidate
Validate + Output         │ <0.1s   │
──────────────────────────┼─────────┼──────────────────────────────────
Total (--split-mixed)     │ ~74s    │
Total (+ --idregex-refine)│ ~74.7s  │ +0.7s negligible

Without --split-mixed:    │ ~13s    │ Fewer groups, faster