docs: update stale docs — remove kORE from default ensemble, add tag preprocessor CLI

AGENTS.md

@@ -26,16 +26,17 @@ g = idregex([['a','b','c'], ['a','b'], ['a','c']], kmax=2, N=3)
 
 ## Architecture
 
-Two inference pipelines:
+Three inference pipelines:
 
 | Pipeline | When to use |
 |----------|-------------|
 | CRX (fast) | Many examples, need speed, CHAREs output |
 | iDRegEx (robust) | Few/noisy examples, need probabilistic handling |
+| Tag Preprocessor (`bex.tag_preprocessor`) | Source code analysis — tree-sitter AST → method-level call sequences → per-package grammars |
 
 ## Running Tests
 ```bash
-python tests/test_bex.py
+python -m pytest tests/
 ```
 
 ## MCP Server
@@ -44,6 +45,17 @@ The primary interface is an MCP server exposing a single tool:
 
 | Tool | Parameters | What it does |
 |------|-----------|-------------|
-| `infer_best_grammar` | `sequences`, `prefer`, `kmax`, `N` | Runs CRX + iDRegEx, picks best by MDL. `prefer='crx'` or `prefer='idregex'` skips ensemble. |
+| `infer_best_grammar` | `sequences`, `prefer`, `kmax`, `N`, `min_coverage` | Runs CRX + iDRegEx, picks best by MDL. `prefer='crx'` or `prefer='idregex'` skips ensemble. `min_coverage < 1.0` runs core+outlier analysis. |
 
 Start it: `python /path/to/bex/mcp_server.py`, then connect any MCP client.
+
+## Tag Preprocessor CLI
+
+For analyzing source code directories:
+
+```bash
+python -m bex.tag_preprocessor.analyze /path/to/codebase --verbose
+python -m bex.tag_preprocessor.analyze /path/to/codebase --slice package --include '**/src/**'
+```
+
+Key flags: `--slice package` (per-directory grammars), `--verbose` (progress), `--include`/`--exclude` (glob filters), `--kore` (enable slow kORE in ensemble).

README.md

@@ -50,10 +50,10 @@ The primary interface is a **Model Context Protocol (MCP)** server. Connect any
 
 | Tool | Parameters | What it does |
 |------|-----------|-------------|
-| `infer_best_grammar` | `sequences`, `prefer`, `kmax`, `N`, `min_coverage` | **The only tool you need.** Runs CRX + iDRegEx + kOREInference, picks best by MDL. Set `prefer` to run only one algorithm. Set `min_coverage < 1.0` for optional core+outlier analysis. |
+| `infer_best_grammar` | `sequences`, `prefer`, `kmax`, `N`, `min_coverage` | **The only tool you need.** Runs CRX + iDRegEx, picks best by MDL. Set `prefer` to run only one algorithm. Set `min_coverage < 1.0` for optional core+outlier analysis. |
 
 **Parameters explained:**
-- **`prefer`**: `'crx'` for full vocabulary (accepts all sequences), `'idregex'` or `'koreinference'` for deterministic minimal core. Omit to let MDL pick the winner across all three.
+- **`prefer`**: `'crx'` for full vocabulary (accepts all sequences), `'idregex'` for deterministic minimal core, `'koreinference'` for k-OA with rwr₀ repair (slow, rarely wins). Omit to let MDL pick the winner across CRX and iDRegEx.
 - **`kmax`** (1–5): Context window for k-ORE inference (iDRegEx, kOREInference). Higher values capture longer-range dependencies but need more data and are slower. Default 2 works for most cases.
 - **`N`** (1–10): Random trials for k-ORE inference. More = better convergence but slower. Default 3.
 - **`min_coverage`** (0.5–1.0): **Optional core+outlier analysis.** When < 1.0, iteratively removes outlier sequences (those with the rarest symbols) until at least this fraction remain. Returns the core CRX grammar for the majority plus a list of removed outliers. Default 1.0 = disabled. Example: `min_coverage=0.8` finds the tight pattern for ~80% of examples while flagging the other ~20% as variants.
@@ -114,6 +114,17 @@ print(f"Grammar: {result['best']['grammar']}")
 print(f"Score: {result['best']['mdl_score']}")
 ```
 
+### Tag Preprocessor (source code analysis)
+
+For analyzing source code directories (tree-sitter based):
+
+```bash
+python -m bex.tag_preprocessor.analyze /path/to/codebase --verbose
+python -m bex.tag_preprocessor.analyze /path/to/codebase --slice package --include '**/src/**'
+```
+
+Key flags: `--slice package` (per-directory grammars), `--verbose` (progress), `--include`/`--exclude` (glob filters), `--kore` (enable slow kORE in ensemble).
+
 ## Why not just use a schema?
 
 Many of the things developers build every day **have no formal schema**. They're free-form scripts, config files, or YAML blobs where the structure is emergent convention, not enforced specification. An LLM generating new content in these domains needs to know the convention — but it's never been written down.
@@ -147,7 +158,7 @@ The sweet spot: **multiple implementations of the same abstract task** with a sh
 |------|-----|-----|
 | Clean, structured data with full vocabulary | **CRX** | Single-pass, deterministic. Accepts all sequences. |
 | Few examples, or want minimal common core | **iDRegEx** or **kOREInference** | Probabilistic EM, finds only what's shared. |
-| Don't know which is better | **Ensemble (default)** | Runs all three, picks best by MDL score. |
+| Don't know which is better | **Ensemble (default)** | Runs CRX + iDRegEx, picks best by MDL score. |
 | Want core pattern + outlier detection | **Ensemble + `min_coverage<1`** | Finds tight grammar for majority, flags outliers. |
 | Data is clearly one type | `prefer='crx'` | Skips ensemble comparison, runs CRX alone. |
 

bex/mcp_server.py

@@ -26,13 +26,19 @@ def infer_best_grammar(
     than passing all examples. Pass the existing sequences, get back a
     pattern you can follow to generate new instances.
 
+    Runs CRX + iDRegEx, picks best by MDL score. kORE is excluded by
+    default (slow, rarely wins on real data). Set prefer='koreinference'
+    to force it.
+
     Args:
         sequences: List of sequences, each a list of strings (symbols in
             the order they appear). Example: [["file","copy","command"],
             ["file","template","command"]].
         prefer: Optional — 'crx' for full vocabulary (accepts all examples),
-            'idregex' for deterministic minimal core. Omit to auto-pick by MDL.
+            'idregex' for deterministic minimal core, 'koreinference' for
-        kmax: Context depth for k-ORE inference. Default 2.
+            k-OA with rwr0 repair (slow). Omit to auto-pick by MDL.
+        kmax: Context depth for k-ORE inference (iDRegEx, kOREInference).
+            Default 2.
         N: Random trials for k-ORE inference (higher = better, slower).
         min_coverage: (Expert) When < 1.0, also runs a **core+outlier analysis**:
             iteratively removes outlier sequences (those with rarest symbols)