diff --git a/AGENTS.md b/AGENTS.md index e36ab9e..d1db4b8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,16 +26,17 @@ g = idregex([['a','b','c'], ['a','b'], ['a','c']], kmax=2, N=3) ## Architecture -Two inference pipelines: +Three inference pipelines: | Pipeline | When to use | |----------|-------------| | CRX (fast) | Many examples, need speed, CHAREs output | | iDRegEx (robust) | Few/noisy examples, need probabilistic handling | +| Tag Preprocessor (`bex.tag_preprocessor`) | Source code analysis — tree-sitter AST → method-level call sequences → per-package grammars | ## Running Tests ```bash -python tests/test_bex.py +python -m pytest tests/ ``` ## MCP Server @@ -44,6 +45,17 @@ The primary interface is an MCP server exposing a single tool: | Tool | Parameters | What it does | |------|-----------|-------------| -| `infer_best_grammar` | `sequences`, `prefer`, `kmax`, `N` | Runs CRX + iDRegEx, picks best by MDL. `prefer='crx'` or `prefer='idregex'` skips ensemble. | +| `infer_best_grammar` | `sequences`, `prefer`, `kmax`, `N`, `min_coverage` | Runs CRX + iDRegEx, picks best by MDL. `prefer='crx'` or `prefer='idregex'` skips ensemble. `min_coverage < 1.0` runs core+outlier analysis. | Start it: `python /path/to/bex/mcp_server.py`, then connect any MCP client. + +## Tag Preprocessor CLI + +For analyzing source code directories: + +```bash +python -m bex.tag_preprocessor.analyze /path/to/codebase --verbose +python -m bex.tag_preprocessor.analyze /path/to/codebase --slice package --include '**/src/**' +``` + +Key flags: `--slice package` (per-directory grammars), `--verbose` (progress), `--include`/`--exclude` (glob filters), `--kore` (enable slow kORE in ensemble). diff --git a/README.md b/README.md index b7f4341..0d45477 100644 --- a/README.md +++ b/README.md @@ -50,10 +50,10 @@ The primary interface is a **Model Context Protocol (MCP)** server. Connect any | Tool | Parameters | What it does | |------|-----------|-------------| -| `infer_best_grammar` | `sequences`, `prefer`, `kmax`, `N`, `min_coverage` | **The only tool you need.** Runs CRX + iDRegEx + kOREInference, picks best by MDL. Set `prefer` to run only one algorithm. Set `min_coverage < 1.0` for optional core+outlier analysis. | +| `infer_best_grammar` | `sequences`, `prefer`, `kmax`, `N`, `min_coverage` | **The only tool you need.** Runs CRX + iDRegEx, picks best by MDL. Set `prefer` to run only one algorithm. Set `min_coverage < 1.0` for optional core+outlier analysis. | **Parameters explained:** -- **`prefer`**: `'crx'` for full vocabulary (accepts all sequences), `'idregex'` or `'koreinference'` for deterministic minimal core. Omit to let MDL pick the winner across all three. +- **`prefer`**: `'crx'` for full vocabulary (accepts all sequences), `'idregex'` for deterministic minimal core, `'koreinference'` for k-OA with rwr₀ repair (slow, rarely wins). Omit to let MDL pick the winner across CRX and iDRegEx. - **`kmax`** (1–5): Context window for k-ORE inference (iDRegEx, kOREInference). Higher values capture longer-range dependencies but need more data and are slower. Default 2 works for most cases. - **`N`** (1–10): Random trials for k-ORE inference. More = better convergence but slower. Default 3. - **`min_coverage`** (0.5–1.0): **Optional core+outlier analysis.** When < 1.0, iteratively removes outlier sequences (those with the rarest symbols) until at least this fraction remain. Returns the core CRX grammar for the majority plus a list of removed outliers. Default 1.0 = disabled. Example: `min_coverage=0.8` finds the tight pattern for ~80% of examples while flagging the other ~20% as variants. @@ -114,6 +114,17 @@ print(f"Grammar: {result['best']['grammar']}") print(f"Score: {result['best']['mdl_score']}") ``` +### Tag Preprocessor (source code analysis) + +For analyzing source code directories (tree-sitter based): + +```bash +python -m bex.tag_preprocessor.analyze /path/to/codebase --verbose +python -m bex.tag_preprocessor.analyze /path/to/codebase --slice package --include '**/src/**' +``` + +Key flags: `--slice package` (per-directory grammars), `--verbose` (progress), `--include`/`--exclude` (glob filters), `--kore` (enable slow kORE in ensemble). + ## Why not just use a schema? Many of the things developers build every day **have no formal schema**. They're free-form scripts, config files, or YAML blobs where the structure is emergent convention, not enforced specification. An LLM generating new content in these domains needs to know the convention — but it's never been written down. @@ -147,7 +158,7 @@ The sweet spot: **multiple implementations of the same abstract task** with a sh |------|-----|-----| | Clean, structured data with full vocabulary | **CRX** | Single-pass, deterministic. Accepts all sequences. | | Few examples, or want minimal common core | **iDRegEx** or **kOREInference** | Probabilistic EM, finds only what's shared. | -| Don't know which is better | **Ensemble (default)** | Runs all three, picks best by MDL score. | +| Don't know which is better | **Ensemble (default)** | Runs CRX + iDRegEx, picks best by MDL score. | | Want core pattern + outlier detection | **Ensemble + `min_coverage<1`** | Finds tight grammar for majority, flags outliers. | | Data is clearly one type | `prefer='crx'` | Skips ensemble comparison, runs CRX alone. | diff --git a/bex/mcp_server.py b/bex/mcp_server.py index 226ff5a..d801927 100644 --- a/bex/mcp_server.py +++ b/bex/mcp_server.py @@ -26,13 +26,19 @@ def infer_best_grammar( than passing all examples. Pass the existing sequences, get back a pattern you can follow to generate new instances. + Runs CRX + iDRegEx, picks best by MDL score. kORE is excluded by + default (slow, rarely wins on real data). Set prefer='koreinference' + to force it. + Args: sequences: List of sequences, each a list of strings (symbols in the order they appear). Example: [["file","copy","command"], ["file","template","command"]]. prefer: Optional — 'crx' for full vocabulary (accepts all examples), - 'idregex' for deterministic minimal core. Omit to auto-pick by MDL. - kmax: Context depth for k-ORE inference. Default 2. + 'idregex' for deterministic minimal core, 'koreinference' for + k-OA with rwr0 repair (slow). Omit to auto-pick by MDL. + kmax: Context depth for k-ORE inference (iDRegEx, kOREInference). + Default 2. N: Random trials for k-ORE inference (higher = better, slower). min_coverage: (Expert) When < 1.0, also runs a **core+outlier analysis**: iteratively removes outlier sequences (those with rarest symbols)