README.md

@@ -51,14 +51,19 @@ The primary interface is a **Model Context Protocol (MCP)** server. Connect any
 | Tool | Parameters | What it does |
 |------|-----------|-------------|
 | `infer_best_grammar` | `sequences`, `prefer`, `kmax`, `N`, `min_coverage` | Infer grammar from raw sequences. Runs CRX + iDRegEx, picks best by MDL. Set `prefer` to run only one algorithm. Set `min_coverage < 1.0` for optional core+outlier analysis. |
-| `analyze_directory` | `directory`, `slice`, `min_coverage`, `prefer`, `kmax`, `include`, `exclude`, `main_only`, `max_mdl`, `persist` | Scan a source code directory and infer behavioral conventions per package. Returns YAML grouped by module. Auto-persists to `{directory}/.dervish/grammars.yml`. |
+| `analyze_directory` | `directory`, `slice`, `min_coverage`, `prefer`, `kmax`, `include`, `exclude`, `main_only`, `max_mdl`, `persist`, `min_structure`, `decompose`, `max_seq_length`, `cluster_method`, `crx_method` | Scan a source code directory and infer behavioral conventions per package. Returns YAML grouped by module. Auto-persists to `{directory}/.dervish/grammars.yml`. |
 
 **`analyze_directory` parameters:**
 - **`directory`**: Path to source code directory to analyze.
 - **`slice`**: `'package'` (per-directory, default) or `'flat'` (one per language).
 - **`main_only`**: Exclude test files when `True`. Default `False`.
 - **`max_mdl`**: Drop groups with MDL above this threshold. Default 200. Lower = tighter patterns only.
+- **`min_structure`**: Minimum grammar structure score (0.0–1.0). Only returns grammars at or above this threshold. Default 0.5.
 - **`persist`**: Write results to `{directory}/.dervish/grammars.yml`. Default `True`.
+- **`decompose`**: Break long sequences into shorter fragments before inference. Helps on diverse codebases. Default `False`.
+- **`max_seq_length`**: Max fragment length when decomposing. Default 5.
+- **`cluster_method`**: `'first-symbol'` (fast) or `'distributional'` (context-similarity). Default `'first-symbol'`.
+- **`crx_method`**: `'standard'` (fast) or `'refined'` (cluster-then-infer). Default `'standard'`.
 - Other parameters same as `infer_best_grammar`.
 
 **Parameters explained for `infer_best_grammar`:**
@@ -130,9 +135,12 @@ For analyzing source code directories (tree-sitter based):
 ```bash
 python -m bex.tag_preprocessor.analyze /path/to/codebase --verbose
 python -m bex.tag_preprocessor.analyze /path/to/codebase --slice package --include '**/src/**'
+python -m bex.tag_preprocessor.analyze /path/to/codebase --slice package --decompose --min-structure 0.5
 ```
 
-Key flags: `--slice package` (per-directory grammars), `--verbose` (progress), `--include`/`--exclude` (glob filters), `--kore` (enable slow kORE in ensemble).
+Key flags: `--slice package` (per-directory grammars), `--decompose` (break long sequences into fragments for better inference), `--min-structure 0.5` (return only high-structure grammars), `--cluster-method distributional` (context-similarity clustering), `--verbose` (progress), `--include`/`--exclude` (glob filters).
+
+The pipeline outputs SORE grammars (converted to GBNF internally for llama.cpp constrained decoding).
 
 ## Why not just use a schema?
 
@@ -159,18 +167,20 @@ Dervish has been tested against public datasets from Ansible Galaxy, Helm, and G
 
 The sweet spot: **multiple implementations of the same abstract task** with a shared but undocumented pattern. Not everything works — Dockerfiles, pre-commit configs, and schema-enforced formats are too rigid or too diverse to yield a convention.
 
-> **kOREInference note:** Algorithm 4 (iDRegEx with MDL, arXiv 1004.2372) is included for paper-faithful correctness. On real tool-sequence data, its rwr₀ repair step returns ∅ because the k-OA is rarely SORE (interconnected symbols). The ensemble falls back to CRX or iDRegEx automatically.
+> **kOREInference note:** Algorithm 4 (iDRegEx with MDL, arXiv 1004.2372) is available via `--kore` flag for paper-faithful correctness. On real tool-sequence data, its rwr₀ repair step returns ∅ because the k-OA is rarely SORE (interconnected symbols). The ensemble falls back to CRX or iDRegEx automatically.
 
 ## Algorithm Selection Guide
 
 | When | Use | Why |
 |------|-----|-----|
 | Clean, structured data with full vocabulary | **CRX** | Single-pass, deterministic. Accepts all sequences. |
-| Few examples, or want minimal common core | **iDRegEx** or **kOREInference** | Probabilistic EM, finds only what's shared. |
+| Few examples, or want minimal common core | **iDRegEx** | Probabilistic EM, finds only what's shared. |
 | Don't know which is better | **Ensemble (default)** | Runs CRX + iDRegEx, picks best by MDL score. |
 | Want core pattern + outlier detection | **Ensemble + `min_coverage<1`** | Finds tight grammar for majority, flags outliers. |
 | Data is clearly one type | `prefer='crx'` | Skips ensemble comparison, runs CRX alone. |
 
+> **kORE note:** kOREInference (Algorithm 4) is available via `--kore` flag but excluded from the default ensemble. On real-world data it returns ∅ in ~80% of cases because the k-OA is rarely SORE. CRX or iDRegEx handle these cases better.
+
 ## When each algorithm wins
 
 | Data property | Winner | Why |
@@ -178,7 +188,7 @@ The sweet spot: **multiple implementations of the same abstract task** with a sh
 | Diverse patterns, full vocabulary needed | CRX | Captures all symbols. iDRegEx returns ∅. |
 | Clean sequences with clear core | iDRegEx | Extracts minimal common subsequence. CRX buries it in optional noise. |
 | Interconnected (non-SORE) data | CRX | kOREInference (rwr₀) returns ∅ when k-OA is not SORE. CRX handles it. |
-| Single sequence | iDRegEx (+ RWR₀) | RWR₀ repair produces a grammatical regex from one example. |
+| Single sequence | iDRegEx | iDRegEx handles noise better. |
 | 2–3 sequences | iDRegEx | CRX overfits. iDRegEx handles noise better. |
 | Many sequences, tight pattern | CRX | Learns precise concatenation with optional suffixes. |
 | Want majority pattern + outlier list | CRX + `min_coverage` | Core analysis finds tight grammar for ~80%, flags the rest. |

bex/mcp_server.py

@@ -41,9 +41,7 @@ def infer_best_grammar(
     than passing all examples. Pass the existing sequences, get back a
     pattern you can follow to generate new instances.
 
-    Runs CRX + iDRegEx, picks best by scoring. kORE is excluded by
-    default (slow, rarely wins on real data). Set prefer='koreinference'
-    to force it.
+    Runs CRX + iDRegEx, picks best by scoring.
 
     Args:
         sequences: List of sequences, each a list of strings (symbols in
@@ -106,8 +104,12 @@ def analyze_directory(
     persist: bool = True,
     method: str = "langsize",
     min_methods: int = 3,
-    min_structure: float = 0.2,
+    min_structure: float = 0.5,
     split_mixed: bool = True,
+    decompose: bool = False,
+    max_seq_length: int = 5,
+    cluster_method: str = "first-symbol",
+    crx_method: str = "standard",
 ) -> str:
     """Scan a source code directory and infer behavioral conventions
     (regular expression grammars) per package. Returns compact patterns
@@ -140,13 +142,25 @@ def analyze_directory(
         method: Scoring method — 'langsize' (default) or 'mdl'.
         min_methods: Minimum methods per group to attempt inference. Default 3.
         min_structure: Minimum grammar structure score (0.0–1.0). Groups
-            below this produce flat bags. Default 0.2.
+            below this produce flat bags. Default 0.5 (only returns
+            high-structure grammars).
         split_mixed: When True (default), recursively split groups with
             diverse first symbols into uniform sub-groups before inference.
+        decompose: When True, decompose long sequences into shorter
+            fragments before inference. Helps on codebases with many
+            unique method patterns. Default False.
+        max_seq_length: Maximum fragment length when decompose=True. Default 5.
+        cluster_method: How to split mixed groups — 'first-symbol' (fast,
+            crude) or 'distributional' (context-similarity clustering).
+            Default 'first-symbol'.
+        crx_method: CRX variant — 'standard' (fast, Algorithm 7) or
+            'refined' (cluster-then-infer, tighter on flat bags).
+            Default 'standard'.
 
     Returns:
         YAML string with grammars grouped by top-level module, sorted
-        by score (tightest/most useful first).
+        by score (tightest/most useful first). Only returns grammars
+        meeting the min_structure threshold.
     """
     results = _analyze_directory(
         directory,
@@ -161,8 +175,12 @@ def analyze_directory(
         min_methods=min_methods,
         split_mixed=split_mixed,
         min_structure=min_structure,
+        decompose=decompose,
+        max_seq_length=max_seq_length,
+        cluster_method=cluster_method,
+        crx_method=crx_method,
     )
-    yaml_content = _build_yaml_output(results, directory, max_mdl=max_mdl)
+    yaml_content = _build_yaml_output(results, directory, max_mdl=max_mdl, min_structure=min_structure)
     if persist:
         _persist_grammars(yaml_content, directory)
     return yaml_content

bex/tag_preprocessor/analyze.py

@@ -883,10 +883,11 @@ def _build_json_output(results):
     return json.dumps(output, indent=2)
 
 
-def _build_yaml_output(results, dir_path, max_mdl=200.0):
+def _build_yaml_output(results, dir_path, max_mdl=200.0, min_structure=0.0):
     """Build YAML output grouped by top-level module, sorted by MDL.
 
-    Filters out (other), no-grammar groups, and groups above max_mdl.
+    Filters out (other), no-grammar groups, groups above max_mdl,
+    and groups below min_structure.
     Returns YAML string.
     """
     import yaml
@@ -908,6 +909,8 @@ def _build_yaml_output(results, dir_path, max_mdl=200.0):
             best = result["best"]
             if best["mdl_score"] > max_mdl:
                 continue
+            if grammar_structure_score(best["grammar"]) < min_structure:
+                continue
 
             # Extract top-level module from package path
             parts = label.replace(os.sep, "/").split("/")
@@ -917,6 +920,8 @@ def _build_yaml_output(results, dir_path, max_mdl=200.0):
             all_grammars = meta.get("all_grammars", [])
             if all_grammars:
                 for leaf_label, leaf_grammar, leaf_score, leaf_count in all_grammars:
+                    if grammar_structure_score(leaf_grammar) < min_structure:
+                        continue
                     entry = {
                         "package": leaf_label,
                         "methods": leaf_count,

experiments/EXPERIMENT_LOG.md

@@ -770,3 +770,22 @@ The pipeline should: (1) run refined CRX, (2) if trivial, fall back to CRX.
 
 **Recommendation:** Make refined CRX the default for `--split-mixed` mode.
 Keep standard CRX as fallback. No need for iDRegEx or kORE in the pipeline.
+
+---
+
+## Round 18: Decomposition Forest (Phase 2)
+
+**Goal:** Break down long sequences into shorter fragments before inference (Crucio Phase 2).
+
+**Method:** Implemented `bex/decompose.py` with prefix/suffix/window extraction.
+
+**Results:**
+- RAGSAK: 21 → 80 grammars (3.8× increase)
+- FastAPI: 111 → 118 grammars (small increase)
+
+**Key insight:** Decomposition creates diverse fragments, so skip diversity check when enabled.
+
+**Files:**
+- `bex/decompose.py`: decompose_sequence(), decompose_all(), decompose_with_coverage()
+- `bex/tag_preprocessor/analyze.py`: --decompose, --max-seq-length flags
+- `tests/test_decompose.py`: 12 new tests