diff --git a/README.md b/README.md index e09743f..9be15e2 100644 --- a/README.md +++ b/README.md @@ -51,14 +51,19 @@ The primary interface is a **Model Context Protocol (MCP)** server. Connect any | Tool | Parameters | What it does | |------|-----------|-------------| | `infer_best_grammar` | `sequences`, `prefer`, `kmax`, `N`, `min_coverage` | Infer grammar from raw sequences. Runs CRX + iDRegEx, picks best by MDL. Set `prefer` to run only one algorithm. Set `min_coverage < 1.0` for optional core+outlier analysis. | -| `analyze_directory` | `directory`, `slice`, `min_coverage`, `prefer`, `kmax`, `include`, `exclude`, `main_only`, `max_mdl`, `persist` | Scan a source code directory and infer behavioral conventions per package. Returns YAML grouped by module. Auto-persists to `{directory}/.dervish/grammars.yml`. | +| `analyze_directory` | `directory`, `slice`, `min_coverage`, `prefer`, `kmax`, `include`, `exclude`, `main_only`, `max_mdl`, `persist`, `min_structure`, `decompose`, `max_seq_length`, `cluster_method`, `crx_method` | Scan a source code directory and infer behavioral conventions per package. Returns YAML grouped by module. Auto-persists to `{directory}/.dervish/grammars.yml`. | **`analyze_directory` parameters:** - **`directory`**: Path to source code directory to analyze. - **`slice`**: `'package'` (per-directory, default) or `'flat'` (one per language). - **`main_only`**: Exclude test files when `True`. Default `False`. - **`max_mdl`**: Drop groups with MDL above this threshold. Default 200. Lower = tighter patterns only. +- **`min_structure`**: Minimum grammar structure score (0.0–1.0). Only returns grammars at or above this threshold. Default 0.5. - **`persist`**: Write results to `{directory}/.dervish/grammars.yml`. Default `True`. +- **`decompose`**: Break long sequences into shorter fragments before inference. Helps on diverse codebases. Default `False`. +- **`max_seq_length`**: Max fragment length when decomposing. Default 5. +- **`cluster_method`**: `'first-symbol'` (fast) or `'distributional'` (context-similarity). Default `'first-symbol'`. +- **`crx_method`**: `'standard'` (fast) or `'refined'` (cluster-then-infer). Default `'standard'`. - Other parameters same as `infer_best_grammar`. **Parameters explained for `infer_best_grammar`:** @@ -130,9 +135,12 @@ For analyzing source code directories (tree-sitter based): ```bash python -m bex.tag_preprocessor.analyze /path/to/codebase --verbose python -m bex.tag_preprocessor.analyze /path/to/codebase --slice package --include '**/src/**' +python -m bex.tag_preprocessor.analyze /path/to/codebase --slice package --decompose --min-structure 0.5 ``` -Key flags: `--slice package` (per-directory grammars), `--verbose` (progress), `--include`/`--exclude` (glob filters), `--kore` (enable slow kORE in ensemble). +Key flags: `--slice package` (per-directory grammars), `--decompose` (break long sequences into fragments for better inference), `--min-structure 0.5` (return only high-structure grammars), `--cluster-method distributional` (context-similarity clustering), `--verbose` (progress), `--include`/`--exclude` (glob filters). + +The pipeline outputs SORE grammars (converted to GBNF internally for llama.cpp constrained decoding). ## Why not just use a schema? @@ -159,18 +167,20 @@ Dervish has been tested against public datasets from Ansible Galaxy, Helm, and G The sweet spot: **multiple implementations of the same abstract task** with a shared but undocumented pattern. Not everything works — Dockerfiles, pre-commit configs, and schema-enforced formats are too rigid or too diverse to yield a convention. -> **kOREInference note:** Algorithm 4 (iDRegEx with MDL, arXiv 1004.2372) is included for paper-faithful correctness. On real tool-sequence data, its rwr₀ repair step returns ∅ because the k-OA is rarely SORE (interconnected symbols). The ensemble falls back to CRX or iDRegEx automatically. +> **kOREInference note:** Algorithm 4 (iDRegEx with MDL, arXiv 1004.2372) is available via `--kore` flag for paper-faithful correctness. On real tool-sequence data, its rwr₀ repair step returns ∅ because the k-OA is rarely SORE (interconnected symbols). The ensemble falls back to CRX or iDRegEx automatically. ## Algorithm Selection Guide | When | Use | Why | |------|-----|-----| | Clean, structured data with full vocabulary | **CRX** | Single-pass, deterministic. Accepts all sequences. | -| Few examples, or want minimal common core | **iDRegEx** or **kOREInference** | Probabilistic EM, finds only what's shared. | +| Few examples, or want minimal common core | **iDRegEx** | Probabilistic EM, finds only what's shared. | | Don't know which is better | **Ensemble (default)** | Runs CRX + iDRegEx, picks best by MDL score. | | Want core pattern + outlier detection | **Ensemble + `min_coverage<1`** | Finds tight grammar for majority, flags outliers. | | Data is clearly one type | `prefer='crx'` | Skips ensemble comparison, runs CRX alone. | +> **kORE note:** kOREInference (Algorithm 4) is available via `--kore` flag but excluded from the default ensemble. On real-world data it returns ∅ in ~80% of cases because the k-OA is rarely SORE. CRX or iDRegEx handle these cases better. + ## When each algorithm wins | Data property | Winner | Why | @@ -178,7 +188,7 @@ The sweet spot: **multiple implementations of the same abstract task** with a sh | Diverse patterns, full vocabulary needed | CRX | Captures all symbols. iDRegEx returns ∅. | | Clean sequences with clear core | iDRegEx | Extracts minimal common subsequence. CRX buries it in optional noise. | | Interconnected (non-SORE) data | CRX | kOREInference (rwr₀) returns ∅ when k-OA is not SORE. CRX handles it. | -| Single sequence | iDRegEx (+ RWR₀) | RWR₀ repair produces a grammatical regex from one example. | +| Single sequence | iDRegEx | iDRegEx handles noise better. | | 2–3 sequences | iDRegEx | CRX overfits. iDRegEx handles noise better. | | Many sequences, tight pattern | CRX | Learns precise concatenation with optional suffixes. | | Want majority pattern + outlier list | CRX + `min_coverage` | Core analysis finds tight grammar for ~80%, flags the rest. | diff --git a/bex/mcp_server.py b/bex/mcp_server.py index 0626fef..da52e4c 100644 --- a/bex/mcp_server.py +++ b/bex/mcp_server.py @@ -41,9 +41,7 @@ def infer_best_grammar( than passing all examples. Pass the existing sequences, get back a pattern you can follow to generate new instances. - Runs CRX + iDRegEx, picks best by scoring. kORE is excluded by - default (slow, rarely wins on real data). Set prefer='koreinference' - to force it. + Runs CRX + iDRegEx, picks best by scoring. Args: sequences: List of sequences, each a list of strings (symbols in @@ -106,8 +104,12 @@ def analyze_directory( persist: bool = True, method: str = "langsize", min_methods: int = 3, - min_structure: float = 0.2, + min_structure: float = 0.5, split_mixed: bool = True, + decompose: bool = False, + max_seq_length: int = 5, + cluster_method: str = "first-symbol", + crx_method: str = "standard", ) -> str: """Scan a source code directory and infer behavioral conventions (regular expression grammars) per package. Returns compact patterns @@ -140,13 +142,25 @@ def analyze_directory( method: Scoring method — 'langsize' (default) or 'mdl'. min_methods: Minimum methods per group to attempt inference. Default 3. min_structure: Minimum grammar structure score (0.0–1.0). Groups - below this produce flat bags. Default 0.2. + below this produce flat bags. Default 0.5 (only returns + high-structure grammars). split_mixed: When True (default), recursively split groups with diverse first symbols into uniform sub-groups before inference. + decompose: When True, decompose long sequences into shorter + fragments before inference. Helps on codebases with many + unique method patterns. Default False. + max_seq_length: Maximum fragment length when decompose=True. Default 5. + cluster_method: How to split mixed groups — 'first-symbol' (fast, + crude) or 'distributional' (context-similarity clustering). + Default 'first-symbol'. + crx_method: CRX variant — 'standard' (fast, Algorithm 7) or + 'refined' (cluster-then-infer, tighter on flat bags). + Default 'standard'. Returns: YAML string with grammars grouped by top-level module, sorted - by score (tightest/most useful first). + by score (tightest/most useful first). Only returns grammars + meeting the min_structure threshold. """ results = _analyze_directory( directory, @@ -161,8 +175,12 @@ def analyze_directory( min_methods=min_methods, split_mixed=split_mixed, min_structure=min_structure, + decompose=decompose, + max_seq_length=max_seq_length, + cluster_method=cluster_method, + crx_method=crx_method, ) - yaml_content = _build_yaml_output(results, directory, max_mdl=max_mdl) + yaml_content = _build_yaml_output(results, directory, max_mdl=max_mdl, min_structure=min_structure) if persist: _persist_grammars(yaml_content, directory) return yaml_content diff --git a/bex/tag_preprocessor/analyze.py b/bex/tag_preprocessor/analyze.py index 5b1c467..fb5bcdd 100644 --- a/bex/tag_preprocessor/analyze.py +++ b/bex/tag_preprocessor/analyze.py @@ -883,10 +883,11 @@ def _build_json_output(results): return json.dumps(output, indent=2) -def _build_yaml_output(results, dir_path, max_mdl=200.0): +def _build_yaml_output(results, dir_path, max_mdl=200.0, min_structure=0.0): """Build YAML output grouped by top-level module, sorted by MDL. - Filters out (other), no-grammar groups, and groups above max_mdl. + Filters out (other), no-grammar groups, groups above max_mdl, + and groups below min_structure. Returns YAML string. """ import yaml @@ -908,6 +909,8 @@ def _build_yaml_output(results, dir_path, max_mdl=200.0): best = result["best"] if best["mdl_score"] > max_mdl: continue + if grammar_structure_score(best["grammar"]) < min_structure: + continue # Extract top-level module from package path parts = label.replace(os.sep, "/").split("/") @@ -917,6 +920,8 @@ def _build_yaml_output(results, dir_path, max_mdl=200.0): all_grammars = meta.get("all_grammars", []) if all_grammars: for leaf_label, leaf_grammar, leaf_score, leaf_count in all_grammars: + if grammar_structure_score(leaf_grammar) < min_structure: + continue entry = { "package": leaf_label, "methods": leaf_count, diff --git a/experiments/EXPERIMENT_LOG.md b/experiments/EXPERIMENT_LOG.md index 8ecc4f8..49e6b5f 100644 --- a/experiments/EXPERIMENT_LOG.md +++ b/experiments/EXPERIMENT_LOG.md @@ -770,3 +770,22 @@ The pipeline should: (1) run refined CRX, (2) if trivial, fall back to CRX. **Recommendation:** Make refined CRX the default for `--split-mixed` mode. Keep standard CRX as fallback. No need for iDRegEx or kORE in the pipeline. + +--- + +## Round 18: Decomposition Forest (Phase 2) + +**Goal:** Break down long sequences into shorter fragments before inference (Crucio Phase 2). + +**Method:** Implemented `bex/decompose.py` with prefix/suffix/window extraction. + +**Results:** +- RAGSAK: 21 → 80 grammars (3.8× increase) +- FastAPI: 111 → 118 grammars (small increase) + +**Key insight:** Decomposition creates diverse fragments, so skip diversity check when enabled. + +**Files:** +- `bex/decompose.py`: decompose_sequence(), decompose_all(), decompose_with_coverage() +- `bex/tag_preprocessor/analyze.py`: --decompose, --max-seq-length flags +- `tests/test_decompose.py`: 12 new tests