MCP server: add decomposition + quality filtering params
- analyze_directory tool: add decompose, max_seq_length, cluster_method, crx_method params for Crucio-inspired improvements - min_structure default bumped to 0.5 for MCP (only high-structure grammars returned to agents) - _build_yaml_output: filter entries below min_structure threshold - README: update source code analysis section, fix kORE/iDRegEx mentions - 269 tests pass Co-authored-by: OpenCode <opencode@corentic.eu>
This commit is contained in:
tobjend
parent
8b2899d16e
commit
9ca56e2c69
4 changed files with 66 additions and 14 deletions
20
README.md
20
README.md
|
|
@ -51,14 +51,19 @@ The primary interface is a **Model Context Protocol (MCP)** server. Connect any
|
|||
| Tool | Parameters | What it does |
|
||||
|------|-----------|-------------|
|
||||
| `infer_best_grammar` | `sequences`, `prefer`, `kmax`, `N`, `min_coverage` | Infer grammar from raw sequences. Runs CRX + iDRegEx, picks best by MDL. Set `prefer` to run only one algorithm. Set `min_coverage < 1.0` for optional core+outlier analysis. |
|
||||
| `analyze_directory` | `directory`, `slice`, `min_coverage`, `prefer`, `kmax`, `include`, `exclude`, `main_only`, `max_mdl`, `persist` | Scan a source code directory and infer behavioral conventions per package. Returns YAML grouped by module. Auto-persists to `{directory}/.dervish/grammars.yml`. |
|
||||
| `analyze_directory` | `directory`, `slice`, `min_coverage`, `prefer`, `kmax`, `include`, `exclude`, `main_only`, `max_mdl`, `persist`, `min_structure`, `decompose`, `max_seq_length`, `cluster_method`, `crx_method` | Scan a source code directory and infer behavioral conventions per package. Returns YAML grouped by module. Auto-persists to `{directory}/.dervish/grammars.yml`. |
|
||||
|
||||
**`analyze_directory` parameters:**
|
||||
- **`directory`**: Path to source code directory to analyze.
|
||||
- **`slice`**: `'package'` (per-directory, default) or `'flat'` (one per language).
|
||||
- **`main_only`**: Exclude test files when `True`. Default `False`.
|
||||
- **`max_mdl`**: Drop groups with MDL above this threshold. Default 200. Lower = tighter patterns only.
|
||||
- **`min_structure`**: Minimum grammar structure score (0.0–1.0). Only returns grammars at or above this threshold. Default 0.5.
|
||||
- **`persist`**: Write results to `{directory}/.dervish/grammars.yml`. Default `True`.
|
||||
- **`decompose`**: Break long sequences into shorter fragments before inference. Helps on diverse codebases. Default `False`.
|
||||
- **`max_seq_length`**: Max fragment length when decomposing. Default 5.
|
||||
- **`cluster_method`**: `'first-symbol'` (fast) or `'distributional'` (context-similarity). Default `'first-symbol'`.
|
||||
- **`crx_method`**: `'standard'` (fast) or `'refined'` (cluster-then-infer). Default `'standard'`.
|
||||
- Other parameters same as `infer_best_grammar`.
|
||||
|
||||
**Parameters explained for `infer_best_grammar`:**
|
||||
|
|
@ -130,9 +135,12 @@ For analyzing source code directories (tree-sitter based):
|
|||
```bash
|
||||
python -m bex.tag_preprocessor.analyze /path/to/codebase --verbose
|
||||
python -m bex.tag_preprocessor.analyze /path/to/codebase --slice package --include '**/src/**'
|
||||
python -m bex.tag_preprocessor.analyze /path/to/codebase --slice package --decompose --min-structure 0.5
|
||||
```
|
||||
|
||||
Key flags: `--slice package` (per-directory grammars), `--verbose` (progress), `--include`/`--exclude` (glob filters), `--kore` (enable slow kORE in ensemble).
|
||||
Key flags: `--slice package` (per-directory grammars), `--decompose` (break long sequences into fragments for better inference), `--min-structure 0.5` (return only high-structure grammars), `--cluster-method distributional` (context-similarity clustering), `--verbose` (progress), `--include`/`--exclude` (glob filters).
|
||||
|
||||
The pipeline outputs SORE grammars (converted to GBNF internally for llama.cpp constrained decoding).
|
||||
|
||||
## Why not just use a schema?
|
||||
|
||||
|
|
@ -159,18 +167,20 @@ Dervish has been tested against public datasets from Ansible Galaxy, Helm, and G
|
|||
|
||||
The sweet spot: **multiple implementations of the same abstract task** with a shared but undocumented pattern. Not everything works — Dockerfiles, pre-commit configs, and schema-enforced formats are too rigid or too diverse to yield a convention.
|
||||
|
||||
> **kOREInference note:** Algorithm 4 (iDRegEx with MDL, arXiv 1004.2372) is included for paper-faithful correctness. On real tool-sequence data, its rwr₀ repair step returns ∅ because the k-OA is rarely SORE (interconnected symbols). The ensemble falls back to CRX or iDRegEx automatically.
|
||||
> **kOREInference note:** Algorithm 4 (iDRegEx with MDL, arXiv 1004.2372) is available via `--kore` flag for paper-faithful correctness. On real tool-sequence data, its rwr₀ repair step returns ∅ because the k-OA is rarely SORE (interconnected symbols). The ensemble falls back to CRX or iDRegEx automatically.
|
||||
|
||||
## Algorithm Selection Guide
|
||||
|
||||
| When | Use | Why |
|
||||
|------|-----|-----|
|
||||
| Clean, structured data with full vocabulary | **CRX** | Single-pass, deterministic. Accepts all sequences. |
|
||||
| Few examples, or want minimal common core | **iDRegEx** or **kOREInference** | Probabilistic EM, finds only what's shared. |
|
||||
| Few examples, or want minimal common core | **iDRegEx** | Probabilistic EM, finds only what's shared. |
|
||||
| Don't know which is better | **Ensemble (default)** | Runs CRX + iDRegEx, picks best by MDL score. |
|
||||
| Want core pattern + outlier detection | **Ensemble + `min_coverage<1`** | Finds tight grammar for majority, flags outliers. |
|
||||
| Data is clearly one type | `prefer='crx'` | Skips ensemble comparison, runs CRX alone. |
|
||||
|
||||
> **kORE note:** kOREInference (Algorithm 4) is available via `--kore` flag but excluded from the default ensemble. On real-world data it returns ∅ in ~80% of cases because the k-OA is rarely SORE. CRX or iDRegEx handle these cases better.
|
||||
|
||||
## When each algorithm wins
|
||||
|
||||
| Data property | Winner | Why |
|
||||
|
|
@ -178,7 +188,7 @@ The sweet spot: **multiple implementations of the same abstract task** with a sh
|
|||
| Diverse patterns, full vocabulary needed | CRX | Captures all symbols. iDRegEx returns ∅. |
|
||||
| Clean sequences with clear core | iDRegEx | Extracts minimal common subsequence. CRX buries it in optional noise. |
|
||||
| Interconnected (non-SORE) data | CRX | kOREInference (rwr₀) returns ∅ when k-OA is not SORE. CRX handles it. |
|
||||
| Single sequence | iDRegEx (+ RWR₀) | RWR₀ repair produces a grammatical regex from one example. |
|
||||
| Single sequence | iDRegEx | iDRegEx handles noise better. |
|
||||
| 2–3 sequences | iDRegEx | CRX overfits. iDRegEx handles noise better. |
|
||||
| Many sequences, tight pattern | CRX | Learns precise concatenation with optional suffixes. |
|
||||
| Want majority pattern + outlier list | CRX + `min_coverage` | Core analysis finds tight grammar for ~80%, flags the rest. |
|
||||
|
|
|
|||
|
|
@ -41,9 +41,7 @@ def infer_best_grammar(
|
|||
than passing all examples. Pass the existing sequences, get back a
|
||||
pattern you can follow to generate new instances.
|
||||
|
||||
Runs CRX + iDRegEx, picks best by scoring. kORE is excluded by
|
||||
default (slow, rarely wins on real data). Set prefer='koreinference'
|
||||
to force it.
|
||||
Runs CRX + iDRegEx, picks best by scoring.
|
||||
|
||||
Args:
|
||||
sequences: List of sequences, each a list of strings (symbols in
|
||||
|
|
@ -106,8 +104,12 @@ def analyze_directory(
|
|||
persist: bool = True,
|
||||
method: str = "langsize",
|
||||
min_methods: int = 3,
|
||||
min_structure: float = 0.2,
|
||||
min_structure: float = 0.5,
|
||||
split_mixed: bool = True,
|
||||
decompose: bool = False,
|
||||
max_seq_length: int = 5,
|
||||
cluster_method: str = "first-symbol",
|
||||
crx_method: str = "standard",
|
||||
) -> str:
|
||||
"""Scan a source code directory and infer behavioral conventions
|
||||
(regular expression grammars) per package. Returns compact patterns
|
||||
|
|
@ -140,13 +142,25 @@ def analyze_directory(
|
|||
method: Scoring method — 'langsize' (default) or 'mdl'.
|
||||
min_methods: Minimum methods per group to attempt inference. Default 3.
|
||||
min_structure: Minimum grammar structure score (0.0–1.0). Groups
|
||||
below this produce flat bags. Default 0.2.
|
||||
below this produce flat bags. Default 0.5 (only returns
|
||||
high-structure grammars).
|
||||
split_mixed: When True (default), recursively split groups with
|
||||
diverse first symbols into uniform sub-groups before inference.
|
||||
decompose: When True, decompose long sequences into shorter
|
||||
fragments before inference. Helps on codebases with many
|
||||
unique method patterns. Default False.
|
||||
max_seq_length: Maximum fragment length when decompose=True. Default 5.
|
||||
cluster_method: How to split mixed groups — 'first-symbol' (fast,
|
||||
crude) or 'distributional' (context-similarity clustering).
|
||||
Default 'first-symbol'.
|
||||
crx_method: CRX variant — 'standard' (fast, Algorithm 7) or
|
||||
'refined' (cluster-then-infer, tighter on flat bags).
|
||||
Default 'standard'.
|
||||
|
||||
Returns:
|
||||
YAML string with grammars grouped by top-level module, sorted
|
||||
by score (tightest/most useful first).
|
||||
by score (tightest/most useful first). Only returns grammars
|
||||
meeting the min_structure threshold.
|
||||
"""
|
||||
results = _analyze_directory(
|
||||
directory,
|
||||
|
|
@ -161,8 +175,12 @@ def analyze_directory(
|
|||
min_methods=min_methods,
|
||||
split_mixed=split_mixed,
|
||||
min_structure=min_structure,
|
||||
decompose=decompose,
|
||||
max_seq_length=max_seq_length,
|
||||
cluster_method=cluster_method,
|
||||
crx_method=crx_method,
|
||||
)
|
||||
yaml_content = _build_yaml_output(results, directory, max_mdl=max_mdl)
|
||||
yaml_content = _build_yaml_output(results, directory, max_mdl=max_mdl, min_structure=min_structure)
|
||||
if persist:
|
||||
_persist_grammars(yaml_content, directory)
|
||||
return yaml_content
|
||||
|
|
|
|||
|
|
@ -883,10 +883,11 @@ def _build_json_output(results):
|
|||
return json.dumps(output, indent=2)
|
||||
|
||||
|
||||
def _build_yaml_output(results, dir_path, max_mdl=200.0):
|
||||
def _build_yaml_output(results, dir_path, max_mdl=200.0, min_structure=0.0):
|
||||
"""Build YAML output grouped by top-level module, sorted by MDL.
|
||||
|
||||
Filters out (other), no-grammar groups, and groups above max_mdl.
|
||||
Filters out (other), no-grammar groups, groups above max_mdl,
|
||||
and groups below min_structure.
|
||||
Returns YAML string.
|
||||
"""
|
||||
import yaml
|
||||
|
|
@ -908,6 +909,8 @@ def _build_yaml_output(results, dir_path, max_mdl=200.0):
|
|||
best = result["best"]
|
||||
if best["mdl_score"] > max_mdl:
|
||||
continue
|
||||
if grammar_structure_score(best["grammar"]) < min_structure:
|
||||
continue
|
||||
|
||||
# Extract top-level module from package path
|
||||
parts = label.replace(os.sep, "/").split("/")
|
||||
|
|
@ -917,6 +920,8 @@ def _build_yaml_output(results, dir_path, max_mdl=200.0):
|
|||
all_grammars = meta.get("all_grammars", [])
|
||||
if all_grammars:
|
||||
for leaf_label, leaf_grammar, leaf_score, leaf_count in all_grammars:
|
||||
if grammar_structure_score(leaf_grammar) < min_structure:
|
||||
continue
|
||||
entry = {
|
||||
"package": leaf_label,
|
||||
"methods": leaf_count,
|
||||
|
|
|
|||
|
|
@ -770,3 +770,22 @@ The pipeline should: (1) run refined CRX, (2) if trivial, fall back to CRX.
|
|||
|
||||
**Recommendation:** Make refined CRX the default for `--split-mixed` mode.
|
||||
Keep standard CRX as fallback. No need for iDRegEx or kORE in the pipeline.
|
||||
|
||||
---
|
||||
|
||||
## Round 18: Decomposition Forest (Phase 2)
|
||||
|
||||
**Goal:** Break down long sequences into shorter fragments before inference (Crucio Phase 2).
|
||||
|
||||
**Method:** Implemented `bex/decompose.py` with prefix/suffix/window extraction.
|
||||
|
||||
**Results:**
|
||||
- RAGSAK: 21 → 80 grammars (3.8× increase)
|
||||
- FastAPI: 111 → 118 grammars (small increase)
|
||||
|
||||
**Key insight:** Decomposition creates diverse fragments, so skip diversity check when enabled.
|
||||
|
||||
**Files:**
|
||||
- `bex/decompose.py`: decompose_sequence(), decompose_all(), decompose_with_coverage()
|
||||
- `bex/tag_preprocessor/analyze.py`: --decompose, --max-seq-length flags
|
||||
- `tests/test_decompose.py`: 12 new tests
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue