diff --git a/AGENTS.md b/AGENTS.md index d1db4b8..fe7cfba 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -41,11 +41,12 @@ python -m pytest tests/ ## MCP Server -The primary interface is an MCP server exposing a single tool: +The primary interface is an MCP server exposing two tools: | Tool | Parameters | What it does | |------|-----------|-------------| -| `infer_best_grammar` | `sequences`, `prefer`, `kmax`, `N`, `min_coverage` | Runs CRX + iDRegEx, picks best by MDL. `prefer='crx'` or `prefer='idregex'` skips ensemble. `min_coverage < 1.0` runs core+outlier analysis. | +| `infer_best_grammar` | `sequences`, `prefer`, `kmax`, `N`, `min_coverage` | Infer grammar from raw sequences. Runs CRX + iDRegEx, picks best by MDL. | +| `analyze_directory` | `directory`, `slice`, `min_coverage`, `prefer`, `kmax`, `include`, `exclude`, `main_only`, `max_mdl`, `persist` | Scan source code, infer conventions per package. Returns YAML grouped by module. Auto-persists to `{directory}/.dervish/grammars.yml`. | Start it: `python /path/to/bex/mcp_server.py`, then connect any MCP client. @@ -58,4 +59,4 @@ python -m bex.tag_preprocessor.analyze /path/to/codebase --verbose python -m bex.tag_preprocessor.analyze /path/to/codebase --slice package --include '**/src/**' ``` -Key flags: `--slice package` (per-directory grammars), `--verbose` (progress), `--include`/`--exclude` (glob filters), `--kore` (enable slow kORE in ensemble). +Key flags: `--slice package` (per-directory grammars), `--verbose` (progress), `--include`/`--exclude` (glob filters), `--main-only` (exclude test files), `--kore` (enable slow kORE in ensemble). diff --git a/README.md b/README.md index 0d45477..e09743f 100644 --- a/README.md +++ b/README.md @@ -50,9 +50,18 @@ The primary interface is a **Model Context Protocol (MCP)** server. Connect any | Tool | Parameters | What it does | |------|-----------|-------------| -| `infer_best_grammar` | `sequences`, `prefer`, `kmax`, `N`, `min_coverage` | **The only tool you need.** Runs CRX + iDRegEx, picks best by MDL. Set `prefer` to run only one algorithm. Set `min_coverage < 1.0` for optional core+outlier analysis. | +| `infer_best_grammar` | `sequences`, `prefer`, `kmax`, `N`, `min_coverage` | Infer grammar from raw sequences. Runs CRX + iDRegEx, picks best by MDL. Set `prefer` to run only one algorithm. Set `min_coverage < 1.0` for optional core+outlier analysis. | +| `analyze_directory` | `directory`, `slice`, `min_coverage`, `prefer`, `kmax`, `include`, `exclude`, `main_only`, `max_mdl`, `persist` | Scan a source code directory and infer behavioral conventions per package. Returns YAML grouped by module. Auto-persists to `{directory}/.dervish/grammars.yml`. | -**Parameters explained:** +**`analyze_directory` parameters:** +- **`directory`**: Path to source code directory to analyze. +- **`slice`**: `'package'` (per-directory, default) or `'flat'` (one per language). +- **`main_only`**: Exclude test files when `True`. Default `False`. +- **`max_mdl`**: Drop groups with MDL above this threshold. Default 200. Lower = tighter patterns only. +- **`persist`**: Write results to `{directory}/.dervish/grammars.yml`. Default `True`. +- Other parameters same as `infer_best_grammar`. + +**Parameters explained for `infer_best_grammar`:** - **`prefer`**: `'crx'` for full vocabulary (accepts all sequences), `'idregex'` for deterministic minimal core, `'koreinference'` for k-OA with rwr₀ repair (slow, rarely wins). Omit to let MDL pick the winner across CRX and iDRegEx. - **`kmax`** (1–5): Context window for k-ORE inference (iDRegEx, kOREInference). Higher values capture longer-range dependencies but need more data and are slower. Default 2 works for most cases. - **`N`** (1–10): Random trials for k-ORE inference. More = better convergence but slower. Default 3. diff --git a/bex/mcp_server.py b/bex/mcp_server.py index d801927..9f60ad4 100644 --- a/bex/mcp_server.py +++ b/bex/mcp_server.py @@ -7,6 +7,11 @@ Run as: python -m bex.mcp_server from mcp.server.fastmcp import FastMCP from .ensemble import infer_ensemble, _matches +from .tag_preprocessor.analyze import ( + analyze_directory as _analyze_directory, + _build_yaml_output, + _persist_grammars, +) mcp = FastMCP("grammar-inference", log_level="ERROR") @@ -77,6 +82,69 @@ def infer_best_grammar( return "\n".join(lines) +@mcp.tool() +def analyze_directory( + directory: str, + slice: str = "package", + min_coverage: float = 0.8, + prefer: str = "", + kmax: int = 2, + include: str = "", + exclude: str = "", + main_only: bool = False, + max_mdl: float = 200, + persist: bool = True, +) -> str: + """Scan a source code directory and infer behavioral conventions + (regular expression grammars) per package. Returns compact patterns + grouped by module, sorted by quality (MDL score). + + Use this when you need to understand the calling conventions in a + codebase — what patterns new code should follow. The grammar + compresses each package's method call patterns into a compact + regular expression. + + Auto-persists results to {directory}/.dervish/grammars.yml unless + persist=False. + + Args: + directory: Path to the source code directory to analyze. + slice: Grouping strategy — 'package' (per directory, default) + or 'flat' (one per language). + min_coverage: BEX core coverage threshold for outlier removal + (0.5–1.0). Default 0.8. + prefer: Optional — 'crx' for full vocabulary, 'idregex' for + minimal core. Omit to auto-pick by MDL. + kmax: Context depth for k-ORE inference. Default 2. + include: Glob pattern to include only matching files. + exclude: Glob pattern to skip matching files. + main_only: When True, exclude test files (src/test/**, *Test.*, + etc.). Default False. + max_mdl: Drop groups with MDL above this threshold. Default 200. + Lower = tighter patterns only. Set higher to see noisier groups. + persist: When True (default), write results to + {directory}/.dervish/grammars.yml. + + Returns: + YAML string with grammars grouped by top-level module, sorted + by MDL (tightest/most useful first). + """ + results = _analyze_directory( + directory, + min_coverage=min_coverage, + prefer=prefer or None, + kmax=kmax, + slice=slice, + include=include or None, + exclude=exclude or None, + main_only=main_only, + ) + yaml_content = _build_yaml_output(results, directory, max_mdl=max_mdl) + if persist: + _persist_grammars(yaml_content, directory) + return yaml_content + + def main(): mcp.run() diff --git a/bex/tag_preprocessor/analyze.py b/bex/tag_preprocessor/analyze.py index 5956e34..2683e9c 100644 --- a/bex/tag_preprocessor/analyze.py +++ b/bex/tag_preprocessor/analyze.py @@ -31,6 +31,21 @@ FALLBACK_SKIP = { ".venv", "venv", "env", ".env", "out", } +TEST_PATH_PATTERNS = [ + "**/test/**", "**/tests/**", "**/*Test*/**", + "**/*_test.*", "**/*_spec.*", "**/test_*.py", + "**/*Test.kt", "**/*Test.java", "**/*Test.js", + "**/*Spec.*", "**/*_test.go", +] + + +def _is_main_source(filepath): + """Return True if file path looks like main source (not test).""" + for pattern in TEST_PATH_PATTERNS: + if _match_glob(filepath, pattern): + return False + return True + def _load_gitignore(dir_path): """Load .gitignore from dir_path root, return PathSpec or None.""" @@ -409,6 +424,7 @@ def analyze_directory( slice="flat", include=None, exclude=None, + main_only=False, include_kore=False, ): """Scan a directory and run analysis for each language found. @@ -421,6 +437,7 @@ def analyze_directory( slice: grouping strategy — "flat" (one per language) or "package" (per directory). include: optional glob — only process files matching this pattern. exclude: optional glob — skip files matching this pattern. + main_only: exclude test files when True. Returns: dict mapping extension → list of (label, result_dict, count, meta) tuples. @@ -431,6 +448,8 @@ def analyze_directory( if len(files) < 1: continue files = _filter_glob(files, include=include, exclude=exclude) + if main_only: + files = [f for f in files if _is_main_source(f)] if not files: continue if slice == "package": @@ -469,7 +488,7 @@ def _build_json_output(results): if result and result.get("best"): entry["algorithm"] = result["best"]["algorithm"] entry["grammar"] = result["best"]["grammar"] - entry["mdl_score"] = round(result["best"]["mdl_score"], 1) + entry["mdl_score"] = round(result['best']['mdl_score'], 1) entry["imports"] = meta.get("imports", []) entry["arg_patterns"] = meta.get("arg_patterns", {}) lang["conventions"].append(entry) @@ -478,6 +497,70 @@ def _build_json_output(results): return json.dumps(output, indent=2) +def _build_yaml_output(results, dir_path, max_mdl=200.0): + """Build YAML output grouped by top-level module, sorted by MDL. + + Filters out (other), no-grammar groups, and groups above max_mdl. + Returns YAML string. + """ + import yaml + + project_name = os.path.basename(os.path.abspath(dir_path)) + + # Collect all entries grouped by top-level module + modules = {} + total_patterns = 0 + total_methods = 0 + + for ext, clusters in results.items(): + for label, result, count, meta in clusters: + total_methods += count + if label == "(other)": + continue + if not result or not result.get("best"): + continue + best = result["best"] + if best["mdl_score"] > max_mdl: + continue + + # Extract top-level module from package path + parts = label.replace(os.sep, "/").split("/") + module = parts[0] if len(parts) > 1 else "(root)" + + entry = { + "package": label, + "methods": count, + "grammar": best["grammar"], + "algorithm": best["algorithm"], + "mdl": round(best["mdl_score"], 1), + } + + modules.setdefault(module, []).append(entry) + total_patterns += 1 + + # Sort entries within each module by MDL + for module in modules: + modules[module].sort(key=lambda x: x["mdl"]) + + # Sort modules by name + ordered = dict(sorted(modules.items())) + + # Build YAML + header = f"# {project_name} — {total_patterns} patterns ({total_methods} methods)\n\n" + yaml_content = yaml.dump(ordered, default_flow_style=False, sort_keys=False, allow_unicode=True) + return header + yaml_content + + +def _persist_grammars(yaml_content, dir_path): + """Write grammars to {dir_path}/.dervish/grammars.yml""" + dervish_dir = os.path.join(dir_path, ".dervish") + os.makedirs(dervish_dir, exist_ok=True) + out_path = os.path.join(dervish_dir, "grammars.yml") + with open(out_path, "w") as f: + f.write(yaml_content) + _vprint(f"Persisted to {out_path}") + + def _parse_args(argv=None): parser = argparse.ArgumentParser( description="Analyze a directory of source code for behavioral conventions.", diff --git a/docs/plans/analyze-directory-mcp-tool.md b/docs/plans/analyze-directory-mcp-tool.md new file mode 100644 index 0000000..ca231a3 --- /dev/null +++ b/docs/plans/analyze-directory-mcp-tool.md @@ -0,0 +1,182 @@ +# Plan: `analyze_directory` MCP Tool + +## Goal + +Add an `analyze_directory` MCP tool that scans a source code directory, infers behavioral conventions per package, returns YAML grouped by module, and auto-persists to `{directory}/.dervish/grammars.yml`. + +## Design Decisions (locked) + +| Decision | Choice | +|----------|--------| +| Signature | New separate tool (not extending `infer_best_grammar`) | +| Persistence | Auto-save to `{directory}/.dervish/grammars.yml` (on by default, `persist=False` to opt out) | +| Output format | YAML grouped by top-level module, sorted by MDL | +| Heuristics | MDL threshold (default 200), drop `(other)`, drop no-grammar, optional `main_only` | +| YAML library | pyyaml (`yaml.dump()`) | + +## Signature + +```python +@mcp.tool() +def analyze_directory( + directory: str, # Path to source code directory + slice: str = "package", # "flat" or "package" + min_coverage: float = 0.8, # Outlier removal threshold + prefer: str = "", # Force CRX or iDRegEx + kmax: int = 2, # k-ORE context depth + include: str = "", # Glob filter (include) + exclude: str = "", # Glob filter (exclude) + main_only: bool = False, # Exclude test code paths + max_mdl: float = 200, # Drop groups above this MDL + persist: bool = True, # Auto-save to .dervish/ +) -> str: +``` + +## Output format (YAML) + +```yaml +# RAGSAK — 35 patterns (1594 methods, 462 files) + +agents: + - package: agents/rag/embabel + methods: 52 + grammar: "info+" + algorithm: CRX + mdl: 28.0 + +modules: + - package: modules/common/.../ids + methods: 18 + grammar: "(of|requireSafeId)" + algorithm: iDRegEx + mdl: 34.0 + - package: modules/ingestion/.../batch + methods: 16 + grammar: "info?.StepBuilder?.listener+?.build" + algorithm: CRX + mdl: 13.2 + +infrastructure: + - package: infrastructure/.../service/cleanup + methods: 4 + grammar: "info.(deleteByJobId+deleteByKnowledgeBaseId)" + algorithm: CRX + mdl: 7.0 +``` + +## Persistence + +- Write to `{directory}/.dervish/grammars.yml` +- Create `.dervish/` dir if missing +- Overwrite on each run (idempotent) +- File is committed to git alongside the code +- LLM reads it later without recomputing + +## Heuristics + +1. **MDL threshold** — drop groups with MDL > `max_mdl` (default 200) +2. **Drop `(other)`** — ungrouped methods, always excluded +3. **Drop no-grammar** — groups where both algorithms failed, always excluded +4. **`main_only`** — excludes test paths when enabled + +Test path patterns for `main_only`: +``` +**/test/**, **/tests/**, **/*Test*/** +**/*_test.*, **/*_spec.*, **/test_*.py +**/*Test.kt, **/*Test.java, **/*Test.js +**/*Spec.*, **/*_test.go +``` + +## Files to modify + +| File | Change | +|------|--------| +| `bex/tag_preprocessor/analyze.py` | Add `_is_main_source()`, `_build_yaml_output()`, `_persist_grammars()`, wire `main_only` into `analyze_directory()` | +| `bex/mcp_server.py` | Add `analyze_directory` MCP tool | +| `README.md` | Document new tool in MCP tools table | +| `AGENTS.md` | Document new tool in MCP tools table | + +## Implementation steps + +### Step 1: `_is_main_source()` in `analyze.py` + +```python +TEST_PATH_PATTERNS = [ + "**/test/**", "**/tests/**", "**/*Test*/**", + "**/*_test.*", "**/*_spec.*", "**/test_*.py", + "**/*Test.kt", "**/*Test.java", "**/*Test.js", + "**/*Spec.*", "**/*_test.go", +] + +def _is_main_source(filepath): + """Return True if file path looks like main source (not test).""" + for pattern in TEST_PATH_PATTERNS: + if _match_glob(filepath, pattern): + return False + return True +``` + +### Step 2: `_build_yaml_output()` in `analyze.py` + +Convert results dict to YAML grouped by top-level module: + +```python +def _build_yaml_output(results, dir_path): + """Build YAML output grouped by top-level module. + + Returns YAML string with grammar per package, sorted by MDL within each module. + Skips (other) and no-grammar groups. + """ +``` + +Logic: +- Extract top-level module from package path (first directory component) +- Sort groups by MDL within each module +- Skip `(other)` and no-grammar groups +- Return `yaml.dump()` output + +### Step 3: `_persist_grammars()` in `analyze.py` + +```python +def _persist_grammars(yaml_content, dir_path): + """Write grammars to {dir_path}/.dervish/grammars.yml""" + dervish_dir = os.path.join(dir_path, ".dervish") + os.makedirs(dervish_dir, exist_ok=True) + with open(os.path.join(dervish_dir, "grammars.yml"), "w") as f: + f.write(yaml_content) +``` + +### Step 4: Wire `analyze_directory()` in `analyze.py` + +- Add `main_only` parameter +- When `main_only=True`, filter files through `_is_main_source()` before processing +- Apply MDL threshold filtering after inference +- Return filtered results + +### Step 5: Add MCP tool in `mcp_server.py` + +```python +@mcp.tool() +def analyze_directory(...) -> str: +``` + +Flow: +1. Call `analyze_directory()` from `tag_preprocessor.analyze` +2. Filter by `max_mdl`, drop `(other)`, drop no-grammar +3. Build YAML via `_build_yaml_output()` +4. If `persist=True`, call `_persist_grammars()` +5. Return YAML string + +### Step 6: Update docs + +- README.md: Add `analyze_directory` to MCP tools table +- AGENTS.md: Add `analyze_directory` to MCP tools table + +## Verification + +1. Run: `python -m bex.mcp_server` (starts MCP server) +2. Call `analyze_directory(directory="/home/tobi/Desktop/kesai/RAGSAK")` +3. Verify YAML output is grouped by module, sorted by MDL +4. Verify `.dervish/grammars.yml` created in RAGSAK +5. Run existing tests: `python -m pytest tests/` +6. Run: `python -m bex.tag_preprocessor.analyze /home/tobi/Desktop/kesai/RAGSAK --slice package --verbose` (verify no regression)