grammar-inference-engine/docs/plans/analyze-directory-mcp-tool.md

# Plan: `analyze_directory` MCP Tool

## Goal

Add an `analyze_directory` MCP tool that scans a source code directory, infers behavioral conventions per package, returns YAML grouped by module, and auto-persists to `{directory}/.dervish/grammars.yml`.

## Design Decisions (locked)

| Decision | Choice |
|----------|--------|
| Signature | New separate tool (not extending `infer_best_grammar`) |
| Persistence | Auto-save to `{directory}/.dervish/grammars.yml` (on by default, `persist=False` to opt out) |
| Output format | YAML grouped by top-level module, sorted by MDL |
| Heuristics | MDL threshold (default 200), drop `(other)`, drop no-grammar, optional `main_only` |
| YAML library | pyyaml (`yaml.dump()`) |

## Signature

```python
@mcp.tool()
def analyze_directory(
    directory: str,              # Path to source code directory
    slice: str = "package",      # "flat" or "package"
    min_coverage: float = 0.8,   # Outlier removal threshold
    prefer: str = "",            # Force CRX or iDRegEx
    kmax: int = 2,               # k-ORE context depth
    include: str = "",           # Glob filter (include)
    exclude: str = "",           # Glob filter (exclude)
    main_only: bool = False,     # Exclude test code paths
    max_mdl: float = 200,        # Drop groups above this MDL
    persist: bool = True,        # Auto-save to .dervish/
) -> str:
```

## Output format (YAML)

```yaml
# RAGSAK — 35 patterns (1594 methods, 462 files)

agents:
  - package: agents/rag/embabel
    methods: 52
    grammar: "info+"
    algorithm: CRX
    mdl: 28.0

modules:
  - package: modules/common/.../ids
    methods: 18
    grammar: "(of|requireSafeId)"
    algorithm: iDRegEx
    mdl: 34.0
  - package: modules/ingestion/.../batch
    methods: 16
    grammar: "info?.StepBuilder?.listener+?.build"
    algorithm: CRX
    mdl: 13.2

infrastructure:
  - package: infrastructure/.../service/cleanup
    methods: 4
    grammar: "info.(deleteByJobId+deleteByKnowledgeBaseId)"
    algorithm: CRX
    mdl: 7.0
```

## Persistence

- Write to `{directory}/.dervish/grammars.yml`
- Create `.dervish/` dir if missing
- Overwrite on each run (idempotent)
- File is committed to git alongside the code
- LLM reads it later without recomputing

## Heuristics

1. **MDL threshold** — drop groups with MDL > `max_mdl` (default 200)
2. **Drop `(other)`** — ungrouped methods, always excluded
3. **Drop no-grammar** — groups where both algorithms failed, always excluded
4. **`main_only`** — excludes test paths when enabled

Test path patterns for `main_only`:
```
**/test/**, **/tests/**, **/*Test*/**
**/*_test.*, **/*_spec.*, **/test_*.py
**/*Test.kt, **/*Test.java, **/*Test.js
**/*Spec.*, **/*_test.go
```

## Files to modify

| File | Change |
|------|--------|
| `bex/tag_preprocessor/analyze.py` | Add `_is_main_source()`, `_build_yaml_output()`, `_persist_grammars()`, wire `main_only` into `analyze_directory()` |
| `bex/mcp_server.py` | Add `analyze_directory` MCP tool |
| `README.md` | Document new tool in MCP tools table |
| `AGENTS.md` | Document new tool in MCP tools table |

## Implementation steps

### Step 1: `_is_main_source()` in `analyze.py`

```python
TEST_PATH_PATTERNS = [
    "**/test/**", "**/tests/**", "**/*Test*/**",
    "**/*_test.*", "**/*_spec.*", "**/test_*.py",
    "**/*Test.kt", "**/*Test.java", "**/*Test.js",
    "**/*Spec.*", "**/*_test.go",
]

def _is_main_source(filepath):
    """Return True if file path looks like main source (not test)."""
    for pattern in TEST_PATH_PATTERNS:
        if _match_glob(filepath, pattern):
            return False
    return True
```

### Step 2: `_build_yaml_output()` in `analyze.py`

Convert results dict to YAML grouped by top-level module:

```python
def _build_yaml_output(results, dir_path):
    """Build YAML output grouped by top-level module.
    
    Returns YAML string with grammar per package, sorted by MDL within each module.
    Skips (other) and no-grammar groups.
    """
```

Logic:
- Extract top-level module from package path (first directory component)
- Sort groups by MDL within each module
- Skip `(other)` and no-grammar groups
- Return `yaml.dump()` output

### Step 3: `_persist_grammars()` in `analyze.py`

```python
def _persist_grammars(yaml_content, dir_path):
    """Write grammars to {dir_path}/.dervish/grammars.yml"""
    dervish_dir = os.path.join(dir_path, ".dervish")
    os.makedirs(dervish_dir, exist_ok=True)
    with open(os.path.join(dervish_dir, "grammars.yml"), "w") as f:
        f.write(yaml_content)
```

### Step 4: Wire `analyze_directory()` in `analyze.py`

- Add `main_only` parameter
- When `main_only=True`, filter files through `_is_main_source()` before processing
- Apply MDL threshold filtering after inference
- Return filtered results

### Step 5: Add MCP tool in `mcp_server.py`

```python
@mcp.tool()
def analyze_directory(...) -> str:
```

Flow:
1. Call `analyze_directory()` from `tag_preprocessor.analyze`
2. Filter by `max_mdl`, drop `(other)`, drop no-grammar
3. Build YAML via `_build_yaml_output()`
4. If `persist=True`, call `_persist_grammars()`
5. Return YAML string

### Step 6: Update docs

- README.md: Add `analyze_directory` to MCP tools table
- AGENTS.md: Add `analyze_directory` to MCP tools table

## Verification

1. Run: `python -m bex.mcp_server` (starts MCP server)
2. Call `analyze_directory(directory="/home/tobi/Desktop/kesai/RAGSAK")`
3. Verify YAML output is grouped by module, sorted by MDL
4. Verify `.dervish/grammars.yml` created in RAGSAK
5. Run existing tests: `python -m pytest tests/`
6. Run: `python -m bex.tag_preprocessor.analyze /home/tobi/Desktop/kesai/RAGSAK --slice package --verbose` (verify no regression)