grammar-inference-engine/docs/plans/analyze-directory-mcp-tool.md

Plan: analyze_directory MCP Tool

Goal

Add an analyze_directory MCP tool that scans a source code directory, infers behavioral conventions per package, returns YAML grouped by module, and auto-persists to {directory}/.dervish/grammars.yml.

Design Decisions (locked)

Decision	Choice
Signature	New separate tool (not extending infer_best_grammar)
Persistence	Auto-save to {directory}/.dervish/grammars.yml (on by default, persist=False to opt out)
Output format	YAML grouped by top-level module, sorted by MDL
Heuristics	MDL threshold (default 200), drop (other), drop no-grammar, optional main_only
YAML library	pyyaml (yaml.dump())

Signature

@mcp.tool()
def analyze_directory(
    directory: str,              # Path to source code directory
    slice: str = "package",      # "flat" or "package"
    min_coverage: float = 0.8,   # Outlier removal threshold
    prefer: str = "",            # Force CRX or iDRegEx
    kmax: int = 2,               # k-ORE context depth
    include: str = "",           # Glob filter (include)
    exclude: str = "",           # Glob filter (exclude)
    main_only: bool = False,     # Exclude test code paths
    max_mdl: float = 200,        # Drop groups above this MDL
    persist: bool = True,        # Auto-save to .dervish/
) -> str:

Output format (YAML)

# RAGSAK — 35 patterns (1594 methods, 462 files)

agents:
  - package: agents/rag/embabel
    methods: 52
    grammar: "info+"
    algorithm: CRX
    mdl: 28.0

modules:
  - package: modules/common/.../ids
    methods: 18
    grammar: "(of|requireSafeId)"
    algorithm: iDRegEx
    mdl: 34.0
  - package: modules/ingestion/.../batch
    methods: 16
    grammar: "info?.StepBuilder?.listener+?.build"
    algorithm: CRX
    mdl: 13.2

infrastructure:
  - package: infrastructure/.../service/cleanup
    methods: 4
    grammar: "info.(deleteByJobId+deleteByKnowledgeBaseId)"
    algorithm: CRX
    mdl: 7.0

Persistence

• Write to {directory}/.dervish/grammars.yml
• Create .dervish/ dir if missing
• Overwrite on each run (idempotent)
• File is committed to git alongside the code
• LLM reads it later without recomputing

Heuristics

1. MDL threshold — drop groups with MDL > max_mdl (default 200)
2. Drop (other) — ungrouped methods, always excluded
3. Drop no-grammar — groups where both algorithms failed, always excluded
4. main_only — excludes test paths when enabled

Test path patterns for main_only:

**/test/**, **/tests/**, **/*Test*/**
**/*_test.*, **/*_spec.*, **/test_*.py
**/*Test.kt, **/*Test.java, **/*Test.js
**/*Spec.*, **/*_test.go

Files to modify

File	Change
bex/tag_preprocessor/analyze.py	Add _is_main_source(), _build_yaml_output(), _persist_grammars(), wire main_only into analyze_directory()
bex/mcp_server.py	Add analyze_directory MCP tool
README.md	Document new tool in MCP tools table
AGENTS.md	Document new tool in MCP tools table

Implementation steps

Step 1: _is_main_source() in analyze.py

TEST_PATH_PATTERNS = [
    "**/test/**", "**/tests/**", "**/*Test*/**",
    "**/*_test.*", "**/*_spec.*", "**/test_*.py",
    "**/*Test.kt", "**/*Test.java", "**/*Test.js",
    "**/*Spec.*", "**/*_test.go",
]

def _is_main_source(filepath):
    """Return True if file path looks like main source (not test)."""
    for pattern in TEST_PATH_PATTERNS:
        if _match_glob(filepath, pattern):
            return False
    return True

Step 2: _build_yaml_output() in analyze.py

Convert results dict to YAML grouped by top-level module:

def _build_yaml_output(results, dir_path):
    """Build YAML output grouped by top-level module.
    
    Returns YAML string with grammar per package, sorted by MDL within each module.
    Skips (other) and no-grammar groups.
    """

Logic:

• Extract top-level module from package path (first directory component)
• Sort groups by MDL within each module
• Skip (other) and no-grammar groups
• Return yaml.dump() output

Step 3: _persist_grammars() in analyze.py

def _persist_grammars(yaml_content, dir_path):
    """Write grammars to {dir_path}/.dervish/grammars.yml"""
    dervish_dir = os.path.join(dir_path, ".dervish")
    os.makedirs(dervish_dir, exist_ok=True)
    with open(os.path.join(dervish_dir, "grammars.yml"), "w") as f:
        f.write(yaml_content)

Step 4: Wire analyze_directory() in analyze.py

• Add main_only parameter
• When main_only=True, filter files through _is_main_source() before processing
• Apply MDL threshold filtering after inference
• Return filtered results

Step 5: Add MCP tool in mcp_server.py

@mcp.tool()
def analyze_directory(...) -> str:

Flow:

1. Call analyze_directory() from tag_preprocessor.analyze
2. Filter by max_mdl, drop (other), drop no-grammar
3. Build YAML via _build_yaml_output()
4. If persist=True, call _persist_grammars()
5. Return YAML string

Step 6: Update docs

• README.md: Add analyze_directory to MCP tools table
• AGENTS.md: Add analyze_directory to MCP tools table

Verification

1. Run: python -m bex.mcp_server (starts MCP server)
2. Call analyze_directory(directory="/home/tobi/Desktop/kesai/RAGSAK")
3. Verify YAML output is grouped by module, sorted by MDL
4. Verify .dervish/grammars.yml created in RAGSAK
5. Run existing tests: python -m pytest tests/
6. Run: python -m bex.tag_preprocessor.analyze /home/tobi/Desktop/kesai/RAGSAK --slice package --verbose (verify no regression)