grammar-inference-engine/experiments/PHASE2_PLAN.md

# Phase 2: Decomposition Forest

## Goal

Break down complex/long behavioral sequences into shorter ones that still capture the pattern.
This helps when sequences are long and diverse, making CRX produce flat bags.

## Current Problem

Long sequences like:
```
["if", "return", "if", "return", "if", "return"]
```

CRX sees 6 symbols, tries to find a pattern, often produces flat bags like `(if|return)*`.

If we decompose into shorter examples:
```
["if", "return"]
["if", "return"]
["if", "return"]
```

CRX sees a clear pattern: `if.return` (repeated).

## Crucio's Approach

Crucio uses three decomposition strategies:
1. **Binary maximum subsequence deletion**: Split in half, delete max from each half
2. **Maximum subsequence deletion**: Delete largest contiguous chunk
3. **Subsequence replacement**: Replace a chunk with a shorter version

Key insight: Decomposed sequences must preserve grammar coverage (be valid under the same grammar).

## Our Adaptation

For behavioral sequences, we need simpler decomposition:
1. **Prefix extraction**: Take first N symbols
2. **Suffix extraction**: Take last N symbols
3. **Window extraction**: Take middle N symbols
4. **Pattern extraction**: Find repeated patterns and extract one instance

## Implementation Plan

### File: `bex/decompose.py` (new)

```python
"""Decomposition forest for behavioral sequences.

Inspired by Crucio's decomposition forest (ICSE 2026).
Breaks down long sequences into shorter ones that preserve patterns.
"""

def decompose_sequence(seq, max_length=5):
    """Decompose a sequence into shorter fragments.
    
    Strategies:
    1. If seq <= max_length, return as-is
    2. Extract prefixes of length 1..max_length
    3. Extract suffixes of length 1..max_length
    4. Extract windows of length max_length
    
    Args:
        seq: List of symbols
        max_length: Maximum fragment length
        
    Returns:
        List of fragments (shorter sequences)
    """
    if len(seq) <= max_length:
        return [seq]
    
    fragments = []
    
    # Prefixes
    for i in range(1, min(max_length + 1, len(seq))):
        fragments.append(seq[:i])
    
    # Suffixes
    for i in range(1, min(max_length + 1, len(seq))):
        fragments.append(seq[-i:])
    
    # Windows
    for start in range(0, len(seq) - max_length + 1):
        fragments.append(seq[start:start + max_length])
    
    return fragments


def decompose_all(sequences, max_length=5):
    """Decompose all sequences in a list.
    
    Args:
        sequences: List of lists of symbols
        max_length: Maximum fragment length
        
    Returns:
        List of fragments (shorter sequences)
    """
    all_fragments = []
    for seq in sequences:
        all_fragments.extend(decompose_sequence(seq, max_length))
    return all_fragments


def filter_by_coverage(fragments, min_coverage=0.5):
    """Keep only fragments that appear in at least min_coverage of original sequences.
    
    This ensures we keep patterns that are common, not rare.
    """
    from collections import Counter
    
    # Count how many original sequences each fragment appears in
    fragment_counts = Counter()
    for frag in fragments:
        fragment_counts[tuple(frag)] += 1
    
    # Keep fragments that appear frequently enough
    min_count = int(len(fragments) * min_coverage)
    return [list(frag) for frag, count in fragment_counts.items() 
            if count >= min_count]
```

### Integration with Pipeline

Add `--decompose` flag:
```python
parser.add_argument('--decompose', action='store_true',
                    help='Decompose long sequences before inference')
parser.add_argument('--max-seq-length', type=int, default=5,
                    help='Maximum sequence length after decomposition')
```

In `_infer_group`:
```python
if decompose:
    symbol_seqs = decompose_all(symbol_seqs, max_length=max_seq_length)
```

## Expected Benefits

1. **Shorter sequences**: CRX works better on shorter inputs
2. **Clearer patterns**: Decomposition reveals underlying structure
3. **Fewer flat bags**: Long diverse sequences become short uniform ones

## Test Plan

1. Unit tests: `tests/test_decompose.py`
2. Integration: Compare grammar count with/without decomposition
3. Metric: `grammar_structure_score()` should improve

## Questions to Answer

1. Does decomposition actually improve grammar quality?
2. What max_length works best?
3. How much slower is it?
4. Does it help on flat bags specifically?