# State Management Guide

Complete guide to using the stateful infinite loop system with self-consistency validation.

## Table of Contents

1. [Introduction](#introduction)
2. [Quick Start](#quick-start)
3. [Core Concepts](#core-concepts)
4. [State Structure](#state-structure)
5. [Self-Consistency Validation](#self-consistency-validation)
6. [Commands Reference](#commands-reference)
7. [Use Cases](#use-cases)
8. [Best Practices](#best-practices)
9. [Troubleshooting](#troubleshooting)
10. [Advanced Topics](#advanced-topics)

## Introduction

The stateful infinite loop system adds robust state management to the infinite agentic loop pattern. It enables:

- **Persistent State**: Runs survive interruptions and can be resumed
- **Deduplication**: URLs and iterations tracked to prevent duplicates
- **Validation**: Self-consistency checks ensure state reliability
- **Recovery**: Graceful handling of failures and interruptions
- **Auditability**: Complete history of all iterations

### What is Self-Consistency?

Self-consistency is a prompting technique that improves reliability by:

1. **Multiple Sampling**: Running multiple independent validation checks
2. **Consistency Analysis**: Comparing results across different approaches
3. **Majority Voting**: Selecting consensus answer as most reliable

Applied to state management, this means validating state integrity through multiple independent methods (file count, iteration records, URL uniqueness, etc.) and computing a consistency score via majority voting.

## Quick Start

### Running Your First Stateful Loop

```bash
# 1. Create a spec (or use example)
cd infinite_variants/infinite_variant_6

# 2. Run with 5 iterations
/infinite-stateful specs/example_spec.md outputs 5

# 3. Check status
/status

# 4. If interrupted, resume
/resume run_20250310_143022
```

### Basic Workflow

```
Create Spec → Run Loop → Monitor Status → Resume if Needed → Validate
```

## Core Concepts

### Run ID

Every execution gets a unique run ID:
- Format: `run_YYYYMMDD_HHMMSS`
- Example: `run_20250310_143022`
- Stored in: `.claude/state/run_20250310_143022.json`

### State File

JSON file containing complete run state:
- Iteration tracking
- URL deduplication
- Progress counters
- Validation metadata
- Timestamps

### Consistency Score

Float from 0.0 to 1.0 indicating state reliability:
- **1.00**: Perfect consistency (all checks pass)
- **0.75-0.99**: Good (minor issues)
- **0.50-0.74**: Warning (some problems)
- **0.00-0.49**: Corrupted (significant issues)

### Iteration Status

Each iteration can be:
- `pending`: Not yet started
- `in_progress`: Currently executing
- `completed`: Finished successfully
- `failed`: Encountered error

### Run Status

Overall run can be:
- `in_progress`: Currently running
- `paused`: Stopped, can resume
- `completed`: Finished successfully
- `failed`: Unrecoverable error

## State Structure

### Complete Schema

```json
{
  "run_id": "run_20250310_143022",
  "spec_path": "specs/example_spec.md",
  "output_dir": "outputs",
  "total_count": 10,
  "url_strategy_path": "specs/url_strategy.json",
  "status": "in_progress",
  "created_at": "2025-03-10T14:30:22Z",
  "updated_at": "2025-03-10T14:35:10Z",
  "completed_iterations": 3,
  "failed_iterations": 0,
  "iterations": [
    {
      "number": 1,
      "status": "completed",
      "output_file": "outputs/visualization_1.html",
      "web_url": "https://observablehq.com/@d3/learn-d3",
      "started_at": "2025-03-10T14:30:25Z",
      "completed_at": "2025-03-10T14:31:40Z",
      "validation_hash": "abc123def456",
      "metadata": {
        "techniques_learned": ["Scales", "Axes", "Transitions"],
        "data_source": "https://api.example.com/data",
        "file_size": 45120
      }
    }
  ],
  "used_urls": [
    "https://observablehq.com/@d3/learn-d3",
    "https://observablehq.com/@d3/bar-chart"
  ],
  "validation": {
    "last_check": "2025-03-10T14:35:10Z",
    "consistency_score": 1.0,
    "issues": []
  }
}
```

### Field Meanings

**Top Level:**
- `run_id`: Unique identifier for this run
- `spec_path`: Specification file being used
- `output_dir`: Where files are generated
- `total_count`: Total iterations requested (or "infinite")
- `url_strategy_path`: Optional URL strategy file
- `status`: Current run status
- `created_at`, `updated_at`: ISO timestamps
- `completed_iterations`: Count of finished iterations
- `failed_iterations`: Count of failed iterations
- `iterations`: Array of iteration records
- `used_urls`: All URLs used (for deduplication)
- `validation`: Consistency validation metadata

**Iteration Record:**
- `number`: Iteration number (1-indexed)
- `status`: Iteration status
- `output_file`: Path to generated file
- `web_url`: Web source used
- `started_at`, `completed_at`: Timestamps
- `validation_hash`: File content hash
- `metadata`: Additional iteration data

## Self-Consistency Validation

### Validation Checks

The system performs 6 independent validation checks:

#### 1. Schema Validation
Verifies all required fields are present in state.

**Pass Criteria**: All required fields exist

#### 2. File Count
Compares output directory file count to completed iterations.

**Pass Criteria**: `file_count >= completed_iterations`

#### 3. Iteration Records
Verifies state iteration records match completion count.

**Pass Criteria**: `completed_iteration_records == completed_iterations`

#### 4. URL Uniqueness
Ensures no duplicate URLs in used_urls list.

**Pass Criteria**: `total_urls == unique_urls`

#### 5. File Existence
Verifies all tracked output files still exist.

**Pass Criteria**: All files exist

#### 6. Timestamp Validity
Checks timestamp chronology (updated >= created).

**Pass Criteria**: Valid chronology

### Majority Voting

Consistency score = (passed checks) / (total checks)

Example:
- 6 checks performed
- 5 checks pass
- Consistency score = 5/6 = 0.83 (83%)
- Status: CONSISTENT

### When Validation Runs

Validation occurs at:
1. **Run initialization** (loading or creating state)
2. **Between batches** (after each wave completes)
3. **Final completion** (end of run)
4. **On demand** (`/status` command)
5. **Before resume** (`/resume` command)

### Interpreting Results

**Score >= 0.8 (CONSISTENT)**
- State is reliable
- Safe to resume
- No action needed

**Score 0.5-0.79 (WARNING)**
- Some inconsistencies detected
- Review specific failed checks
- Consider rebuilding state
- Manual inspection recommended

**Score < 0.5 (CORRUPTED)**
- Significant problems
- State may be unreliable
- Rebuild or start new run
- Manual repair likely needed

## Commands Reference

### /infinite-stateful

Main command for running stateful infinite loop.

**Syntax:**
```bash
/infinite-stateful <spec_path> <output_dir> <count> [url_strategy] [run_id]
```

**Examples:**
```bash
# New run
/infinite-stateful specs/viz.md outputs 10

# With URL strategy
/infinite-stateful specs/viz.md outputs 20 specs/urls.json

# Resume existing
/infinite-stateful specs/viz.md outputs infinite specs/urls.json run_20250310_143022

# Infinite mode
/infinite-stateful specs/viz.md outputs infinite
```

### /status

View run status and validate consistency.

**Syntax:**
```bash
/status [run_id]
```

**Examples:**
```bash
# List all runs
/status

# Detailed status for specific run
/status run_20250310_143022
```

**Output Includes:**
- Run information
- Progress metrics
- Consistency validation results
- Recent iterations
- Resumability status
- State file info

### /resume

Resume interrupted run.

**Syntax:**
```bash
/resume <run_id>
```

**Examples:**
```bash
/resume run_20250310_143022
```

**Process:**
1. Locate state file
2. Validate state consistency
3. Load parameters
4. Continue from last completed iteration
5. Verify progress

### /reset-state

Reset or rebuild state.

**Syntax:**
```bash
/reset-state <run_id> [--mode]
```

**Modes:**
- `--verify`: Check integrity (default)
- `--rebuild`: Reconstruct from files
- `--delete`: Remove state file

**Examples:**
```bash
# Verify state
/reset-state run_20250310_143022

# Rebuild from files
/reset-state run_20250310_143022 --rebuild

# Delete state
/reset-state run_20250310_143022 --delete
```

## Use Cases

### Use Case 1: Long-Running Generation

**Scenario:** Generate 100 visualizations, takes several hours

**Approach:**
```bash
# Start run
/infinite-stateful specs/viz.md outputs 100 specs/urls.json

# If interrupted after 47 iterations...
# State automatically saved

# Resume exactly where left off
/resume run_20250310_143022

# Continues from iteration 48
```

**Benefits:**
- No wasted work
- No duplicate iterations
- Seamless continuation

### Use Case 2: Infinite Mode with Context Limits

**Scenario:** Generate continuously until context limit

**Approach:**
```bash
# Start infinite mode
/infinite-stateful specs/viz.md outputs infinite specs/urls.json

# System automatically pauses at context limit
# State saved at: run_20250310_143022.json

# New session, resume
/resume run_20250310_143022

# Continues from saved point
```

**Benefits:**
- Unlimited iterations across sessions
- No URL duplication
- Continuous progress

### Use Case 3: State Corruption Recovery

**Scenario:** Manual file deletion caused state mismatch

**Approach:**
```bash
# Check status
/status run_20250310_143022
# Output: Consistency Score: 0.67 (WARNING)

# Verify specific issues
/reset-state run_20250310_143022 --verify
# Shows: File Existence Check: FAIL

# Rebuild state from files
/reset-state run_20250310_143022 --rebuild
# State reconstructed from output directory

# Verify fix
/status run_20250310_143022
# Output: Consistency Score: 1.00 (CONSISTENT)
```

**Benefits:**
- Automatic state repair
- No manual JSON editing
- Validated result

### Use Case 4: Progressive Web Learning

**Scenario:** Learn from URLs progressively without duplication

**Approach:**
```bash
# URL strategy with progression
# specs/urls.json:
{
  "foundation": ["url1", "url2", "url3"],
  "intermediate": ["url4", "url5"],
  "advanced": ["url6", "url7"],
  "expert": ["url8", "url9"]
}

# Run with strategy
/infinite-stateful specs/viz.md outputs 20 specs/urls.json

# State tracks used URLs:
# - Iteration 1: url1 (foundation)
# - Iteration 2: url2 (foundation)
# - ...
# - Iteration 5: url4 (intermediate)
# No URL used twice
```

**Benefits:**
- Guaranteed URL deduplication
- Progressive difficulty
- Complete learning path

## Best Practices

### State Management

**DO:**
- Let system manage state automatically
- Use `/status` to check consistency regularly
- Resume interrupted runs rather than restart
- Backup state files before manual edits
- Use run_id for all resume operations

**DON'T:**
- Manually edit state files unless necessary
- Delete state files without backup
- Reuse run_id for different specs
- Assume state is always consistent
- Skip validation checks

### Resumability

**DO:**
- Use consistent spec and output_dir
- Preserve output directory between sessions
- Check state before resuming
- Let system determine next iteration

**DON'T:**
- Modify output files manually
- Change spec between resume
- Delete completed iterations
- Manually adjust iteration counters

### URL Deduplication

**DO:**
- Use URL strategy files for organization
- Let state track used URLs
- Trust deduplication system
- Provide fallback search terms

**DON'T:**
- Manually manage URL lists
- Duplicate URLs across strategy levels
- Modify used_urls array directly
- Ignore URL exhaustion warnings

### Validation

**DO:**
- Run validation before critical operations
- Investigate low consistency scores
- Use multiple validation approaches
- Trust majority voting results

**DON'T:**
- Ignore consistency warnings
- Proceed with corrupted state
- Skip pre-resume validation
- Disable validation checks

## Troubleshooting

### Problem: Low Consistency Score

**Symptoms:**
- `/status` shows score < 0.8
- Specific validation checks fail
- Warning or corrupted status

**Diagnosis:**
```bash
# Check detailed status
/status run_20250310_143022

# Verify specific checks
/reset-state run_20250310_143022 --verify

# Use validator script
./validators/check_state_consistency.sh run_20250310_143022
```

**Solutions:**

**If File Count Mismatch:**
```bash
# Files manually deleted
/reset-state run_20250310_143022 --rebuild
# Rebuilds state from remaining files
```

**If URL Duplicates:**
```bash
# State corruption
/reset-state run_20250310_143022 --rebuild
# Re-extracts URLs, deduplicates
```

**If Schema Invalid:**
```bash
# State file corrupted
# Restore from backup or rebuild
/reset-state run_20250310_143022 --rebuild
```

### Problem: Cannot Resume

**Symptoms:**
- `/resume` fails
- State file not found
- Parameters mismatch

**Solutions:**

**State File Missing:**
```bash
# List available runs
/status

# Use correct run_id
/resume run_20250310_143022
```

**Wrong Parameters:**
```bash
# State contains original parameters
# System will use those automatically
/resume run_20250310_143022
```

**Corrupted State:**
```bash
# Try rebuild first
/reset-state run_20250310_143022 --rebuild

# Then resume
/resume run_20250310_143022
```

### Problem: Duplicate URLs

**Symptoms:**
- Same URL used multiple times
- URL uniqueness check fails
- Web learning not diverse

**Solutions:**

**Check State:**
```bash
/status run_20250310_143022
# Look at URL Uniqueness Check
```

**Rebuild State:**
```bash
/reset-state run_20250310_143022 --rebuild
# Re-extracts URLs from files
# Deduplicates automatically
```

**Verify URL Strategy:**
```bash
# Ensure URL strategy has unique URLs
cat specs/url_strategy.json
```

### Problem: Missing Output Files

**Symptoms:**
- File existence check fails
- Files tracked in state don't exist
- Manual deletion occurred

**Solutions:**

**Accept Loss:**
```bash
# Rebuild state from existing files
/reset-state run_20250310_143022 --rebuild
# State updated to match reality
```

**Regenerate:**
```bash
# Delete state, regenerate missing iterations
/reset-state run_20250310_143022 --delete
# Start new run to regenerate
```

## Advanced Topics

### Custom Validation Checks

Add domain-specific validation:

```python
from state_manager import StateManager

def validate_output_quality(state):
    """Check output file quality"""
    for iteration in state['iterations']:
        if iteration['status'] == 'completed':
            # Check file size
            file_size = os.path.getsize(iteration['output_file'])
            if file_size < 1000:  # Less than 1KB
                return False

            # Check metadata
            with open(iteration['output_file']) as f:
                content = f.read()
                if 'web_source' not in content:
                    return False

    return True

# Use in validation
sm = StateManager()
state = sm.load_state('run_20250310_143022')
quality_check = validate_output_quality(state)
```

### State Migration

Migrate state schema across versions:

```python
def migrate_v1_to_v2(state):
    """Migrate state from v1 to v2 schema"""
    # Add version field
    state['version'] = 2

    # Add new fields with defaults
    state['metadata'] = state.get('metadata', {})

    # Migrate iteration records
    for iteration in state['iterations']:
        if 'validation_hash' not in iteration:
            # Compute hash for existing files
            if os.path.exists(iteration['output_file']):
                iteration['validation_hash'] = compute_hash(
                    iteration['output_file']
                )

    return state
```

### Parallel Run Detection

Prevent concurrent runs (optional):

```python
import fcntl

def acquire_lock():
    """Acquire exclusive lock for state directory"""
    lock_file = '.claude/state/.lock'
    lock = open(lock_file, 'w')

    try:
        fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
        return lock
    except IOError:
        print("Another run is in progress")
        exit(1)

# Use at run start
lock = acquire_lock()
# ... run loop ...
lock.close()
```

### State Compression

For very long runs, compress old state:

```bash
# Compress states older than 30 days
find .claude/state -name "run_*.json" -mtime +30 -exec gzip {} \;

# Decompress when needed
gunzip .claude/state/run_20250310_143022.json.gz
```

### Batch State Analysis

Analyze multiple runs:

```python
from state_manager import StateManager

sm = StateManager()

# Get all runs
runs = sm.list_runs()

# Analyze each
for run_id in runs:
    state = sm.load_state(run_id)
    score, _ = sm.validate_consistency(state)

    print(f"{run_id}:")
    print(f"  Status: {state['status']}")
    print(f"  Iterations: {state['completed_iterations']}")
    print(f"  Consistency: {score:.2f}")
    print()
```

### Export State Summary

Generate report across runs:

```bash
# Create summary
for f in .claude/state/run_*.json; do
  run_id=$(basename $f .json)
  echo "=== $run_id ==="
  jq -r '"Status: \(.status)\nIterations: \(.completed_iterations)\nConsistency: \(.validation.consistency_score)"' $f
  echo ""
done > state_summary.txt
```

## Conclusion

The stateful infinite loop system provides robust state management with self-consistency validation. Key takeaways:

- **State is persistent** and survives interruptions
- **Self-consistency validation** ensures reliability
- **URL deduplication** prevents repetition
- **Resume capability** enables long-running tasks
- **Multiple validation checks** provide high confidence

Use the system to:
- Generate large numbers of iterations safely
- Learn from web sources without duplication
- Recover gracefully from interruptions
- Maintain audit trail of all work

For support:
- Review this guide
- Use `/status` for diagnostics
- Run validator script
- Check `.claude/state/README.md`