infinite-agents-public/infinite_variants/infinite_variant_6/README.md

# Stateful Infinite Loop - Variant 6

Robust state management system for infinite agentic loops with self-consistency validation, resumability, and deduplication.

## Innovation

This variant adds **persistent state management** to the infinite loop pattern, enabling:

### Core Features

- **Persistent State**: Runs survive interruptions and can resume exactly where they left off
- **Self-Consistency Validation**: Multiple independent checks with majority voting ensure state reliability
- **URL Deduplication**: Tracks all used web sources to prevent duplicates
- **Graceful Recovery**: Handles failures, interruptions, and corruption with automated repair
- **Complete Auditability**: Full history of iterations, URLs, and validation results

### Self-Consistency Principle

This system applies **self-consistency prompting** (research-backed AI technique) to state validation:

1. **Multiple Sampling**: 6 independent validation approaches check state integrity
2. **Consistency Analysis**: Each check provides pass/fail assessment
3. **Majority Voting**: Consistency score = (passed checks) / (total checks)
4. **High Confidence**: Multiple methods ensure reliable validation even if one method has edge cases

**Web Learning Source**: Research on self-consistency prompting for improving AI reliability through multiple sampling and majority voting, applied to state management validation.

## Quick Start

### Installation

```bash
cd infinite_variants/infinite_variant_6
```

### Basic Usage

```bash
# 1. Run with 5 iterations
/infinite-stateful specs/example_spec.md outputs 5

# 2. Check status
/status

# 3. If interrupted, resume
/resume run_20250310_143022
```

### With URL Strategy

```bash
# Create URL strategy (optional)
cat > specs/my_urls.json << 'EOF'
{
  "foundation": [
    "https://observablehq.com/@d3/learn-d3",
    "https://www.d3-graph-gallery.com/intro_d3js.html"
  ],
  "intermediate": [
    "https://observablehq.com/@d3/force-directed-graph"
  ],
  "advanced": [
    "https://observablehq.com/@d3/hierarchical-edge-bundling"
  ]
}
EOF

# Run with URL strategy
/infinite-stateful specs/example_spec.md outputs 10 specs/my_urls.json
```

### Infinite Mode

```bash
# Generate continuously until context limit
/infinite-stateful specs/example_spec.md outputs infinite specs/my_urls.json

# In new session, resume from exact point
/resume run_20250310_143022
```

## Architecture

### Directory Structure

```
infinite_variant_6/
├── .claude/
│   ├── commands/
│   │   ├── infinite-stateful.md    # Main orchestration command
│   │   ├── resume.md               # Resume interrupted runs
│   │   ├── status.md               # View run status
│   │   └── reset-state.md          # State management utilities
│   ├── settings.json               # Tool permissions
│   └── state/                      # State files directory
│       ├── README.md               # State system documentation
│       └── run_*.json              # Individual run states
├── specs/
│   └── example_spec.md             # Example specification
├── templates/
│   ├── run_state.json              # State template
│   ├── url_tracker.json            # URL tracking template
│   └── iteration_metadata.json     # Iteration record template
├── docs/
│   └── state_management_guide.md   # Complete usage guide
├── validators/
│   └── check_state_consistency.sh  # State validation script
├── state_manager.py                # State management utilities
├── README.md                       # This file
└── CLAUDE.md                       # Project instructions
```

### State File Structure

```json
{
  "run_id": "run_20250310_143022",
  "spec_path": "specs/example_spec.md",
  "output_dir": "outputs",
  "total_count": 10,
  "status": "in_progress",
  "completed_iterations": 3,
  "iterations": [...],
  "used_urls": [...],
  "validation": {
    "consistency_score": 1.0,
    "issues": []
  }
}
```

## Commands Reference

### /infinite-stateful

Main command for running stateful infinite loops.

**Syntax:**
```bash
/infinite-stateful <spec_path> <output_dir> <count> [url_strategy] [run_id]
```

**Features:**
- Automatic state persistence
- URL deduplication
- Self-consistency validation
- Graceful interruption handling
- Resume capability

### /status

View run status and validate consistency.

**Syntax:**
```bash
/status [run_id]
```

**Output:**
- Run information (spec, progress, timing)
- Consistency validation results (6 independent checks)
- Recent iterations
- Resumability status
- URL usage summary

### /resume

Resume interrupted run.

**Syntax:**
```bash
/resume <run_id>
```

**Process:**
1. Validates state consistency
2. Loads original parameters
3. Continues from last completed iteration
4. Preserves URL deduplication

### /reset-state

State management utilities.

**Syntax:**
```bash
/reset-state <run_id> [--mode]
```

**Modes:**
- `--verify`: Check integrity (default)
- `--rebuild`: Reconstruct state from files
- `--delete`: Remove state file (with backup)

## Usage Examples

### Example 1: Simple Run

```bash
# Generate 5 visualizations
/infinite-stateful specs/example_spec.md viz_output 5

# Output:
# - viz_output/visualization_1.html
# - viz_output/visualization_2.html
# - ...
# - .claude/state/run_20250310_143022.json (state file)
```

### Example 2: Interrupted Run Recovery

```bash
# Start generation of 100 items
/infinite-stateful specs/example_spec.md outputs 100

# ... generates 47 items, then context limit or interruption ...
# State automatically saved at .claude/state/run_20250310_143022.json

# Check what happened
/status run_20250310_143022
# Shows: 47 of 100 completed

# Resume exactly where left off
/resume run_20250310_143022
# Continues from iteration 48
```

### Example 3: Infinite Mode Across Sessions

```bash
# Session 1: Start infinite mode
/infinite-stateful specs/example_spec.md outputs infinite specs/urls.json
# ... generates 50 iterations, hits context limit ...

# Session 2: Resume
/resume run_20250310_143022
# ... generates another 50 iterations ...

# Session 3: Resume again
/resume run_20250310_143022
# Continues from iteration 101
# No URL duplicates across all sessions
```

### Example 4: State Corruption Recovery

```bash
# Check status
/status run_20250310_143022
# Consistency Score: 0.67 (WARNING)
# File Existence Check: FAIL - 3 missing files

# Rebuild state from existing files
/reset-state run_20250310_143022 --rebuild
# State reconstructed successfully

# Verify fix
/status run_20250310_143022
# Consistency Score: 1.00 (CONSISTENT)
```

## Comparison with Base Infinite Loop

| Feature | Base Loop | Stateful Loop (This Variant) |
|---------|-----------|------------------------------|
| State Persistence | No | Yes - survives interruptions |
| Resume Capability | No | Yes - exact continuation point |
| URL Deduplication | Manual | Automatic via state tracking |
| Validation | None | Self-consistency with 6 checks |
| Corruption Recovery | N/A | Automatic rebuild from files |
| Audit Trail | Limited | Complete iteration history |
| Cross-Session | No | Yes - resume in new session |
| Failure Handling | Stop | Graceful with recovery options |

## Self-Consistency Validation

The system validates state using 6 independent checks:

1. **Schema Validation**: All required fields present
2. **File Count**: Output files match iteration count
3. **Iteration Records**: State records match completions
4. **URL Uniqueness**: No duplicate URLs
5. **File Existence**: All tracked files exist
6. **Timestamp Validity**: Chronology is valid

**Consistency Score** = (passed checks) / 6

**Interpretation:**
- **≥0.8**: State is consistent and reliable
- **0.5-0.79**: Warnings, review recommended
- **<0.5**: Corrupted, rebuild needed

This multi-check approach with majority voting ensures high-confidence state validation.

## Extension Points

### Custom Validation Checks

Add domain-specific validation:

```python
from state_manager import StateManager

def validate_custom(state):
    # Your custom checks
    pass

sm = StateManager()
state = sm.load_state('run_20250310_143022')
custom_valid = validate_custom(state)
```

### State Migration

Migrate state schema across versions:

```python
def migrate_v1_to_v2(state):
    state['version'] = 2
    # Add new fields
    return state
```

### Custom Commands

Add variant-specific commands:

```markdown
**CUSTOM ANALYSIS COMMAND**

Analyze state for specific patterns...
```

### Integration with Other Systems

```python
# Export state to external system
import json
from state_manager import StateManager

sm = StateManager()
state = sm.load_state('run_20250310_143022')

# Export summary
summary = {
    'run_id': state['run_id'],
    'completed': state['completed_iterations'],
    'urls_used': len(state['used_urls'])
}

with open('export.json', 'w') as f:
    json.dump(summary, f)
```

## Best Practices

### State Management

- Let system manage state automatically
- Use `/status` to check consistency regularly
- Resume interrupted runs rather than restart
- Backup state files before manual edits

### Resumability

- Use consistent spec and output_dir
- Preserve output directory between sessions
- Check state before resuming
- Let system determine next iteration

### URL Deduplication

- Use URL strategy files for organization
- Let state track used URLs automatically
- Provide fallback search terms for exhaustion
- Trust deduplication system

### Validation

- Run validation before critical operations
- Investigate low consistency scores
- Use multiple validation approaches
- Trust majority voting results

## Troubleshooting

### Low Consistency Score

```bash
# Diagnose
/status run_20250310_143022

# Rebuild if needed
/reset-state run_20250310_143022 --rebuild
```

### Cannot Resume

```bash
# List available runs
/status

# Use correct run_id
/resume run_20250310_143022
```

### Missing Files

```bash
# Rebuild state from existing files
/reset-state run_20250310_143022 --rebuild
```

### Duplicate URLs

```bash
# Check URL uniqueness
/status run_20250310_143022

# Rebuild to deduplicate
/reset-state run_20250310_143022 --rebuild
```

## Documentation

- **[State Management Guide](docs/state_management_guide.md)**: Complete usage guide
- **[State README](.claude/state/README.md)**: State system documentation
- **[CLAUDE.md](CLAUDE.md)**: Project instructions for Claude Code

## Use Cases

### Research Projects

Generate large datasets across multiple sessions with guaranteed uniqueness.

### Long-Running Tasks

Handle context limits gracefully with automatic resume.

### Quality Assurance

Self-consistency validation ensures reliable operation.

### Progressive Learning

Track web sources to ensure diverse learning without duplication.

## Future Enhancements

Potential extensions:

- **Distributed State**: Sync state across machines
- **State Compression**: Compress old state files
- **Custom Validators**: Plugin system for validation checks
- **State Analytics**: Dashboard for run analysis
- **Collaborative Runs**: Multiple users sharing state
- **Version Control Integration**: Git integration for state

## Contributing

To extend this variant:

1. Add custom commands in `.claude/commands/`
2. Extend state schema with migrations
3. Add custom validation checks
4. Integrate with external systems
5. Document extensions

## License

Part of the Infinite Agents project demonstrating Claude Code capabilities.

## Credits

- **Self-Consistency Technique**: Applied from AI prompting research (multiple sampling + majority voting)
- **State Management Pattern**: Inspired by database transaction systems
- **Validation Approach**: Multi-method consensus validation

---

**Web Learning Applied**: This variant demonstrates learning from self-consistency prompting research, applying the technique of multiple sampling and majority voting to state validation instead of language model outputs. The principle generalizes: multiple independent approaches + consensus = higher reliability.