infinite-agents-public/infinite_variants/infinite_variant_5/README.md

# Configuration-Driven Infinite Loop Orchestration

An advanced infinite loop variant that implements **configuration-driven orchestration** with **chain prompting** patterns for multi-stage workflow execution.

---

**Web Learning Source**: https://docs.claude.com/en/docs/build-with-claude/prompt-engineering/chain-prompts

**Key Techniques Applied**:
1. **Workflow Decomposition**: Complex orchestration broken into 7 focused stages (Load → Validate → Merge → Analyze → Plan → Execute → Validate)
2. **State Passing via XML**: Each stage outputs structured state using XML tags for clear handoffs and traceability
3. **Self-Correction Loops**: Validation failures trigger automatic retry and refinement based on feedback
4. **Single-Task Focus**: Each stage has one clear goal ensuring Claude's full attention on that subtask

---

## Overview

This variant removes all hardcoded values and makes the entire orchestration system configurable through JSON files. It implements chain prompting (sequential workflow decomposition) to process configurations and execute generations in focused, single-task stages.

## Key Features

### 1. Configuration-Driven Architecture
- **Zero Hardcoded Values**: All orchestration parameters externalized to config files
- **Hierarchical Configuration**: Defaults → Profile → Custom → Runtime overrides
- **Multiple Profiles**: Pre-configured for development, production, and research use cases
- **JSON Schema Validation**: Ensures configuration correctness before execution
- **Runtime Overrides**: Command-line configuration overrides without file changes

### 2. Chain Prompting Implementation
Based on [Anthropic's chain prompting documentation](https://docs.claude.com/en/docs/build-with-claude/prompt-engineering/chain-prompts), implements:

- **Sequential Workflow Stages**: Load → Validate → Merge → Analyze → Plan → Execute → Validate
- **State Passing via XML**: Each stage outputs structured state for the next stage
- **Single-Task Focus**: Each stage has one clear goal for maximum attention
- **Self-Correction Loops**: Automatic retry and refinement when validation fails
- **Traceability**: Full workflow audit trail through XML state documents

### 3. Configuration Profiles

**Development Profile** (`development.json`)
- Small batches for quick testing
- Verbose logging for debugging
- Review stage enabled for quality checks
- Lower uniqueness thresholds for faster iteration

**Production Profile** (`production.json`)
- Large batches for scale
- Minimal logging for performance
- Optimized for throughput
- High uniqueness thresholds

**Research Profile** (`research.json`)
- Maximum quality settings
- Extensive web priming
- Comprehensive logging
- Extended timeouts for complex work

### 4. Advanced Features
- Configuration validation with detailed error reporting
- Interactive configuration creation and editing
- Configuration comparison and optimization
- Configuration merging with conflict resolution
- Web enhancement integration
- Progressive sophistication in infinite mode
- Automatic index generation (configurable)

## Quick Start

### 1. Use Default Configuration
```bash
claude
/project:infinite-config specs/example_spec.md output 5
```

### 2. Use Development Profile
```bash
/project:infinite-config specs/example_spec.md output_dev 3 development
```

### 3. Use Production Profile
```bash
/project:infinite-config specs/example_spec.md output_prod 20 production
```

### 4. Use Custom Configuration
```bash
/project:infinite-config specs/example_spec.md output 10 custom examples/custom_config.json
```

### 5. Use Inline Overrides
```bash
/project:infinite-config specs/example_spec.md output 5 development '{"orchestration":{"max_parallel_agents":8}}'
```

## Configuration System

### Configuration Hierarchy

Configurations merge in this order (later overrides earlier):

```
1. defaults.json (base configuration)
   ↓
2. profiles/{profile}.json (if specified)
   ↓
3. custom_config.json (if file path provided)
   ↓
4. inline JSON overrides (if provided)
   ↓
Final merged configuration
```

### Configuration Sections

**orchestration** - Parallel execution settings
- `max_parallel_agents`: How many sub-agents run simultaneously (1-10)
- `batch_size`: Iterations per batch (1-50)
- `infinite_mode_wave_size`: Iterations per wave in infinite mode (1-20)
- `context_budget_per_agent`: Token budget per agent (10000-200000)
- `agent_timeout_ms`: Timeout for agent execution (30000-600000)
- `enable_progressive_sophistication`: Increase complexity over time (boolean)

**generation** - Output file settings
- `output_directory`: Where to save generated files
- `naming_pattern`: File naming with placeholders: `{theme}`, `{iteration}`, `{variant}`, `{timestamp}`
- `file_format`: Output format (html, markdown, json, text)
- `include_metadata`: Embed generation metadata (boolean)
- `metadata_format`: Format for metadata (html_comment, yaml_frontmatter, json_header)

**quality** - Quality control settings
- `min_uniqueness_threshold`: Required uniqueness score 0.0-1.0
- `enable_validation`: Validate outputs (boolean)
- `enable_review_stage`: Add chain prompting review step (boolean)
- `max_retry_attempts`: Retry failed generations (0-5)
- `require_spec_compliance_check`: Validate spec compliance (boolean)

**web_enhancement** - Web learning settings
- `enabled`: Enable web-enhanced generation (boolean)
- `initial_priming_urls`: URLs for initial knowledge priming (0-10)
- `urls_per_iteration`: URLs to fetch per iteration (0-5)
- `progressive_difficulty`: Use progressive URL difficulty (boolean)
- `enable_web_search_fallback`: Search when URLs exhausted (boolean)
- `cache_web_content`: Cache fetched content (boolean)
- `web_fetch_timeout_ms`: Web fetch timeout (5000-60000)

**logging** - Logging settings
- `level`: Logging level (debug, info, warn, error)
- `log_agent_outputs`: Log sub-agent outputs (boolean)
- `log_web_fetches`: Log web operations (boolean)
- `log_config_loading`: Log config loading process (boolean)
- `verbose`: Maximum detail logging (boolean)

**chain_prompting** - Chain prompting settings
- `enabled`: Enable chain prompting workflow (boolean)
- `stages`: List of workflow stages to execute
- `enable_self_correction`: Enable self-correction loops (boolean)
- `pass_state_via_xml`: Use XML for state passing (boolean)

**features** - Feature flags
- `enable_url_strategy`: Support URL strategy files (boolean)
- `enable_theme_evolution`: Evolve themes across iterations (boolean)
- `enable_cross_iteration_learning`: Learn from previous iterations (boolean)
- `enable_automatic_indexing`: Auto-generate index files (boolean)

**limits** - Safety limits
- `max_iterations`: Maximum iterations before stopping (1+)
- `max_file_size_kb`: Maximum size per file in KB (10+)
- `max_total_output_mb`: Maximum total output in MB (1+)
- `warn_at_iteration`: Iteration to show warning (1+)

## Chain Prompting Workflow

### Stage 1: Load Configuration
**Goal**: Load and parse all configuration sources

Loads configurations in hierarchical order and prepares for validation.

### Stage 2: Validate Configuration
**Goal**: Validate all configurations against schema

Checks:
- Schema compliance (types, required fields, constraints)
- Value ranges and enums
- Pattern matching for strings

### Stage 3: Merge Configuration
**Goal**: Merge validated configurations hierarchically

Creates final configuration by deep-merging all sources in order.

### Stage 4: Analyze Specification
**Goal**: Deeply analyze specification file

Extracts requirements, quality standards, naming patterns, technical requirements.

### Stage 5: Plan Execution
**Goal**: Create detailed execution plan

Plans waves, agent assignments, web URL strategy, resource allocation.

### Stage 6: Execute Generation
**Goal**: Execute parallel agent generation

Runs initial web priming (if enabled), deploys agent waves, manages execution, handles errors.

### Stage 7: Validate Output (Optional)
**Goal**: Validate generated outputs

Checks spec compliance, uniqueness, file sizes. Enables self-correction loops.

## Commands

### /project:infinite-config
Main orchestration command with full chain prompting workflow.

```bash
/project:infinite-config <spec_path> <output_dir> <count> [profile] [config_overrides]
```

See `.claude/commands/infinite-config.md` for complete documentation.

### /project:validate-config
Validate configuration files with detailed error reporting.

```bash
/project:validate-config [config_path] [profile]
```

Validates against schema, checks semantic correctness, cross-field validation.

See `.claude/commands/validate-config.md` for complete documentation.

### /project:configure
Interactive configuration creation and management.

```bash
/project:configure [action] [profile] [output_path]
```

Actions: create, edit, compare, optimize, merge

See `.claude/commands/configure.md` for complete documentation.

## Usage Examples

### Example 1: Quick Development Test
```bash
# Use development profile for quick 3-iteration test
/project:infinite-config specs/example_spec.md test_output 3 development
```

### Example 2: Production Batch
```bash
# Use production profile for 50 high-quality iterations
/project:infinite-config specs/example_spec.md production_output 50 production
```

### Example 3: Research with Custom Config
```bash
# Start with research profile, add custom settings
/project:infinite-config specs/example_spec.md research_output 20 research examples/custom_config.json
```

### Example 4: Infinite Mode
```bash
# Infinite mode with production profile
/project:infinite-config specs/example_spec.md infinite_output infinite production
```

### Example 5: Runtime Override
```bash
# Development profile but with 8 parallel agents
/project:infinite-config specs/example_spec.md output 10 development '{"orchestration":{"max_parallel_agents":8}}'
```

### Example 6: Create Custom Configuration
```bash
# Interactive configuration creation
/project:configure create production my_custom.json
```

### Example 7: Validate Before Running
```bash
# Validate custom configuration
/project:validate-config examples/custom_config.json

# If valid, run it
/project:infinite-config specs/example_spec.md output 10 custom examples/custom_config.json
```

### Example 8: Compare Profiles
```bash
# Compare development vs production settings
/project:configure compare development production
```

### Example 9: Optimize for Speed
```bash
# Create speed-optimized configuration
/project:configure optimize speed
```

## Configuration Best Practices

### For Development
- Use small `batch_size` (3-5) for quick feedback
- Enable `verbose` logging for debugging
- Enable `enable_review_stage` for quality
- Lower `min_uniqueness_threshold` (0.7) for faster iteration
- Small `max_iterations` (10-20) to prevent runaway

### For Production
- Larger `batch_size` (10-20) for efficiency
- Minimal logging (`warn` level)
- Disable `enable_review_stage` for speed
- High `min_uniqueness_threshold` (0.85-0.9)
- Appropriate `max_iterations` for your use case
- Monitor `max_total_output_mb` to prevent disk issues

### For Research
- Medium `batch_size` (5-10) for balance
- Enable `verbose` logging to understand behavior
- Enable `enable_review_stage` for quality
- High `min_uniqueness_threshold` (0.9-0.95)
- Extensive web priming (5-8 URLs)
- Enable `enable_cross_iteration_learning`

### General Tips
- Always validate configurations before running
- Use profiles as starting points, customize as needed
- Use inline overrides for one-off parameter changes
- Monitor resource usage (context budget, output size)
- Test with small batches before running large ones
- Keep custom configurations in version control

## File Structure

```
infinite_variant_5/
├── .claude/
│   ├── commands/
│   │   ├── infinite-config.md      # Main orchestration command
│   │   ├── validate-config.md      # Configuration validation
│   │   └── configure.md            # Interactive config utility
│   ├── config/
│   │   ├── defaults.json           # Default configuration
│   │   ├── schema.json             # JSON schema for validation
│   │   └── profiles/
│   │       ├── development.json    # Development profile
│   │       ├── production.json     # Production profile
│   │       └── research.json       # Research profile
│   └── settings.json               # Tool permissions
├── specs/
│   └── example_spec.md             # Example specification
├── examples/
│   └── custom_config.json          # Example custom configuration
├── docs/
│   └── configuration_guide.md      # Complete configuration guide
├── README.md                       # This file
└── CLAUDE.md                       # Project instructions for Claude Code
```

## Comparison to Other Variants

| Feature | Variant 1 (Original) | Variant 5 (Config-Driven) |
|---------|---------------------|---------------------------|
| Configuration | Hardcoded | Fully configurable |
| Profiles | None | 3 built-in + custom |
| Workflow | Single-stage | Chain prompting (7 stages) |
| Validation | Basic | Schema + semantic + cross-field |
| Flexibility | Low | High |
| Learning Curve | Low | Medium |
| Production-Ready | No | Yes |
| Self-Correction | No | Yes (configurable) |

## Benefits

1. **Flexibility**: Every parameter can be adjusted without code changes
2. **Reproducibility**: Save configurations for consistent results
3. **Quality**: Multi-stage validation ensures correctness
4. **Scalability**: Profiles optimize for different scales
5. **Maintainability**: Configuration separate from logic
6. **Experimentation**: Easy to test different settings
7. **Collaboration**: Share configurations across team
8. **Transparency**: Clear audit trail through chain prompting stages

## Chain Prompting Advantages

Based on Anthropic's documentation, chain prompting provides:

1. **Focused Attention**: Each stage gets full Claude attention on single task
2. **Debuggability**: Isolate and fix problems in specific stages
3. **State Visibility**: XML state passing shows exact workflow progress
4. **Self-Correction**: Natural feedback loops for quality improvement
5. **Modularity**: Stages can be added, removed, or reordered
6. **Traceability**: Complete audit trail of decisions and transformations

## Next Steps

1. Explore the configuration profiles in `.claude/config/profiles/`
2. Read the complete guide in `docs/configuration_guide.md`
3. Try the example specification with different profiles
4. Create your own custom configuration with `/project:configure create`
5. Validate your configurations with `/project:validate-config`
6. Run generations with `/project:infinite-config`

## Support

For questions or issues:
1. Read `docs/configuration_guide.md` for detailed documentation
2. Use `/project:validate-config` to check configurations
3. Use `/project:configure compare` to understand profile differences
4. Check schema.json for all available configuration options
5. Review example_spec.md for specification guidelines

## License

Part of the Infinite Agents experimental project.