9.6 KiB

Raw Blame History

Infinite Loop with Quality Evaluation & Ranking System

You are orchestrating an Infinite Agentic Loop with Automated Quality Evaluation using the ReAct pattern (Reasoning + Acting).

ReAct Integration

This command implements the Thought-Action-Observation cycle:

THOUGHT Phase: Reason about quality dimensions, evaluation strategy, and improvement opportunities
ACTION Phase: Execute evaluations, generate content, score iterations
OBSERVATION Phase: Analyze results, identify patterns, adapt strategy for next wave

Command Syntax

/project:infinite-quality <spec_path> <output_dir> <count|infinite> [quality_config]

Parameters:

spec_path - Path to specification file (must include quality criteria)
output_dir - Directory for generated iterations
count - Number of iterations (1-50) or "infinite" for continuous mode
quality_config - Optional: Path to custom scoring weights config

Examples:

/project:infinite-quality specs/example_spec.md output/ 5
/project:infinite-quality specs/example_spec.md output/ infinite config/scoring_weights.json

Execution Flow with ReAct Pattern

Phase 1: THOUGHT - Initial Reasoning

Duration: 30 seconds

Analyze Specification with Quality Lens
- Read spec file completely
- Identify explicit quality criteria
- Extract technical requirements
- Understand creative dimensions
- Map spec compliance checkpoints
Reason About Evaluation Strategy
- Determine which quality dimensions are most important
- Plan evaluation sequence (technical → creativity → compliance)
- Identify potential quality pitfalls
- Design scoring rubric based on spec
Survey Existing Context
- Check output directory for previous iterations
- If iterations exist, perform quick quality scan
- Identify quality trends and gaps
- Reason about what's missing or underrepresented
Plan Quality-Driven Generation Strategy
- Decide creative directions that maximize quality diversity
- Plan evaluation checkpoints
- Design improvement feedback loop

Output: Internal reasoning document outlining:

Quality dimensions identified
Evaluation strategy
Generation plan informed by quality goals

Phase 2: ACTION - Generate Iterations

Duration: Variable based on count

Launch Parallel Sub-Agents

For each iteration (batch size based on count):
- Assign unique creative direction with quality targets
- Provide spec + quality standards
- Each agent generates iteration with quality documentation
Batch Sizing:
- count 1-3: Sequential (1 at a time)
- count 4-10: Small batches (2-3 parallel)
- count 11-20: Medium batches (4-5 parallel)
- count 21+: Large batches (6-8 parallel)
- infinite: Waves of 6-8, continuous

Sub-Agent Quality Instructions

Each sub-agent receives:

You are generating iteration {N} for this specification.

SPECIFICATION: {spec_content}

QUALITY STANDARDS: {quality_standards}

CREATIVE DIRECTION: {unique_direction}

QUALITY TARGETS:
- Technical: {technical_targets}
- Creativity: {creativity_targets}
- Compliance: {compliance_targets}

REQUIREMENTS:
1. Follow specification exactly
2. Implement creative direction uniquely
3. Meet all quality targets
4. Document design decisions
5. Include self-assessment comments

OUTPUT: Generate complete iteration with quality documentation.

Phase 3: OBSERVATION - Evaluate & Analyze

Duration: 1-2 minutes per wave

Execute Evaluation Pipeline

For each generated iteration:

A. Technical Quality Evaluation
- Use /evaluate technical {iteration_path}
- Scores: Code quality, architecture, performance, robustness
- Weight: 35% (configurable)
B. Creativity Score Evaluation
- Use /evaluate creativity {iteration_path}
- Scores: Originality, innovation, uniqueness, aesthetic
- Weight: 35% (configurable)
C. Spec Compliance Evaluation
- Use /evaluate compliance {iteration_path} {spec_path}
- Scores: Requirements met, naming, structure, standards
- Weight: 30% (configurable)

Calculate Composite Scores

For each iteration:

composite_score = (technical * 0.35) + (creativity * 0.35) + (compliance * 0.30)

Range: 0-100

Rank Iterations

Use /rank {output_dir} to:
- Sort iterations by composite score
- Identify top performers (top 20%)
- Identify low performers (bottom 20%)
- Calculate mean, median, std deviation
- Detect quality outliers
Generate Quality Report

Use /quality-report {output_dir} to create:
- Overall quality metrics
- Individual iteration scores
- Ranking table
- Quality distribution charts (text-based)
- Insights and patterns
- Improvement recommendations

Phase 4: THOUGHT - Reasoning About Results

Duration: 30 seconds

After observation, reason about:

Quality Pattern Analysis
- What makes top iterations successful?
- What causes low scores?
- Are there quality trade-offs? (technical vs creative)
- Which quality dimension needs most improvement?
Strategic Insights
- Is the spec clear enough for high compliance?
- Are creative directions too conservative or too wild?
- Do technical standards need adjustment?
- Are evaluation criteria fair and meaningful?
Next Wave Planning (for infinite mode)
- Learn from top performers: Extract successful patterns
- Address low scores: Identify missing creative directions
- Adjust difficulty: Push boundaries in weak areas
- Diversify quality: Ensure all dimensions are represented

Output: Reasoning summary with actionable insights

Phase 5: ACTION - Adapt and Continue (Infinite Mode Only)

Based on Phase 4 reasoning:

Adjust Generation Strategy
- Incorporate lessons from top-ranked iterations
- Assign creative directions that address quality gaps
- Increase challenge in areas of strength
- Explore underrepresented creative spaces
Update Quality Targets
- Raise bar in dimensions with high scores
- Provide scaffolding in weak dimensions
- Balance technical and creative excellence
Launch Next Wave
- Return to Phase 2 with updated strategy
- Maintain quality evaluation for all new iterations
- Continue Thought-Action-Observation cycle

Infinite Mode Behavior

Wave Structure:

Wave 1: Foundation (6-8 iterations) → Evaluate → Reason → Report
Wave 2: Informed (6-8 iterations) → Evaluate → Reason → Report
Wave 3+: Progressive refinement with quality-driven adaptation

Quality Progression:

Early waves: Establish baseline quality
Mid waves: Push boundaries in specific dimensions
Late waves: Optimize composite scores, explore quality frontiers

Termination:

Continue until context limits approached
Final comprehensive quality report
Summary of quality evolution across all waves

Quality Report Format

After each wave (or final batch), generate:

# Quality Evaluation Report - Wave {N}

## Summary Statistics
- Total Iterations: {count}
- Mean Score: {mean}
- Median Score: {median}
- Std Deviation: {std}
- Top Score: {max}
- Lowest Score: {min}

## Rankings (Top 5)
1. iteration_{X} - Score: {score} - Strengths: {strengths}
2. iteration_{Y} - Score: {score} - Strengths: {strengths}
...

## Quality Dimension Breakdown
- Technical Quality: Mean {mean_tech}, Range {min_tech}-{max_tech}
- Creativity Score: Mean {mean_creative}, Range {min_creative}-{max_creative}
- Spec Compliance: Mean {mean_compliance}, Range {min_compliance}-{max_compliance}

## Insights & Patterns
- {observation_1}
- {observation_2}
- {observation_3}

## Recommendations for Next Wave
- {recommendation_1}
- {recommendation_2}
- {recommendation_3}

Key Implementation Notes

ReAct Principle Application:
- Every evaluation is preceded by reasoning
- Every action produces observations
- Observations inform next reasoning cycle
- Continuous feedback loop improves quality over time
Quality-Driven Diversity:
- Don't just generate random variations
- Target specific quality dimensions with each iteration
- Use evaluation to discover quality frontiers
Transparent Reasoning:
- Document thought process before actions
- Explain evaluation logic
- Justify strategic decisions
- Make quality criteria explicit
Adaptive Learning:
- Low scores trigger investigation and adjustment
- High scores reveal successful patterns to amplify
- Quality trends inform strategic direction changes
Evaluation Integrity:
- Apply consistent criteria across all iterations
- Use objective metrics where possible
- Document subjective judgments with reasoning
- Avoid evaluation drift over time

Success Criteria

A successful quality evaluation system demonstrates:

Meaningful score differentiation (not all similar scores)
Clear correlation between scores and actual quality
Actionable insights from quality reports
Visible quality improvement in infinite mode
Transparent reasoning at every decision point
ReAct pattern implementation throughout

Error Handling

If spec lacks quality criteria: Use default standards from specs/quality_standards.md
If evaluation fails: Document failure, assign neutral score, continue
If all scores are identical: Increase evaluation granularity
If infinite mode stalls: Generate quality-improvement reasoning, adjust strategy

Remember: Quality evaluation is not just scoring - it's a reasoning process. Think before you evaluate, observe after you act, and let observations guide your next thoughts.

9.6 KiB Raw Blame History