infinite-agents-public/SDG_NETWORK_GUIDE.md

# SDG Network Visualization - Web-Enhanced Infinite Loop Guide

## Overview

This guide explains how to use the web-enhanced infinite agentic loop to generate progressively sophisticated SDG (Sustainable Development Goals) network visualizations that integrate multiple open data APIs.

## What Gets Generated

Each iteration produces a **self-contained HTML file** with:
- **Force-directed network graph** using D3.js
- **Real data** from open APIs (environmental, scientific, SDG-related)
- **Progressive enhancements** in visualization quality and interactivity
- **Web-sourced techniques** applied and documented
- **API discovery** through automated web searches

## Current Baseline

The existing SDG network visualization (`ai_docs/sdg-network-modelling/`) provides:
- Basic D3.js v3 force-directed graph
- UN SDG API integration
- Simple node/edge rendering
- Basic hover tooltips
- Geographic and SDG goal relationships

## Progressive Enhancement Strategy

### **Phase 1: Foundation (Iterations 1-5)**
**Visual Enhancements:**
- Node size based on data magnitude
- Color coding by categories (using D3 color scales)
- Smart label positioning
- Rich hover tooltips with formatted data
- Edge width based on relationship strength
- Directional arrows and edge patterns

**Interactivity:**
- Drag nodes to reposition
- Zoom and pan controls
- Click to highlight connected nodes
- Basic filtering by node type
- Legend for color coding

**Data:**
- Single API per iteration
- Examples: UN SDG API, World Bank, OpenAQ, USGS Water

### **Phase 2: Intermediate (Iterations 6-12)**
**Visual Enhancements:**
- Multi-property color encoding (fill color + border color)
- Icon overlays on nodes
- Curved edges for better visibility
- Node clustering by community
- Edge bundling for cleaner layouts
- Custom node shapes for different entity types

**Interactivity:**
- Search functionality to find nodes
- Multi-select for comparison
- Time-based animation
- Expandable/collapsible groups
- Side panel with statistics
- Export visualization and data

**Data:**
- Combine 2-3 API sources
- Automatic data merging
- Examples: NASA + NOAA + EPA, WHO + World Bank + UN

### **Phase 3: Advanced (Iterations 13-20)**
**Visual Enhancements:**
- Force-based clustering algorithms
- Community detection visualization
- Centrality measures displayed
- Node importance ranking
- Hierarchical edge bundling
- Animated state transitions

**Interactivity:**
- Real-time data updates
- Graph layout algorithm switching
- Advanced filtering (boolean logic)
- Shortest path finding
- Neighborhood exploration
- Graph metric dashboards

**Data:**
- 4+ API sources integrated
- Automated API discovery via Swagger search
- Cross-API entity resolution
- Streaming data integration

### **Phase 4: Expert (Iterations 21+)**
**Visual Enhancements:**
- ML-based optimal node positioning
- Anomaly detection highlighting
- Predictive attributes visualization
- 3D network projections
- VR/AR capabilities

**Interactivity:**
- Natural language graph queries
- AI-powered insights
- Automatic pattern detection
- Graph comparison tools
- Collaborative features

**Data:**
- Federated queries across APIs
- Knowledge graph construction
- Semantic web integration
- Blockchain data sources

## API Discovery Process

Each iteration automatically discovers new data sources:

### **Target Domains:**
1. **Environmental**: Air quality, water quality, climate
2. **Scientific**: Biodiversity, satellites, research datasets
3. **Atmospheric**: Weather, air monitoring
4. **Ecological**: Forests, species, conservation
5. **Ocean**: Marine life, coastal data
6. **Geophysical**: Earthquakes, volcanoes
7. **Health**: Disease, mortality, healthcare
8. **SDG**: Development indicators

### **Discovery Method:**
1. Web search: `"[domain] API swagger site:github.com"`
2. Parse Swagger/OpenAPI specifications
3. Evaluate API quality and relevance
4. Map data to network structure (nodes/edges)
5. Integrate and visualize

### **API Examples:**
- **UN SDG API**: https://unstats.un.org/SDGAPI/v1/sdg/
- **OpenAQ**: Air quality data worldwide
- **USGS Water Services**: Water quality and flow
- **Global Forest Watch**: Deforestation and forest data
- **NASA Earthdata**: Satellite and earth science
- **NOAA**: Weather, ocean, climate data
- **GBIF**: Biodiversity occurrence data
- **World Bank**: Development indicators

## Web Learning Integration

Each iteration learns from curated web resources:

### **Foundation Topics:**
- D3.js fundamentals and data binding
- Basic force-directed graphs
- Color scales and node styling
- Interactive tooltips and zoom
- Canvas rendering for performance

### **Intermediate Topics:**
- Force-based clustering
- Custom force implementations
- Advanced color schemes
- Animated transitions
- Modular component architecture

### **Advanced Topics:**
- Edge bundling algorithms
- 3D projections and rotations
- Adaptive rendering for large graphs
- Real-time streaming data
- Multi-view interfaces

### **Expert Topics:**
- Graph theory algorithms (centrality, shortest path)
- Community detection
- Semantic zoom
- Export capabilities
- ML-enhanced layouts

## Usage Examples

### **Single Iteration with Default Settings:**
```bash
claude
> /project:infinite-web specs/sdg_network_progressive.md sdg_viz 1
```

This generates `sdg_viz/sdg_viz_1.html` with:
- UN SDG API baseline
- Basic force graph from D3.js documentation
- Foundation-level enhancements

### **Small Batch (5 iterations):**
```bash
> /project:infinite-web specs/sdg_network_progressive.md sdg_viz 5
```

Generates 5 files with:
- Different API sources (UN SDG, World Bank, OpenAQ, etc.)
- Progressive techniques from foundation URLs
- Each builds upon insights from previous iterations

### **Medium Batch with Custom URL Strategy:**
```bash
> /project:infinite-web specs/sdg_network_progressive.md sdg_viz 12 specs/sdg_network_url_strategy.json
```

Generates 12 iterations:
- Iterations 1-5: Foundation techniques
- Iterations 6-12: Intermediate techniques
- Multiple APIs combined
- Advanced interactivity

### **Infinite Mode (Continuous until context limit):**
```bash
> /project:infinite-web specs/sdg_network_progressive.md sdg_viz infinite specs/sdg_network_url_strategy.json
```

Runs continuously with:
- Progressive difficulty waves
- Automatic API discovery via web search
- Increasingly sophisticated visualizations
- Stops gracefully at context limit

## Expected Output Structure

Each `sdg_viz_[N].html` file contains:

```html
<!DOCTYPE html>
<html>
<head>
    <title>SDG Network Visualization [N] - [Data Theme]</title>
    <style>/* Complete styling */</style>
</head>
<body>
    <div id="visualization-container">
        <svg id="network-graph"></svg>
        <div id="controls">/* Interactive controls */</div>
        <div id="info-panel">/* Data details */</div>
        <div id="legend">/* Visual legend */</div>
    </div>

    <script src="https://d3js.org/d3.v7.min.js"></script>
    <script>
        // Fetch data from API(s)
        // Build network graph
        // Render force-directed visualization
        // Add interactivity
    </script>

    <footer>
        <p><strong>Data Sources:</strong> [APIs used]</p>
        <p><strong>Web Learning:</strong> [URL + techniques]</p>
    </footer>
</body>
</html>
```

## Key Improvements Over Baseline

### **Data Integration:**
- ❌ **Old**: Single hardcoded API (UN SDG only)
- ✅ **New**: Automatic API discovery, multiple sources combined

### **Visualization:**
- ❌ **Old**: Basic circles and lines, single color scheme
- ✅ **New**: Rich node/edge properties, dynamic sizing, multi-attribute encoding

### **Interactivity:**
- ❌ **Old**: Simple hover tooltip
- ✅ **New**: Search, filter, zoom, click actions, side panels, exports

### **Learning:**
- ❌ **Old**: Static implementation
- ✅ **New**: Each iteration learns from web and improves

### **Technology:**
- ❌ **Old**: D3.js v3 (outdated)
- ✅ **New**: D3.js v7 (modern), latest techniques

## Quality Metrics

Each iteration is evaluated on:

### **Data Richness:**
- Number of API sources (1 → 5+)
- Total nodes/edges
- Data update frequency

### **Visual Quality:**
- Color coding sophistication
- Label readability
- Animation smoothness (60fps)
- Layout clarity

### **Interactivity:**
- Number of features (hover → search → filter → AI)
- Response time
- Information density

### **Technical Innovation:**
- New techniques applied
- Performance improvements
- Code quality

## Advanced Use Cases

### **Compare Algorithms:**
Generate iterations with different force algorithms:
```bash
# Each iteration learns a different layout algorithm
> /project:infinite-web specs/sdg_network_progressive.md algorithm_comparison 8
```

### **Domain-Specific Networks:**
Focus on specific data domains by modifying the spec:
```bash
# Ocean/marine data network
> /project:infinite-web specs/ocean_network_progressive.md ocean_viz 10

# Climate change network
> /project:infinite-web specs/climate_network_progressive.md climate_viz 15
```

### **Real-Time Dashboards:**
Advanced iterations include streaming data:
```bash
# Generate real-time monitoring dashboards
> /project:infinite-web specs/sdg_network_progressive.md realtime_sdg 20
```

## Troubleshooting

### **API Access Issues:**
- Some APIs require keys → iterations handle gracefully with fallbacks
- CORS errors → iterations use proxy or alternatives
- Rate limits → iterations implement caching

### **Performance:**
- Large graphs → iterations use Canvas or WebGL
- Slow rendering → iterations implement LOD and culling
- Memory issues → iterations use virtualization

### **Browser Compatibility:**
- All iterations target modern browsers (Chrome, Firefox, Safari, Edge)
- Fallbacks for older browsers where reasonable
- Progressive enhancement ensures basic functionality

## Next Steps

1. **Run your first iteration:**
   ```bash
   claude
   > /project:infinite-web specs/sdg_network_progressive.md sdg_viz 1
   ```

2. **Review the output:**
   - Open `sdg_viz/sdg_viz_1.html` in browser
   - Check data sources and web learning attribution
   - Test interactivity

3. **Generate a batch:**
   ```bash
   > /project:infinite-web specs/sdg_network_progressive.md sdg_viz 5
   ```

4. **Compare iterations:**
   - See progressive improvements
   - Note new techniques applied
   - Observe API diversity

5. **Go infinite:**
   ```bash
   > /project:infinite-web specs/sdg_network_progressive.md sdg_viz infinite
   ```

## Resources

### **Specifications:**
- `specs/sdg_network_progressive.md` - Main visualization spec
- `specs/sdg_network_url_strategy.json` - Progressive learning URLs

### **Commands:**
- `.claude/commands/infinite-web.md` - Web-enhanced loop implementation

### **Baseline Reference:**
- `ai_docs/sdg-network-modelling/` - Original SDG network project

### **Generated Output:**
- `sdg_viz/` - Output directory for iterations (create via command)

## Contributing

To extend this system:

1. **Add new API sources** to `sdg_network_url_strategy.json` search templates
2. **Add learning resources** as new URLs in the strategy file
3. **Create domain-specific specs** by copying and modifying the base spec
4. **Share successful iterations** as examples

## Credits

- **Original SDG Network**: First Green Bank Network visualization
- **D3.js**: Michael Bostock and contributors
- **Observable**: Interactive notebook platform
- **Open APIs**: UN, World Bank, NOAA, USGS, NASA, and many others
- **Claude Code**: Infinite agentic loop implementation