canvas-website/DATA_CONVERSION_GUIDE.md

Data Conversion Guide: TLDraw Sync to Automerge Sync

This guide explains the data conversion process from the old TLDraw sync format to the new Automerge sync format, and how to verify the conversion is working correctly.

Data Format Changes

Old Format (TLDraw Sync)

{
  "documents": [
    { "state": { "id": "shape:abc123", "typeName": "shape", ... } },
    { "state": { "id": "page:page", "typeName": "page", ... } }
  ],
  "schema": { ... }
}

New Format (Automerge Sync)

{
  "store": {
    "shape:abc123": { "id": "shape:abc123", "typeName": "shape", ... },
    "page:page": { "id": "page:page", "typeName": "page", ... }
  },
  "schema": { ... }
}

Conversion Process

The conversion happens automatically when a document is loaded from R2. The AutomergeDurableObject.getDocument() method detects the format and converts it:

1. Automerge Array Format: Detected by Array.isArray(rawDoc)

   • Converts via convertAutomergeToStore()
   • Extracts record.state and uses it as the store record

2. Store Format: Detected by rawDoc.store existing

   • Already in correct format, uses as-is
   • No conversion needed

3. Old Documents Format: Detected by rawDoc.documents existing but no store

   • Converts via migrateDocumentsToStore()
   • Maps doc.state.id to store[doc.state.id] = doc.state

4. Shape Property Migration: After format conversion, all shapes are migrated via migrateShapeProperties()

   • Ensures required properties exist (x, y, rotation, isLocked, opacity, meta, index)
   • Moves w/h from top-level to props for geo shapes
   • Fixes richText structure
   • Preserves custom shape properties

Validation & Error Handling

The conversion functions now include comprehensive validation:

• Missing state.id: Skipped with warning
• Missing state.typeName: Skipped with warning
• Null/undefined records: Skipped with warning
• Invalid ID types: Skipped with warning
• Malformed shapes: Fixed during shape migration

All validation errors are logged with detailed statistics.

Custom Records

Custom record types (like obsidian_vault:) are preserved during conversion:

• Tracked during conversion
• Verified in logs
• Preserved in the final store

Custom Shapes

Custom shape types are preserved:

• ObsNote
• Holon
• FathomMeetingsBrowser
• HolonBrowser
• LocationShare
• ObsidianBrowser

All custom shape properties are preserved during migration.

Logging

The conversion process logs comprehensive statistics:

📊 Automerge to Store conversion statistics:
  - total: Number of records processed
  - converted: Number successfully converted
  - skipped: Number skipped (invalid)
  - errors: Number of errors
  - customRecordCount: Number of custom records
  - errorCount: Number of error details

Similar statistics are logged for:

• Documents to Store migration
• Shape property migration

Testing

Test Edge Cases

Run the test script to verify edge case handling:

npx tsx test-data-conversion.ts

This tests:

• Missing state.id
• Missing state.typeName
• Null/undefined records
• Missing state property
• Invalid ID types
• Custom records
• Malformed shapes
• Empty documents
• Mixed valid/invalid records

Test with Real R2 Data

To test with actual R2 data:

1. Check Worker Logs: When a document is loaded, check the Cloudflare Worker logs for conversion statistics

2. Verify Data Integrity: After conversion, verify:

   • All shapes appear correctly
   • All properties are preserved
   • No validation errors in TLDraw
   • Custom records are present
   • Custom shapes work correctly

3. Monitor Conversion: Watch for:

   • High skip counts (may indicate data issues)
   • Errors during conversion
   • Missing custom records
   • Shape migration issues

Migration Checklist

• Format detection (Automerge array, store format, old documents format)
• Validation for malformed records
• Error handling and logging
• Custom record preservation
• Custom shape preservation
• Shape property migration
• Comprehensive logging
• Edge case testing

Troubleshooting

High Skip Counts

If many records are being skipped:

1. Check error details in logs
2. Verify data format in R2
3. Check for missing required fields

Missing Custom Records

If custom records are missing:

1. Check logs for custom record count
2. Verify records start with expected prefix (e.g., obsidian_vault:)
3. Check if records were filtered during conversion

Shape Validation Errors

If shapes have validation errors:

1. Check shape migration logs
2. Verify required properties are present
3. Check for w/h in wrong location (should be in props for geo shapes)

Backward Compatibility

The conversion is backward compatible:

• Old format documents are automatically converted
• New format documents are used as-is
• No data loss during conversion
• All properties are preserved

Future Improvements

Potential improvements:

1. Add migration flag to track converted documents
2. Add backup before conversion
3. Add rollback mechanism
4. Add conversion progress tracking for large documents