Debug This Architecture

Critical Issues in Proposed Architecture

1. WebSocket Broadcast Isolation

Issue: Each server only broadcasts to its own WebSocket clients → users connected to different servers won't see real-time updates from each other.

• Severity: Critical - breaks real-time collaboration
• Race Condition: User A (server 1) and User B (server 2) editing same document won't see each other's changes in real-time

Solution: Use Redis Pub/Sub for cross-server WebSocket broadcasting

// When server receives change:
redis.publish(`doc:${docId}`, changeEvent);
// All servers subscribe to channel:
redis.subscribe(`doc:${docId}`, (change) => broadcastToLocalClients(change));

Trade-off: Adds network hop latency (~1-2ms), requires Redis cluster for high availability

2. Client Clock Synchronization

Issue: Last-write-wins using client timestamps is fundamentally broken

• Severity: Critical - data loss guaranteed
• Race Condition: User with wrong system clock (even seconds off) can overwrite newer changes

Solution: Use server-generated monotonic sequence numbers

-- PostgreSQL sequence per document:
CREATE SEQUENCE doc_123_version_seq;
-- Each change: nextval('doc_123_version_seq')

Trade-off: Requires database round-trip before broadcasting (~5-10ms added latency)

3. Database Polling Bottleneck

Issue: 2-second polling interval across all servers will overwhelm database at scale

• Severity: High - doesn't scale beyond ~50 servers
• Bottleneck: Each server polling all documents → O(n²) query load

Solution: Replace polling with PostgreSQL LISTEN/NOTIFY

-- Server listens for document changes:
LISTEN doc_changes_123;
-- On change:
NOTIFY doc_changes_123, '{"version": 456}';

Trade-off: PostgreSQL connection limit (~500-1000 connections), requires connection pooling strategy

4. Full HTML Snapshot Storage

Issue: Saving entire document every 30 seconds wastes storage/bandwidth

• Severity: Medium - scales poorly with large documents
• Example: 10MB document × 100 edits/min = 20GB/hour of redundant data

Solution: Operational Transform (OT) or Conflict-free Replicated Data Types (CRDTs)

• Store only diffs/operations
• Reconstruct snapshots periodically (e.g., every 100 ops)

// Store operations instead of full HTML:
{
  "op": "insert",
  "pos": 42,
  "text": "new text",
  "version": 123
}

Trade-off: Complex implementation, requires operation history cleanup strategy

5. JWT in localStorage Security Risk

Issue: localStorage vulnerable to XSS attacks

• Severity: High - token theft leads to account compromise
• Risk: Any XSS vulnerability exposes all user tokens

Solution: HttpOnly cookies + CSRF tokens

• Set httpOnly; secure; sameSite=strict flags
• Include CSRF token in API requests

// Frontend includes CSRF token in headers
axios.defaults.headers.common['X-CSRF-Token'] = getCSRFToken();

Trade-off: More complex auth flow, requires sticky sessions or JWT in cookies

6. API Response Caching

Issue: Caching API responses for 5 minutes breaks real-time collaboration

• Severity: High - users see stale document versions
• Example: CDN serves 5-minute-old document while real-time changes occur

Solution:

1. No caching for document endpoints: Cache-Control: no-store
2. Use ETags for conditional requests: If-None-Match: "version123"
3. Cache only static metadata: User profiles, organization info Trade-off: Higher origin load, but necessary for consistency

7. WebSocket Connection State

Issue: Server crash loses all WebSocket connections

• Severity: Medium - poor user experience during deployments/outages
• Failure Mode: Load balancer round-robin → users randomly disconnected

Solution:

1. Sticky sessions: Configure load balancer for WebSocket affinity
2. Connection draining: Graceful shutdown signaling
3. Client reconnection logic with exponential backoff

// Client-side reconnection:
function reconnect() {
  const delay = Math.min(1000 * 2 ** attempts, 30000);
  setTimeout(connectWebSocket, delay);
}

Trade-off: Sticky sessions reduce load distribution effectiveness

8. Database Partitioning Strategy

Issue: Partitioning only by organization ID leads to hotspots

• Severity: Medium - large organizations overwhelm single partitions
• Example: Google (organization) vs small startup have same partitioning

Solution: Composite partitioning key

-- Partition by (organization_id, document_id_hash)
PARTITION BY HASH((organization_id::text || document_id::text)::uuid)

Trade-off: More complex queries for cross-organization operations (admin views)

9. Conflict Resolution Blind Spots

Issue: Last-write-wins loses intermediate changes

• Severity: High - collaborative editing needs merge semantics
• Example: User A types "Hello", User B types "World" at same position → one loses

Solution: Implement Operational Transform (OT) with central server sequencing

// Server as single sequencer:
class OTServer {
  applyOperation(doc, operation) {
    const transformed = OT.transform(operation, this.pendingOps);
    this.history.push(transformed);
    return transformed;
  }
}

Trade-off: Significant implementation complexity, requires undo/redo handling

10. Scaling Bottlenecks

Issue: Single points of failure in each component

• Severity: High - no redundancy described

Solution: Multi-AZ deployment with failover

Component      Redundancy Strategy
-------------  --------------------
PostgreSQL     Multi-AZ RDS with read replicas
Redis          Redis Cluster (6 nodes, 3 master/3 slave)
WebSocket      Redis Pub/Sub + auto-scaling group
Load Balancer  Multi-AZ ALB/NLB

Trade-off: 2-3x higher infrastructure cost

11. Missing Monitoring & Alerting

Issue: No visibility into system health

• Severity: Medium - failures go undetected

Solution: Implement observability stack

• Metrics: WebSocket connections per server, operation latency, conflict rate
• Logging: Structured JSON logs with request IDs
• Alerting: PagerDuty/OpsGenie for SLO violations

# SLO example:
availability: 99.95% over 30 days
latency: p95 < 200ms for document operations

Trade-off: Operational overhead, but essential for production

Recommended Architecture Changes

1. Immediate Fixes (Week 1):

   • Replace client timestamps with server sequence numbers
   • Implement Redis Pub/Sub for cross-server broadcasting
   • Remove API response caching for document endpoints

2. Short-term (Month 1):

   • Implement OT/CRDTs instead of last-write-wins
   • Move JWT to HttpOnly cookies
   • Add PostgreSQL LISTEN/NOTIFY to replace polling

3. Long-term (Quarter 1):

   • Implement comprehensive observability
   • Multi-region deployment for geographic redundancy
   • Advanced conflict resolution with rich text awareness

This architecture can support ~10,000 concurrent editors with these changes, scaling to ~100,000 with additional Redis clustering and database optimization.