Debug This Architecture

This architecture contains several critical flaws that would prevent real-time collaboration, create data consistency issues, and cause catastrophic failures at scale. Here are the specific problems and solutions:

1. The "Split-Brain" Real-Time Failure

Problem: With round-robin load balancing and server-isolated WebSocket broadcasts, users connected to different API servers won't see each other's changes in real-time. Server A broadcasts only to its clients, while Server B discovers changes by polling PostgreSQL every 2 seconds.

Race Condition: User A (Server 1) and User B (Server 2) edit simultaneously. User A sees their change immediately; User B sees it 2 seconds later. During that window, User B edits stale content, creating a conflict that appears as a "jump" when the merge happens.

Solution: Implement Redis Pub/Sub (or NATS/RabbitMQ) as a message bus between API servers.

• Server writes change to Redis Pub/ub channel doc:{id}:changes
• All servers subscribe and broadcast to their local WebSocket clients
• Eliminate PostgreSQL polling entirely

Trade-offs:

• Pros: True real-time (<50ms latency), reduces DB load by 99%
• Cons: Redis becomes a single point of failure (mitigate with Redis Cluster); adds ~5ms latency per hop

2. Clock Skew Catastrophe

Problem: Using client timestamps for "last-write-wins" is unreliable. If User A's laptop clock is 5 minutes fast (common with manual time changes or NTP failures), their edits permanently overwrite User B's concurrent edits, even if B typed later.

Race Condition:

T+0: User B (correct time) edits paragraph
T+1: User A (clock +5min) edits same paragraph  
T+2: Server receives B's edit (timestamp later than A's local time)
T+3: A's edit "wins" despite being chronologically second

Solution: Implement Hybrid Logical Clocks (HLC) or server-assigned monotonic sequence numbers.

• Server assigns version = max(server_timestamp, client_timestamp) + 1
• Use Operational Transform (OT) or CRDTs (Yjs/Automerge protocol) instead of last-write-wins
• Conflict resolution at character/operational level, not paragraph level

Trade-offs:

• Pros: Correct ordering regardless of client clock drift; supports offline editing
• Cons: OT requires complex server-side transformation logic; CRDTs increase memory usage (2-3x document size)

3. Database Write Amplification (Scaling Bottleneck)

Problem: Writing every keystroke to PostgreSQL creates an O(n²) write storm. With 1000 concurrent users typing 3 chars/second = 3000 writes/sec, plus read replicas lagging, causing replication delay.

Failure Mode: During traffic spikes, PostgreSQL connection pool exhaustion causes cascading failures. The 2-second polling from N servers creates N/2 queries per second per document.

Solution: Implement Event Sourcing with Kafka + In-Memory CRDT State.

• Buffer operations in Redis Streams (sorted by HLC)
• Flush compressed operation logs to PostgreSQL every 5 seconds (batch insert)
• Keep active document state in Redis (not PostgreSQL)
• PostgreSQL becomes the cold storage/audit log, not the hot path

Trade-offs:

• Pros: Supports 100k+ concurrent editors; sub-10ms persistence
• Cons: Risk of 5-second data loss on Redis failure (mitigate with Redis AOF persistence every second)

4. CDN Cache Poisoning

Problem: CloudFront caching API responses for 5 minutes means:

• User sees 5-minute stale document state on refresh
• Authenticated content may be cached and served to wrong users (if cache key doesn't include JWT)
• WebSocket auth handshake may fail if CDN intercepts the upgrade request

Solution: Disable caching for all /api/* and /ws/* routes. Use CDN only for static assets (React bundle, CSS, images). Implement separate domains: static.example.com (CDN) vs api.example.com (no cache).

Trade-offs:

• Pros: Data consistency, security
• Cons: Higher origin server load (mitigate with the Redis hot-path above)

5. Security: XSS via localStorage JWT

Problem: Storing JWT in localStorage makes it vulnerable to XSS attacks. A malicious script can steal the token and impersonate the user for 24 hours.

Solution: Use HttpOnly, Secure, SameSite=Strict cookies for the session ID.

• Short-lived access tokens (5 min) in memory
• Long-lived refresh tokens (24h) in HttpOnly cookie
• Implement CSRF tokens for non-WebSocket HTTP endpoints

Trade-offs:

• Pros: Immunity to XSS token theft
• Cons: Slightly more complex auth flow; requires /refresh endpoint

6. WebSocket Sticky Session Failures

Problem: Round-robin load balancing without sticky sessions means:

• If Server 1 crashes, all its clients reconnect simultaneously to random servers
• Server 2 might receive 10k reconnection attempts instantly (thundering herd)
• No guarantee users reconnect to servers holding their document state (if any)

Solution: Implement IP Hash or Cookie-based sticky sessions on the load balancer.

• Use Redis Session Store to share connection metadata across servers
• Implement exponential backoff with jitter on client reconnection logic

Trade-offs:

• Pros: Even load distribution during failures
• Cons: Slight imbalance in server utilization; requires load balancer support

7. Snapshot Inconsistency & Data Loss

Problem: Full HTML snapshots every 30 seconds create:

• Write amplification: 1MB document × 1000 users = 1GB/minute write throughput
• Consistency issues: If snapshot fails mid-write, document is corrupted
• Conflict ambiguity: HTML doesn't preserve edit intention (e.g., "bold this word" vs "replace entire paragraph")

Solution: Store operation logs (deltas) not snapshots.

• Use ProseMirror or Quill operation format (retain/delete/insert)
• Compress operations with gzip before PostgreSQL storage
• Generate snapshots asynchronously via background workers

Trade-offs:

• Pros: 90% storage reduction; full edit history for audit/undo
• Cons: Requires replaying operations to reconstruct document (mitigate with periodic snapshots every 5 minutes, not 30 seconds)

8. The "Ghost Edit" Failure Mode

Problem: If Server 1 crashes after writing to PostgreSQL but before broadcasting via Redis Pub/Sub, the edit is persisted but never reaches other users. They continue editing an old version, creating a "fork" in the document history.

Solution: Implement Server-Sent Events (SSE) for critical updates + Vector Clocks for version tracking.

• Clients maintain a vector clock of seen operations
• On reconnection, client sends last_seen_version to server
• Server replays missing operations from Redis Streams (which persists for 24h)

Trade-offs:

• Pros: Guaranteed eventual consistency; automatic conflict repair
• Cons: Increased memory usage for operation history (prune after 24h)

9. Horizontal Scaling Bottleneck

Problem: As you add more API servers, the 2-second polling interval creates a thundering herd on PostgreSQL. With 100 servers polling 1000 active documents: 100 × 1000 / 2 = 50,000 queries/second just for polling.

Solution: Reverse the polling direction. Use Redis Keyspace Notifications or PostgreSQL LISTEN/NOTIFY (if using Postgres 14+ with logical replication).

• When Server 1 writes, it publishes to doc:{id}:notify
• Other servers block on SUBSCRIBE instead of polling
• Eliminates 99% of read load

Trade-offs:

• Pros: Scales to 1000+ servers; instant notification
• Cons: Redis connection limits (use connection pooling)

10. No Operational Transform (OT) or CRDT

Problem: "Last-write-wins at paragraph level" provides terrible UX. If two users type in the same paragraph simultaneously, one user's entire edit disappears.

Solution: Integrate Yjs or Automerge protocol.

• Server acts as "awareness" broker (cursor positions) and persistence layer
• Clients sync via Yjs binary protocol over WebSocket
• Server merges updates using CRDT logic (no conflicts possible)

Trade-offs:

• Pros: Google Docs-level collaboration quality; offline support
• Cons: Adds ~200KB to client bundle; server needs 2-4x RAM to hold CRDT state

Summary of Critical Fixes

Priority	Fix	Impact
P0	Replace polling with Redis Pub/Sub	Enables real-time collaboration
P0	Remove CDN caching on API	Prevents data corruption
P0	Move JWT to HttpOnly cookies	Security compliance
P1	Implement CRDTs (Yjs)	Correct conflict resolution
P1	Buffer writes in Redis, batch to Postgres	Supports >10k concurrent users
P2	Add sticky sessions	Prevents reconnection storms

Without these changes, the system will fail under load of ~100 concurrent users due to database contention and will provide a broken collaboration experience (2-second delays, lost edits).