Debug This Architecture

Your real-time collaborative document editor architecture is well-structured for a scalable, low-latency system — but it contains several critical failure modes, race conditions, and scaling bottlenecks that could degrade user experience or cause data loss. Below is a detailed analysis of each issue with specific solutions and trade-offs.

---

✅ 1. Failure Mode: Inconsistent State Across API Servers (No Cross-Server Sync)

🔍 Problem:

• Each API server maintains its own WebSocket connections.
• Changes are only broadcasted within the same server's client pool.
• If User A connects to Server 1 and User B connects to Server 2, edits from User A won’t be seen by User B unless Server 2 polls PostgreSQL — which happens every 2 seconds (latency).
• This creates eventual consistency across servers, leading to visible lag in real-time collaboration.

🛠️ Solution: Use a Distributed Event Bus (e.g., Redis Pub/Sub)

• Have all API servers subscribe to a shared Redis channel.
• When an edit is written to the DB, publish the change event to Redis.
• All servers listen to this channel and broadcast the update to their connected clients via WebSocket, even if the originating client was on another server.

⚖️ Trade-offs:

Pros	Cons
Real-time sync across servers	Adds dependency on Redis; increased complexity
Low latency (~100ms)	Higher operational cost due to pub/sub traffic
Eliminates polling delays	Risk of message duplication (handle idempotency)

✅ Implementation Tip: Use Redis Streams instead of simple Pub/Sub for better message durability and consumer group management.

---

✅ 2. Race Condition: Client Clock Timestamps Are Unreliable

🔍 Problem:

• You rely on client-side timestamps for "last-write-wins" conflict resolution.
• Clients can manipulate clocks (bad NTP sync, malicious users).
• Two users editing simultaneously may have nearly identical timestamps → order becomes unpredictable → inconsistent results.

🛠️ Solution: Use Server-Side Monotonically Increasing Timestamps (or Vector Clocks)

• Replace client clock timestamps with server-generated unique IDs (e.g., uuidv1, or incrementing counters per document).
• Or use vector clocks / Lamport timestamps with server coordination.
• Alternatively, use CRDTs (Conflict-Free Replicated Data Types) for deterministic merge logic.

⚖️ Trade-offs:

Pros	Cons
Deterministic, reliable ordering	Requires changes to data model and client logic
Eliminates clock skew issues	CRDTs increase payload size (metadata overhead)
Can enable true real-time merging	Learning curve for developers

✅ Recommended: Start with Lamport timestamps + server-side validation, then migrate to CRDTs (like Yjs) for richer collaboration.

---

✅ 3. Scaling Bottleneck: PostgreSQL Polling Every 2 Seconds

🔍 Problem:

• Every API server polls PostgreSQL every 2 seconds for updates.
• With many documents and servers, this generates high query volume (~500–1000 queries/sec per server).
• Can overwhelm the database, especially under high load.

🛠️ Solution: Use Database Notifications (PostgreSQL LISTEN/NOTIFY)

• Enable LISTEN on the document table via triggers.
• When a change is committed, fire a NOTIFY document_updated, 'doc_id'.
• All API servers subscribe to these notifications using pg_notify listeners.
• No polling needed — near-instantaneous event delivery.

⚖️ Trade-offs:

Pros	Cons
Near-zero latency notification	Requires active listeners (add complexity)
Zero polling overhead	Single point of failure if listener dies
Efficient for high-frequency events	Needs monitoring & restart mechanisms

✅ Bonus: Combine with Redis Pub/Sub as fallback if PostgreSQL notification drops occur.

---

✅ 4. Failure Mode: Document Partitioning by Org ID ≠ Load Distribution

🔍 Problem:

• Document partitioning by organization ID helps scalability, but:
  • Some orgs grow much faster than others (e.g., 10k docs vs. 10).
  • This causes hot partitions, where one shard (org) saturates its API server(s), while others sit idle.
  • Worse: a large org might outgrow a single server.

🛠️ Solution: Dynamic Sharding Based on Document Count/Activity

• Use shard key = hash(org_id + doc_id) instead of just org_id.
• Implement shard-aware routing:
  • A central metadata service tracks which shard hosts which document.
  • Or use a sharding proxy (e.g., Vitess) or consistent hashing ring.
• Optionally, allow hot orgs to be split into multiple shards (via migration).

⚖️ Trade-offs:

Pros	Cons
Even distribution across servers	Harder to implement; more stateful
Avoids hotspots	Requires complex routing logic
Scales better over time	Increased latency for cross-shard ops

✅ Alternative: Use multi-tier sharding: org_id → tenant shard → document hash within shard.

---

✅ 5. Race Condition: Concurrent Edits Without Delta Tracking

🔍 Problem:

• You're storing full HTML snapshots every 30 seconds.
• But edits are sent as raw text deltas via WebSocket.
• If two users edit different parts of the document, and both send changes before the snapshot is saved, you risk:
  • Overwriting each other’s changes.
  • Losing intermediate state during snapshot interval.

🛠️ Solution: Store Deltas + Apply Incremental Updates

• Instead of relying solely on full snapshots, maintain a delta log (list of operations).
• Use Operational Transformation (OT) or CRDTs to merge deltas safely.
• Save full snapshots only when necessary (e.g., backup, crash recovery).
• Only save deltas to DB — store full renderable state as a derived view.

⚖️ Trade-offs:

Pros	Cons
Minimizes network/data usage	Complex delta merging logic
High fidelity, low bandwidth	Can lead to bloat if not cleaned up
Enables undo/redo	Higher CPU/memory on server/client

✅ Better Choice: Use Yjs (CRDT-based) for efficient, conflict-free editing at scale.

---

✅ 6. Failure Mode: JWT Expiry & Session Reconnection Loss

🔍 Problem:

• JWT tokens expire after 24 hours and are stored in localStorage.
• If a user loses connection (network drop), reconnects, but token is expired → they must re-authenticate.
• During reconnection, all unsent edits are lost unless cached locally.

🛠️ Solution: Local Edit Buffer + Token Refresh Mechanism

• On the client, buffer unsent edits in IndexedDB until:
  • Successful server sync OR
  • Token refreshed via silent refresh (if using refresh tokens).
• On reconnect, replay buffered edits.
• Backend should support token refresh via /refresh endpoint with long-lived refresh token (stored securely in HTTP-only cookie, not localStorage).

⚖️ Trade-offs:

Pros	Cons
Survives network/auth outages	Adds complexity to client-side state management
Prevents data loss	IndexedDB limits vary by browser
Seamless reconnection	Risk of stale edits if not properly deduplicated

✅ Critical: Never trust localStorage for sensitive tokens. Use HttpOnly cookies + CSRF protection for session integrity.

---

✅ 7. Scaling Bottleneck: CDN Caches API Responses for 5 Minutes

🔍 Problem:

• CDN caches API responses (e.g., /docs/:id) for 5 minutes.
• But the document content changes frequently in real-time.
• Users may see outdated content despite live WebSocket updates.
• Worse: CDN cache invalidation is slow and manual.

🛠️ Solution: Cache-Control Strategy & Cache Invalidation

• Set Cache-Control: no-cache, must-revalidate on dynamic APIs.
• Use cache tags (via CloudFront Origin Response Lambda) to tag responses by doc_id.
• When a write occurs, invalidate only that doc’s cache via AWS API call.
• Or, avoid caching dynamic content entirely — cache only static assets (JS/CSS/images).

⚖️ Trade-offs:

Pros	Cons
Fresh content always served	Reduced CDN efficiency
Real-time accuracy	More API requests hitting origin
Fast cache invalidation	Costly if done too often

✅ Best Practice: Cache only immutable static assets. Keep dynamic endpoints uncached or short-lived.

---

✅ 8. Failure Mode: Single Point of Failure in Redis (for Pub/Sub)

🔍 Problem:

• Redis is used for session cache AND WebSocket event broadcasting.
• If Redis fails, both authentication and real-time sync break.
• Also, Redis is typically single-instance unless clustered.

🛠️ Solution: Redis Cluster + Fallback to DB

• Deploy Redis Cluster (3+ nodes) with replication and failover.
• Implement fallback mechanism: if Redis is down, temporarily persist events to PostgreSQL and retry later.
• Use circuit breakers to prevent cascading failures.

⚖️ Trade-offs:

Pros	Cons
High availability	Higher cost and operational complexity
Fault tolerance	More difficult to debug
Self-healing	Latency spikes during failover

✅ Use Case: Redis Sentinel or Redis Cluster depending on scale and budget.

---

📊 Summary Table: Key Issues & Solutions

Issue	Solution	Trade-off
Cross-server sync delay	Redis Pub/Sub + Notify	Added complexity, cost
Client clock timestamp attacks	Server-generated Lamport/UUID	Payload increase, logic change
Frequent DB polling	PostgreSQL LISTEN/NOTIFY	Listener maintenance, failure handling
Hot partitions	Dynamic sharding (hash-based)	Routing complexity
Delta loss during downtime	Client-side edit buffering	Storage overhead, deduplication logic
JWT expiry causing disconnect	Refresh tokens + IndexedDB buffer	Security risks if misused
CDN caching stale content	Cache control + Tag-based invalidation	Reduced CDN savings
Redis single-point failure	Redis Cluster + fallback to DB	Operational cost

---

✅ Final Recommendations

1. Replace last-write-wins with CRDTs (e.g., Yjs) → enables true real-time, conflict-free editing.
2. Switch from polling to PostgreSQL LISTEN/NOTIFY → eliminate 2-second lag.
3. Migrate auth to HttpOnly cookies with refresh tokens → better security.
4. Implement robust client-side edit buffering → resilience against network loss.
5. Use Redis Cluster + cache invalidation strategy → avoid single points of failure.
6. Avoid caching dynamic API calls → ensure real-time accuracy.

---

By addressing these issues systematically, your system can evolve from a “good enough” prototype into a production-grade, highly available, real-time collaborative editor capable of serving thousands of concurrent users without data loss or noticeable lag.