Debug This Architecture

This architecture for a real-time collaborative document editor has several promising elements (WebSockets, Redis, PostgreSQL, CDN), but it also contains significant flaws in consistency, scalability, and reliability. Below is a comprehensive breakdown of failure modes, race conditions, and scaling bottlenecks, along with specific solutions and their trade-offs.

---

🔴 Critical Issues

1. Client Clocks for Timestamps → Inconsistent Ordering (Major Race Condition)

• Issue: Using client local clocks for "last-write-wins" causes incorrect conflict resolution due to clock skew. Two clients may submit changes milliseconds apart but with vastly different timestamps.
• Example: User A (clock fast) edits at 13:00:05, User B (clock slow) edits at 13:00:04 → B’s change overwrites A’s even though it happened earlier.
• Failure Mode: Data loss, broken collaboration.

✅ Solution: Use Lamport Clocks or Operational Transformation (OT) / CRDTs

• Lamport Clocks: Each server assigns monotonically increasing logical timestamps. Ensures causal ordering.
• OT / CRDTs: Transform concurrent edits so they can be applied in any order. Google Docs uses OT.
• Alternative: Use server-assigned timestamps (UTC) with sequence numbers per document.

Recommended: Use CRDTs (Conflict-Free Replicated Data Types) for real-time sync (e.g., Yjs, Automerge). Best for scalability and offline support.

⚖️ Trade-offs:

• CRDTs/OT: Adds complexity to frontend and backend logic.
• Server timestamps: Simpler, but doesn’t prevent overwrites; still needs transformation semantics.
• Lamport clocks: Better than raw client timestamps, but still not sufficient for true convergence.

---

2. WebSocket Broadcast Limited to One Server → Incomplete Real-Time Sync

• Issue: Server A receives a change and broadcasts only to its own WebSocket clients. Other servers don’t receive real-time updates — they poll every 2 seconds.
• Result: Users connected to Server B see changes from Server A up to 2 seconds later → not real-time.
• Race Condition: User on Server B might edit stale data, leading to conflicts.

✅ Solution: Shared Message Bus (Redis Pub/Sub or Kafka)

• On change, server publishes to a topic like doc:123:updates.
• All API servers subscribe and forward updates to their connected WebSocket clients.
• Ensures all clients get updates in <100ms, regardless of which server they’re on.

⚖️ Trade-offs:

• Redis Pub/Sub: Simple, but messages lost if server crashes (no durability). Good for real-time, not audit.
• Kafka/Pulsar: Durable, replayable, but heavier ops overhead.
• Latency vs. Reliability: Redis Pub/Sub is faster; Kafka enables message replay.

✅ Use Redis Pub/Sub for real-time sync, add Kafka for audit log/event sourcing if needed.

---

3. Polling PostgreSQL Every 2s → High DB Load & Latency

• Issue: Each server polling every 2 seconds for changes creates:
  • O(n_servers × n_documents) queries.
  • Stale reads up to 2 seconds.
  • Full table scans or index bloat if not carefully indexed.
• Scaling Bottleneck: Becomes infeasible at 100+ servers or 10k+ active docs.

✅ Solution: Replace Polling with Push via Message Queue

• On write, server publishes change to Redis Pub/Sub or Kafka.
• Other servers consume via subscription → no DB polling.
• Use document-level channels (e.g., doc:123) for efficient routing.

⚖️ Trade-offs:

• Added complexity with message bus.
• Message loss risk with Redis Pub/Sub (but acceptable if combined with DB persistence).
• Kafka adds durability but latency.

✅ Use Redis Pub/Sub for low-latency sync, and store full history in PostgreSQL for recovery.

---

4. Last-Write-Wins on Paragraphs → Data Loss in Concurrent Edits

• Issue: Two users editing the same paragraph will have one change clobbered.
• Example: User A adds a sentence at the start; User B adds one at the end → one full paragraph overwrites the other.
• Failure Mode: Frequent data loss in collaborative environments.

✅ Solution: Fine-Grained Operational Transformation or CRDTs

• Represent edits as insert/delete operations at character positions.
• Apply transformations so both changes can be merged.
• Requires shared document state and operation coordination.

Example: Use Yjs (CRDT) on frontend and backend — syncs deltas, not full state.

⚖️ Trade-offs:

• High complexity vs. LWW.
• Memory overhead for tracking document state.
• Client SDK required (e.g., integrate Yjs into React app).

✅ CRDTs are the modern standard for collaborative editing (e.g., Figma, Google Docs).

---

5. Full HTML Snapshots Every 30 Seconds → Data Loss & Bandwidth Waste

• Issue: Only full snapshots saved → lose up to 30 seconds of work on crash.
• Also inefficient: sending full HTML for small changes.
• No version history.

✅ Solution: Delta-Based Persistence with Journaling

• Save individual operations (CRDT/OT ops) to DB.
• Reconstruct document from op log.
• Optionally, take periodic compressed snapshots for fast loading.

⚖️ Trade-offs:

• Read complexity: Replaying ops on load (can cache latest snapshot).
• Storage growth: But compressible and manageable with TTL or compaction.
• Durability: Every keystroke persisted → stronger consistency.

✅ Combine: Snapshot every 10 mins + op log. On load: get snapshot + replay recent ops.

---

6. No Global Consistency Across Servers → Stale Reads

• Issue: Each server serves its own WebSocket clients independently.
• No coordination → multiple servers may believe they control a document.
• Clients on different servers may have divergent views.

✅ Solution: Distributed Locking or Document Affinity

• Option A: Use Redis-based document leader election.
  • Only one server handles writes per document at a time.
• Option B: Route all clients editing a document to the same server (via sticky sessions or routing tier).
• Option C: Stateless servers + CRDTs: Let clients sync via op log, servers just relay.

✅ Best: Use CRDTs with stateless servers — no leader needed.

⚖️ Trade-offs:

• Sticky sessions: Breaks horizontal scalability; hard to rebalance.
• Leader election: Adds latency and single point of failure per doc.
• CRDTs: Eliminate need for coordination — preferred.

---

7. JWT in localStorage → XSS Vulnerability

• Issue: Storing JWTs in localStorage makes them accessible to client-side scripts → vulnerable to XSS attacks.
• Attack: Malicious script steals JWT → full account takeover.

✅ Solution: Use httpOnly, Secure, SameSite Cookies

• Store JWT in httpOnly cookie → inaccessible to JavaScript.
• Pair with CSRF protection (e.g., double-submit cookie or anti-CSRF token).
• Use short-lived access tokens + long-lived refresh tokens in httpOnly cookies.

⚖️ Trade-offs:

• CSRF risk: Mitigated with SameSite=Lax/Strict and anti-CSRF tokens.
• Complexity: Slightly harder to debug auth.
• Security win: Much more robust against XSS.

✅ Use httpOnly cookies + short-lived JWTs.

---

8. CDN Caching API Responses → Stale Data

• Issue: CDN (CloudFront) caches API responses for 5 minutes.
• Users may see outdated document content.
• Invalidating cache per document change is hard.

✅ Solution: Do Not Cache Real-Time Data in CDN

• CDN: Only for static assets (JS, CSS, images).
• API responses: Set Cache-Control: no-store or very short TTL.
• For metadata (e.g., user profiles), use short cache + invalidation via Redis.

⚖️ Trade-offs:

• Increased origin load.
• But necessary: Real-time data must be fresh.

✅ Disable CDN caching for all document/editing APIs.

---

9. No Message Acknowledgment or Reconnection Logic

• Issue: If WebSocket drops, changes may be lost.
• No retry mechanism or offline queue.

✅ Solution: Client-Side Change Queue with ACKs

• Client queues changes until ACK from server.
• On reconnect, replay unacknowledged ops.
• Use client-generated temp IDs for local changes.

⚖️ Trade-offs:

• Complexity in frontend.
• Risk of duplicate ops → must be idempotent (CRDTs help here).

✅ Use Yjs or similar — handles offline, sync, and conflict.

---

10. Database as Single Source of Truth → Write Bottleneck

• Issue: All writes go to PostgreSQL → becomes bottleneck at scale.
• Full HTML writes every 30s are expensive (large payloads, row locks).

✅ Solution: Write to Redis First, Then Async to PostgreSQL

• Hot path: Save ops to Redis (fast).
• Background worker: Batch and persist to PostgreSQL.
• Use Redis as primary for real-time, PostgreSQL for durability.

⚖️ Trade-offs:

• Risk of Redis data loss → use AOF + replication + periodic dump.
• Eventual consistency: Slight delay in DB persistence.

✅ Use Redis as working store, PostgreSQL as cold storage/backup.

---

11. No Document Partitioning Strategy Defined

• Issue: "Partition by org ID" is mentioned, but unclear how routing works.
• Risk of hot partitions (e.g., large orgs with 10k+ active docs).

✅ Solution: Fine-Grained Sharding (e.g., by doc ID)

• Use consistent hashing on doc_id → map to Redis cluster or DB shard.
• Or use Citus (PostgreSQL extension) for sharding.
• Combine with routing tier to direct requests.

⚖️ Trade-offs:

• Cross-shard transactions: Hard (avoid them).
• Rebalancing: Complex, but tools like Citus help.

✅ Use Citrus or custom sharding by doc_id.

---

12. No Operational Monitoring, Backpressure, or Rate Limiting

• Issue: No limits on client message rate → one user can flood server.
• No monitoring of WebSocket health, latency, or backlog.

✅ Solution: Add Rate Limiting & Metrics

• Use Redis to track per-user message rate.
• Drop/queue excess messages.
• Expose metrics (Prometheus) for WebSocket count, latency, errors.

⚖️ Trade-offs:

• Slight latency on throttled clients.
• Improved stability.

✅ Use token bucket rate limiting per user/connection.

---

✅ Summary of Key Fixes

Issue	Solution	Trade-off
Client clock skew	Use CRDTs or server-assigned logical clocks	Complexity ↑
Incomplete real-time sync	Add Redis Pub/Sub between servers	New dependency
Polling DB every 2s	Replace with message bus push	Ops complexity ↑
LWW overwrites	Use CRDTs (e.g., Yjs)	Learning curve
Full HTML snapshots	Delta + op log + snapshots	Storage ↑, consistency ↑
JWT in localStorage	Use httpOnly cookies	CSRF risk → needs mitigation
CDN caching API	Disable for dynamic routes	Higher origin load
WebSocket drops	Client queue + ACKs	Offline complexity
DB write bottleneck	Redis as write buffer	Risk of Redis data loss
No global consistency	CRDTs + message bus	Stateless, scalable
Sharding	Shard by doc_id	Cross-shard queries hard

---

✅ Recommended Final Architecture

[Client] 
  → (WebSocket + Yjs) 
  → API Server 
  → Redis (Pub/Sub for real-time, storage for ops)
  → Background Workers → PostgreSQL (persistent storage)
  → Auth via httpOnly JWT cookies
  → CDN only for static assets

Key Technologies:

• Yjs or Automerge for CRDT-based collaboration
• Redis for Pub/Sub and ephemeral state
• PostgreSQL with Citus for sharded persistence
• Node.js servers stateless, scaled horizontally
• Kafka (optional) for audit log and recovery

---

🏁 Conclusion

This system as described will not work reliably for real-time collaboration due to clock skew, incomplete sync, polling, and destructive conflict resolution.

Fix it by:

1. Replacing LWW with CRDTs (e.g., Yjs).
2. Replacing polling with Redis Pub/Sub.
3. Using delta sync instead of full snapshots.
4. Securing auth with httpOnly cookies.
5. Removing CDN caching for APIs.
6. Making state management global and consistent.

With these changes, you can build a scalable, resilient, Google Docs–like system.