Debug This Architecture

This architecture has several critical flaws that would manifest under real-world usage, especially with multiple concurrent users. Below is a systematic analysis of failure modes, race conditions, and scaling bottlenecks—each with concrete solutions and trade-offs.

---

🔴 1. Last-Write-Wins (LWW) with Client Clocks → Data Corruption & Silent Overwrites

Problem

• Client clocks are unsynchronized (e.g., NTP drift, manual settings). A user editing at 10:00:00.000 (client A) may actually be later than another user’s 10:00:00.050 edit (client B) in real time—but due to clock skew, LWW picks the earlier edit as "winner".
• Result: Silent data loss. User A’s edit is overwritten with no visibility.
• Even if clocks were synced, concurrent edits to the same byte range cause corruption (e.g., overlapping insertions/deletions).

Solution

• Replace LWW with Operational Transformation (OT) or CRDTs:
  • OT (e.g., ShareDB, Yjs): Transform each operation against prior ops to ensure convergence. Best for structured text.
  • CRDTs (e.g., Yjs with YATA): Mathematically guaranteed convergence. Better for offline editing & simpler conflict resolution.
• If LWW must be kept: Use server-generated monotonic counters (e.g., PostgreSQL SERIAL or pg_advisory_xact_lock on document ID) instead of client timestamps.

Trade-offs

Approach	Pros	Cons
OT	Strong consistency, human-intuitive merge	Complex to implement; requires server-side OT engine
CRDT (Yjs/YATA)	Converges offline; simple server logic	Larger payloads; less intuitive for non-text (e.g., images)
Server monotonic counters	Simple; avoids clock skew	Still overwrites within same counter (e.g., two ops at counter=100)

✅ Recommendation: Use Yjs (CRDT-based) for real-time ops, with PostgreSQL for persistence. Store only final document state in DB; ops live in Redis stream for replay.

---

🔴 2. WebSocket Broadcast Scope → Partial Updates & Stale Clients

Problem

• Step 4 says: "Other servers poll PostgreSQL every 2 seconds for changes". But Step 3 only broadcasts to clients on the same server. Clients connected to other servers may wait up to 2s for updates—causing:
  • Stale view: User A sees their own edits instantly, but User B’s edits lag.
  • Race in broadcast: If Server 1 broadcasts op1 at t=0, Server 2 polls at t=1.9s, but op2 arrives at t=2.0s (just after polling), op2 may be missed → divergence.
  • Duplicate processing: Polling may re-read ops if not marked as "broadcast".

Solution

• Use Redis Pub/Sub or Streams for cross-server broadcast:
  • When Server X receives an op via WebSocket, publish it to Redis channel doc:{id}.
  • All servers subscribe to their document channels. On receive, broadcast locally to their WebSocket clients.
  • Avoid polling entirely for real-time sync.
• Add op IDs (e.g., op_id = doc_id + server_timestamp + random) to dedupe.

Trade-offs

Approach	Pros	Cons
Redis Pub/Sub	Real-time, low latency, no polling	Adds Redis dependency; pub/sub is fire-and-forget (ops lost on restart)
Redis Streams + Consumer Groups	Persistent, replayable, acks	More complex; requires offset management

✅ Recommendation: Use Redis Streams with XADD doc:{id} * {op_json} + consumer groups per server. Each server consumes and broadcasts ops. Commit to PostgreSQL after successful broadcast to avoid inconsistency.

---

🔴 3. Full-Document HTML Snapshots → Scalability & Sync Overhead

Problem

• Saving full HTML every 30s is wasteful:
  • Bandwidth: A 100KB doc → 3KB ops/sec vs. 100KB every 30s = 33x more data.
  • Sync latency: Clients see changes only after snapshot (up to 30s lag).
  • Conflict risk: Two snapshots in 30s overwrite each other (even with OT/CRDT ops, snapshots break continuity).
• Polling every 2s for all docs (not just active ones) → DB overload at scale.

Solution

• Store only ops in Redis/PostgreSQL, not snapshots:
  • Keep op log in Redis Stream (low-latency) + PostgreSQL for durability.
  • Reconstruct latest state on demand (e.g., via a materialized view or background worker).
• Snapshot only for archival/backups (e.g., hourly), not sync.
• Use change tracking (e.g., Yjs’ Y.applyUpdate) to sync only diffs.

Trade-offs

Approach	Pros	Cons
Op log only	Minimal data, real-time sync, supports offline	Rehydration requires replaying all ops (slow for long docs)
Hybrid: Snapshot + op log	Fast read, small ops	Sync complexity: clients need both snapshot + ops to catch up

✅ Recommendation:

• Store Yjs updates (binary diffs) in Redis Stream.
• Take hourly snapshots in PostgreSQL (document_snapshots table).
• On connect, server sends: snapshot + ops_since_snapshot_timestamp.

---

🔴 4. JWT in localStorage → Security & Session Management Risks

Problem

• localStorage is vulnerable to XSS (e.g., stolen tokens).
• No way to revoke tokens before expiry (24h).
• No session invalidation on logout or suspicious activity.

Solution

• Use short-lived access tokens (5–15 min) + refresh tokens:
  • Access token: JWT in memory (e.g., React state), not localStorage.
  • Refresh token: HttpOnly, Secure cookie (immutable to JS).
• Store refresh tokens in Redis with TTL to enable revocation.
• Implement token rotation on refresh.

Trade-offs

Approach	Pros	Cons
Access + refresh tokens	Secure, revocable, scalable	More complex auth flow; requires token refresh logic
Long-lived JWT in localStorage	Simple	Vulnerable to XSS; no revocation

✅ Recommendation: Use OAuth2-style flow with refresh tokens. For real-time WebSocket auth, pass access token in Authorization header during handshake.

---

🔴 5. Load Balancer + Per-Server WebSocket State → Session Affinity Lost

Problem

• WebSocket connections are stateful (e.g., user session, pending ops).
• With round-robin LB, a client’s 2nd connection may hit a different server that has no context.
• Result: Lost ops, duplicate broadcasts, or auth errors.

Solution

• Enable sticky sessions (session affinity) on the LB.
  • Or better: Move session state to Redis:
    • On WebSocket connect, store client_id → server_id mapping in Redis.
    • When LB routes to wrong server, it redirects to correct server (or fetches session state).
• Alternative: Use Yjs + ShareDB which handles session state in Redis.

Trade-offs

Approach	Pros	Cons
Sticky sessions	Simple	Breaks scaling (can’t rebalance servers); single point of failure if server dies
Redis-backed session	Scalable, fault-tolerant	Adds Redis dependency; session sync latency

✅ Recommendation: Use Redis to track active WebSocket sessions (HSET websocket:sessions client_id server_id). When server A receives op for client X, it checks Redis and forwards to server B if needed.

---

🔴 6. Document Partitioning by Org ID → Hotspotting & Imbalanced Load

Problem

• Partitioning by org_id causes hotspots:
  • Large orgs (e.g., enterprise) dominate load.
  • Small orgs underutilize capacity.
• Read replicas won’t help if writes are concentrated.

Solution

• Use document-level sharding with consistent hashing:
  • Shard key: doc_id (hashed via doc_id % N → shard).
  • Or use Redis Cluster for op log sharding.
• Dynamic rebalancing: Add/remove shards as load changes.
• For PostgreSQL: Use pg_partman to partition by doc_id range or hash.

Trade-offs

Approach	Pros	Cons
Org-based partitioning	Simple, co-locate org data	Hotspots, poor utilization
Doc-based partitioning	Balanced load, horizontal scaling	Cross-doc queries harder; more complex routing

✅ Recommendation: Partition by doc_id, use a shard router service to map doc_id → shard. Cache mappings in Redis.

---

🔴 7. No Backpressure → System Overload During Spikes

Problem

• If 10k users type simultaneously, PostgreSQL + Redis get flooded.
• WebSocket buffers fill → dropped messages → data loss.

Solution

• Add backpressure at multiple layers:
  • WebSocket layer: Throttle ops (e.g., 100 ops/sec per user) with XADD rate limiting in Redis.
  • Database: Use connection pooling (e.g., pgbouncer), batch writes (e.g., 100 ops/batch).
  • Server layer: Drop low-priority ops during overload (e.g., formatting changes), keep text ops.
• Implement circuit breakers (e.g., Hystrix-style) to fail gracefully.

Trade-offs

Approach	Pros	Cons
Rate limiting	Protects backend	User sees lag; may need UI feedback
Batching writes	Reduces DB load	Increases latency (ops batched for 100ms)

✅ Recommendation: Use Redis Streams with XADD rate limiting (via INCR + EXPIRE). If rate limit exceeded, send throttle message to client.

---

🔴 8. CDN Caching API Responses → Stale Document States

Problem

• CDN caches API responses for 5 min (e.g., /api/docs/{id}). After a user edits, others see stale content for up to 5 min.
• Critical failure: Public docs become unusable.

Solution

• Never cache document content via CDN.
  • Cache only static assets (/static/, /favicon.ico).
  • For dynamic endpoints, use:
    • Cache-Control: no-store for /api/docs/*.
    • Or use WebSocket for updates (CDN irrelevant).
• If caching is needed (e.g., for analytics), cache non-sensitive metadata (e.g., doc_meta).

Trade-offs

Approach	Pros	Cons
Disable CDN caching for docs	Fresh data	Higher origin load; slower initial load (but real-time sync compensates)
TTL-based invalidation	Simpler	Still stale during TTL window

✅ Recommendation: Set Cache-Control: private, no-store for all /api/ endpoints. Use CDN only for assets.

---

🧩 Summary of Critical Fixes

Issue	Severity	Solution
LWW with client clocks	🔴 Critical	Replace with CRDTs (Yjs) or server-monotonic ops
WebSocket broadcast scope	🔴 Critical	Redis Pub/Sub/Streams for cross-server sync
Full-document snapshots	🟠 High	Store ops only; snapshot for archival
JWT in localStorage	🟠 High	Short-lived access tokens + HttpOnly refresh tokens
No session affinity	🟠 High	Redis-backed WebSocket session tracking
Org-based partitioning	🟠 High	Document-level sharding with consistent hashing
No backpressure	🟠 High	Rate limiting, batching, circuit breakers
CDN caching API responses	🟠 High	Disable caching for /api/docs/*

---

🛠️ Revised Architecture Highlights

1. Real-time sync:
   Client → WebSocket → Server → Redis Stream (doc:ops) → [all servers] → local WebSocket clients
2. Persistence:
   Server → Batch ops → PostgreSQL (with upsert)
3. State reconstruction:
   On connect: snapshot (hourly) + ops since snapshot time
4. Scaling:
   • Shard by doc_id
   • Redis Cluster for op log
   • Read replicas for historical queries
5. Auth:
   Access token (JWT, in-memory) + refresh token (HttpOnly cookie, Redis-backed)

This design is production-grade (used by companies like Notion, Coda, and Linear). The biggest upfront cost is implementing CRDTs/OT—but libraries like Yjs reduce this to days, not months.

Let me know if you'd like a deep dive on Yjs integration, shard routing, or crisis-runbook for failure scenarios!