Debug This Architecture

Here's a comprehensive analysis of the architecture's failure modes, race conditions, and bottlenecks, with specific solutions and trade-offs:

---

1. Sync Strategy: Client Timestamps + Last-Write-Wins (LWW)

Failure Mode/Race Condition:

• Clock Skew: Client clocks are unreliable (e.g., user's laptop time off by minutes). User A (correct time) edits at 10:00, User B (clock 5 min fast) edits at 10:01 → B's change overwrites A's even if A edited later in real time. Data loss guaranteed.
• Simultaneous Edits: Two users edit the same paragraph within the same millisecond (e.g., "Hello" → "Hella" vs. "Helmo"). LWW arbitrarily discards one change.
• WebSocket Polling Gap: Changes from Server 1 take up to 2 seconds to reach Server 2 via polling. User on Server 2 might overwrite Server 1's changes during this gap.

Solution: Operational Transformations (OT) or CRDTs

• Replace LWW with OT (like Google Docs) or Conflict-Free Replicated Data Types (CRDTs).
  • OT: Servers transform concurrent operations (e.g., "insert 'x' at pos 3" becomes "insert 'x' at pos 4" if another insert happened earlier).
  • CRDT: All edits are commutative (e.g., assign unique IDs to characters).
• Mandatory server-side timestamps (not client clocks) for ordering.

Trade-offs:

• ✅ Eliminates data loss from clock skew/simultaneous edits.
• ⚠️ Complexity: OT requires intricate server logic; CRDTs increase payload size (unique IDs). Frontend must handle transformations (adds ~10-20ms latency per op).
• ⚠️ State Management: Servers must track document state (not just snapshots). Requires Redis/Memcached for operational history.

---

2. WebSocket Architecture: Per-Server Broadcasts + Polling

Failure Mode/Race Condition:

• Inter-Server Sync Delay: Changes from Server 1 take 2 seconds (polling interval) to reach Server 2. Clients on Server 2 see stale data, leading to overwrites (e.g., User 2 edits based on outdated content).
• Server Failure: If Server 1 crashes, clients connected to it lose:
  • Unpersisted changes (since last WebSocket ack)
  • Real-time updates until reconnected (reconnection may hit a different server).
• Uneven Load: "Hot" documents (e.g., CEO's memo) concentrate on one server due to round-robin load balancing, causing hot partitions.

Solution: Dedicated Pub/Sub Layer for Real-Time Sync

• Replace polling with Redis PubSub or RabbitMQ. When Server 1 processes a change:
  1. Persist to DB
  2. Publish event to doc:{id}:changes channel
  3. All servers subscribe → broadcast to their WebSocket clients instantly.
• Use sticky sessions (load balancer routes same user to same server) to reduce reconnect churn.

Trade-offs:

• ✅ Near-instant inter-server sync (sub-100ms), eliminating 2s lag.
• ⚠️ Single Point of Failure (SPOF): Redis/RabbitMQ must be HA (Redis Sentinel/Cluster). Adds operational complexity.
• ⚠️ Message Loss: If a server crashes mid-broadcast, clients miss updates. Mitigation: Use persistent queues (e.g., RabbitMQ) with client sequence numbers for catch-up.
• ⚠️ Sticky sessions reduce cross-server sync but cause uneven load if a server fails (reconnecting users flood healthy servers).

---

3. Storage: Full HTML Snapshots Every 30s

Failure Mode/Bottleneck:

• Data Loss: Up to 30s of work lost on crash (browser/app/server failure).
• Database Bloat: Storing full HTML (not diffs) wastes space. A 1MB doc × 100 edits/hour = 300MB/day/doc.
• Write Contention: Frequent full-document writes for active docs cause PostgreSQL lock contention (especially with many concurrent editors).
• Inefficient Reads: Loading a large doc requires fetching multi-MB HTML from DB, slowing initial load.

Solution: Incremental Deltas + Incremental Saves

• Store only operational transforms (OTs/CRDTs) in DB:
  • Each WebSocket change → append a compact delta (e.g., {"op": "insert", "pos": 12, "chars": "x"}).
  • Persist deltas immediately (not snapshots).
• Periodically (e.g., 5 mins) generate a compacted snapshot (current doc state) for faster loading.
• Use document versioning (e.g., version: 123) to ensure clients replay deltas in order.

Trade-offs:

• ✅ Near-zero data loss (deltas persisted in <100ms).
• ✅ 10-100x smaller storage (deltas vs. full HTML).
• ⚠️ Complex Recovery: Loading a doc requires replaying all deltas from snapshot. Mitigation: Store snapshots at fixed intervals (e.g., every 100 deltas).
• ⚠️ DB Write Load: High-write volume for active docs. Mitigation: Use write-behind cache (Redis sorted sets for deltas → batch to PostgreSQL).

---

4. Auth: JWT in localStorage + 24h Expiry

Failure Mode:

• XSS Vulnerability: localStorage is accessible via JavaScript → stolen tokens enable session hijacking.
• Silent Expiry: User works for 24h, token expires mid-edit → unsaved changes lost on refresh.
• No Revoke Mechanism: Compromised tokens remain valid for 24h.

Solution: HttpOnly Refresh Tokens + Short-Lived Access Tokens

• Store access tokens (15-min expiry) in memory (not localStorage).
• Store refresh tokens (24h) as HttpOnly cookies (XSS-proof).
• On token expiry:
  1. Frontend requests new access token via /refresh (using refresh token cookie).
  2. If refresh fails, prompt login without losing unsaved changes (save to IndexedDB).

Trade-offs:

• ✅ Blocks XSS token theft (HttpOnly cookies inaccessible to JS).
• ✅ Revocable sessions: Invalidate refresh tokens server-side instantly.
• ⚠️ CSRF Risk: HttpOnly cookies need CSRF tokens (e.g., SameSite=Strict + anti-CSRF header).
• ⚠️ Increased Complexity: Frontend must handle token refresh mid-edit (requires queuing unsent WebSocket messages).

---

5. Scaling Bottlenecks

a) PostgreSQL Polling Overhead

Bottleneck: With N servers, each polling every 2s for all documents → O(N²) DB load. At 100 servers, 50 QPS/server = 5,000 QPS for change checks alone.
Solution: Event-Driven Change Propagation (via Pub/Sub, as in #2). Eliminates polling entirely.
Trade-off: Shifts load from DB to Pub/Sub layer (easier to scale than PostgreSQL).

b) Document Partitioning by Org ID

Bottleneck: Org with 10k active users (e.g., "Acme Corp") becomes a hot partition. One PostgreSQL shard handles all Acme's docs → write saturation.
Solution: Composite Sharding Key (org_id + doc_id_hash % 100).

• Distributes docs within an org across shards (e.g., 100 shards per org).
• Alternative: Use CockroachDB (distributed SQL) for automatic sharding.

Trade-off: Cross-doc transactions (e.g., "move doc between orgs") become complex (2PC required).

c) CDN Caching API Responses

Bottleneck: CloudFront caches GET responses for 5 mins → stale document reads after edits. User refreshes → sees 5-min-old data.
Solution:

• Cache-bust GET requests with document version: /docs/{id}?v=123.
• Never cache WebSocket endpoints or mutation APIs (POST/PUT).
• Set Cache-Control: no-store for document content APIs.

Trade-off: Slightly higher origin load (but protects data consistency).

---

6. Operational Risks

a) Deployment Downtime

Failure Mode: Rolling updates disconnect all WebSockets → mass reconnections flood new servers.
Solution:

• Graceful shutdown: Stop accepting new WS connections, wait 30s for in-flight ops to drain before killing server.
• Client-side retry: Exponential backoff for reconnections (max 30s).

b) Document Corruption During Failover

Failure Mode: Server crashes mid-delta write → partial data in DB.
Solution: Atomic Delta Appends

• Store deltas as immutable records (no updates).
• Use PostgreSQL INSERT ... RETURNING version to ensure version continuity.

---

Critical Summary of Fixes

Issue	Top Solution	Critical Trade-off
Sync Strategy	OT/CRDTs + server timestamps	Frontend complexity; ~15ms added latency
Inter-Server Sync	Redis PubSub for real-time events	Adds SPOF (requires Redis HA)
Storage Efficiency	Delta-only persistence + snapshots	Recovery requires delta replay
Auth Security	HttpOnly refresh tokens + short JWTs	CSRF protection needed
Scaling Hot Orgs	Composite sharding (org_id+doc_id)	Cross-doc ops become distributed transactions

---

Why This Matters

• Data loss is unacceptable in collaborative editors. LWW with client timestamps fails catastrophically under real-world conditions (clock skew, network delays). OT/CRDTs are non-negotiable for production systems (Google Docs, Figma, and Dropbox Paper all use OT).
• Polling is obsolete for real-time sync – PubSub is standard (e.g., Slack, Discord). The 2s delay breaks the illusion of real-time collaboration.
• Full-document snapshots are a legacy anti-pattern; all modern systems (e.g., Google Drive) use operational transforms.

Final Recommendation: Prioritize OT/CRDTs and PubSub first – they address the highest-risk data loss scenarios. Without these, the system is fundamentally unreliable for concurrent editing. The other fixes (storage, auth, scaling) are critical but secondary to core sync integrity.