Debug This Architecture

Of course. This is an excellent exercise. The proposed architecture has several critical flaws that would prevent it from functioning as a reliable, real-time, or scalable system.

Here is a breakdown of the issues, categorized for clarity, along with specific solutions and their trade-offs.

---

Category 1: Race Conditions & Data Integrity Issues

Issue #1: Sync Strategy - "Last-Write-Wins with Client Clocks"

This is the most severe data integrity problem. Client-side clocks cannot be trusted; they drift, can be set incorrectly, or can be manipulated by a user. This will lead to definite data loss.

• Scenario:

  1. User A (Clock: 10:00:05) types "Hello World".
  2. User B (Clock: 10:00:03, slightly slow) types "Hi Earth" in the same spot, a moment after User A.
  3. The server receives User B's change first, then User A's.
  4. Because User A's timestamp (10:00:05) is "later" than User B's (10:00:03), User A's change will overwrite User B's, even though User B made their edit after User A in real-world time. This is data loss.

• Solution: Implement a proper concurrency control algorithm like Operational Transformation (OT) or use a Conflict-free Replicated Data Type (CRDT).

  • Operational Transformation (OT): This is the algorithm used by Google Docs. Edits are not "the new state" but are "operations" (e.g., insert 'a' at position 5, delete 1 char at position 12). A central server receives operations, transforms them against other recent operations to ensure they can be applied correctly, and then broadcasts the transformed operation. Each operation is versioned sequentially.
  • Conflict-free Replicated Data Type (CRDT): This is a newer approach. The document's data structure is designed in such a way that concurrent edits can be merged mathematically without conflicts, always converging to the same state regardless of the order they are received.

• Trade-offs:

  • OT: Very complex to implement correctly, especially the transformation logic for all possible concurrent operations. The central server that manages transformations can be a single point of failure or bottleneck if not designed carefully. However, it is a proven, powerful model for collaborative text.
  • CRDT: Can be simpler on the server-side (often just a dumb relay), and naturally supports offline editing better than OT. However, CRDTs can have higher data overhead (each character might need metadata) and some operations can be less intuitive to model than in OT.

Issue #2: Stale Data from CDN Caching

Caching API responses for a real-time collaborative document for 5 minutes is fundamentally incorrect and will cause major consistency problems.

• Scenario:

  1. A document is actively being edited.
  2. A new user opens the document. CloudFront serves them a cached, 5-minute-old version of the document from the API.
  3. The user's client then connects to the WebSocket and receives a flood of real-time updates, causing the entire document to "jump" or re-render, creating a jarring UX and potential merge conflicts on the client side.

• Solution: Do not cache the API endpoints that serve document content. The CDN (CloudFront) should only be used for its primary purpose: caching static assets like JavaScript bundles, CSS files, images, and fonts. API calls for dynamic content like a document body must always go to the origin servers.

• Trade-offs:

  • Pro: The system is now correct. Users always fetch the latest version of a document upon opening it.
  • Con: Higher load on the origin servers for initial document fetches. This is a necessary trade-off for correctness. This load can be managed effectively with database read replicas, as mentioned in the scaling plan.

---

Category 2: Scaling Bottlenecks

Issue #3: Siloed WebSocket Connections & DB Polling

This is the most severe scaling bottleneck. The architecture does not support real-time communication between users connected to different servers. The 2-second polling is a slow, inefficient, and unscalable workaround.

• Scenario:

  1. 100 users are on a document. 50 are connected to Server A, 50 to Server B.
  2. A user on Server A makes an edit. It is instantly broadcast to the other 49 users on Server A.
  3. The 50 users on Server B see nothing.
  4. Up to 2 seconds later, Server B's polling mechanism reads the change from PostgreSQL and broadcasts it to its 50 users. This is not "real-time."
  5. As you scale to 100 servers, each server hammers the primary database every 2 seconds, creating a massive, unnecessary read load (100 servers * 30 queries/min = 3000 queries/min) that will crash the database.

• Solution: Use a dedicated Pub/Sub Message Bus. Decouple real-time messaging from the API servers. Use a service like Redis Pub/Sub, RabbitMQ, or Kafka.

  • New Data Flow:
    1. User types → change event sent via WebSocket to their connected server (e.g., Server A).
    2. Server A publishes this event to a specific channel on the message bus (e.g., doc-changes:document-123).
    3. All API servers (A, B, C...) are subscribed to these channels. They all receive the event from the bus almost instantly.
    4. Each server then broadcasts the change to all of its connected WebSocket clients.
    5. The database write can happen asynchronously in the background.

• Trade-offs:

  • Pro: Enables true horizontal scaling. Adding more API servers doesn't degrade performance; it improves it. Real-time communication latency is reduced from seconds to milliseconds. The database is removed from the real-time path, dramatically reducing its load.
  • Con: Introduces a new component (the message bus) that must be managed, scaled, and monitored. This adds operational complexity but is standard practice for distributed real-time systems.

Issue #4: Saving Full HTML Snapshots

Writing the entire document every 30 seconds is highly inefficient.

• Scenario: A 1MB document is being edited. A user changes one character. The system writes 1MB to the database. This creates massive write I/O, wastes storage space, and makes it impossible to have a granular "Version History" feature. It also means up to 29 seconds of work can be lost if a server crashes.

• Solution: Store Deltas/Operations, Not Snapshots. Align the storage strategy with the sync strategy (OT or CRDT).

  1. When a server processes an operation (e.g., insert 'a' at position 5, version 43), it writes that small operation to a database table (e.g., document_operations).
  2. To load a document, you fetch the last known full snapshot and replay all subsequent operations to reconstruct the current state.
  3. A background worker can periodically create new full snapshots and archive old operations to keep reconstruction times fast.

• Trade-offs:

  • Pro: Drastically reduced write load and storage costs. Enables a granular version history feature. Reduces data loss window from 30 seconds to near-zero.
  • Con: Reading/reconstructing a document is more computationally expensive than just reading a single blob. This is a good trade-off, as document loads are less frequent than edits, and the process can be heavily optimized.

---

Category 3: Failure Modes & Reliability Issues

Issue #5: Lack of Message Delivery Guarantees

The current WebSocket broadcast is "fire-and-forget." If a client has a momentary network blip, they will miss an update and their document state will permanently diverge from everyone else's until the next full save/refresh.

• Solution: Implement a Versioning and Catch-up Mechanism. This is a natural extension of using OT or CRDTs.

  1. Every operation/delta applied to a document increments a version number (e.g., from v42 to v43).
  2. The client keeps track of the document version it has.
  3. If a client disconnects and reconnects, it tells the server the version it has (I am on v42).
  4. The server can then query the document_operations table for all operations after v42 and send them to the client to "catch up."

• Trade-offs:

  • Pro: Creates a resilient system that can handle temporary network disruptions, which are common. This is essential for a reliable user experience.
  • Con: Adds complexity to the client and server to manage versioning and the catch-up logic.

Issue #6: JWT in localStorage

Storing the JWT in localStorage is a known security vulnerability.

• Problem: If your application has a Cross-Site Scripting (XSS) vulnerability (e.g., from a dependency or user-generated content that isn't properly sanitized), an attacker can inject a script that reads the JWT from localStorage. They can then use this token to impersonate the user completely.

• Solution: Store the JWT in a Secure, httpOnly Cookie. When the user logs in, the server sets the JWT in a cookie with the httpOnly and Secure flags.

  • httpOnly: Prevents JavaScript from accessing the cookie, mitigating XSS-based token theft.
  • Secure: Ensures the cookie is only sent over HTTPS. The browser will automatically include this cookie in all subsequent HTTP API requests and in the initial WebSocket connection handshake for authentication.

• Trade-offs:

  • Pro: Significantly improves security against XSS.
  • Con: You must now protect against Cross-Site Request Forgery (CSRF) attacks, since cookies are sent automatically by the browser. This is typically done by using the SameSite=Strict or SameSite=Lax cookie attribute or by implementing anti-CSRF tokens. This is a standard and well-understood security trade-off.