Critical Issues Analysis

🔴 CRITICAL: Data Loss & Consistency Issues

1. Client Clock-Based Timestamps (SEVERE)

Problem:

• Client clocks can be wrong, manipulated, or skewed
• User with clock set to future will always "win" conflicts
• No guaranteed ordering of operations

Failure Scenarios:

User A (clock: 2024-01-01 10:00:00) types "Hello"
User B (clock: 2024-01-01 09:59:59) types "World"
Result: "Hello" wins even though "World" was actually typed later

Solutions:

Approach	Pros	Cons
Server-assigned Lamport clocks	Logical ordering, no clock sync needed	Requires coordination
Hybrid logical clocks (HLC)	Combines wall-clock + logical counter	More complex implementation
Operational Transform (OT)	Industry standard, handles intent	Complex to implement correctly
CRDT (Conflict-free Replicated Data Types)	Mathematically guaranteed convergence	Memory overhead, learning curve

Recommended: Implement CRDT (Yjs or Automerge library)

• Trade-off: 2-3x memory overhead vs guaranteed consistency
• Eliminates need for conflict resolution entirely

---

2. WebSocket Partitioning (CRITICAL)

Problem:

• Users on different servers don't receive real-time updates
• 2-second polling creates massive delays and inconsistency
• Race conditions between WebSocket broadcasts and polling

Failure Scenario:

Time 0s: User A (Server 1) types "A"
Time 0.5s: User B (Server 2) types "B" 
Time 2s: Server 2 polls, gets "A", broadcasts to User B
Time 2.5s: User C (Server 1) sees "AB", User D (Server 2) sees "BA"

Solutions:

Approach	Latency	Complexity	Cost
Redis Pub/Sub	<50ms	Low	$
RabbitMQ/Kafka	<100ms	Medium	$$
Dedicated WebSocket service (Socket.io with Redis adapter)	<30ms	Low	$

Recommended: Redis Pub/Sub with sticky sessions

// On any server receiving a change
redis.publish('document:${docId}', JSON.stringify(change));

// All servers subscribe
redis.subscribe('document:*', (channel, message) => {
  const docId = channel.split(':')[1];
  broadcastToLocalClients(docId, JSON.parse(message));
});

• Trade-off: Redis becomes single point of failure (mitigate with Redis Sentinel/Cluster)

---

3. Full HTML Snapshots (DATA LOSS RISK)

Problem:

• 30-second window for data loss if server crashes
• No audit trail or version history
• Cannot undo/replay operations
• HTML storage is inefficient and risky (XSS, formatting loss)

Solutions:

Approach	Storage	Recovery	History
Event sourcing	10x more	Complete	Full
Operational log + snapshots	3x more	Good	Configurable
Differential snapshots	2x more	Good	Limited

Recommended: Event Sourcing with Periodic Snapshots

-- Operations table
CREATE TABLE operations (
  id BIGSERIAL PRIMARY KEY,
  document_id UUID,
  user_id UUID,
  operation JSONB,  -- CRDT operation
  server_timestamp TIMESTAMPTZ DEFAULT NOW(),
  lamport_clock BIGINT
);

-- Snapshots table (every 100 operations)
CREATE TABLE snapshots (
  document_id UUID,
  version BIGINT,
  content JSONB,
  created_at TIMESTAMPTZ
);

• Trade-off: 3-5x storage increase vs complete audit trail and zero data loss

---

🟠 SEVERE: Race Conditions

4. Concurrent Paragraph Edits

Problem:

• "Last write wins" on paragraph level causes character-level data loss
• No operational transformation

Example:

Initial: "The cat"
User A: "The black cat" (inserts "black ")
User B: "The fat cat" (inserts "fat ")
Last-write-wins result: "The fat cat" (User A's work lost)
Correct result: "The black fat cat" or "The fat black cat"

Solution: Use character-level CRDT (Yjs RichText type)

import * as Y from 'yjs';

const ydoc = new Y.Doc();
const ytext = ydoc.getText('content');

// Automatically handles concurrent inserts
ytext.insert(4, 'black ');  // User A
ytext.insert(4, 'fat ');    // User B
// Result preserves both edits with deterministic ordering

---

5. Database Write Conflicts

Problem:

• Multiple servers writing to same document simultaneously
• PostgreSQL row-level locking causes deadlocks
• No transaction coordination

Solution: Optimistic locking with version numbers

CREATE TABLE documents (
  id UUID PRIMARY KEY,
  version BIGINT NOT NULL,
  content JSONB,
  updated_at TIMESTAMPTZ
);

-- Update with version check
UPDATE documents 
SET content = $1, version = version + 1, updated_at = NOW()
WHERE id = $2 AND version = $3
RETURNING version;

-- If no rows updated, version conflict occurred

---

🟡 MAJOR: Scaling Bottlenecks

6. PostgreSQL Write Bottleneck

Problem:

• Every keystroke = database write
• PostgreSQL limited to ~10k writes/sec on single node
• Read replicas don't help write-heavy workload

Calculation:

100 concurrent users × 60 keystrokes/min = 100 writes/sec ✓
1,000 concurrent users = 1,000 writes/sec ✓
10,000 concurrent users = 10,000 writes/sec (at limit) ⚠️
100,000 concurrent users = 100,000 writes/sec ✗

Solutions:

Approach	Throughput	Consistency	Complexity
Write-through cache (Redis)	100k+ ops/sec	Eventual	Low
Batch operations	50k+ ops/sec	Strong	Medium
Sharded PostgreSQL (Citus)	500k+ ops/sec	Strong	High

Recommended: Redis Write-Through Cache + Async Persistence

// Write to Redis immediately (fast)
await redis.zadd(`ops:${docId}`, timestamp, JSON.stringify(op));

// Async worker drains to PostgreSQL in batches
setInterval(async () => {
  const ops = await redis.zrange(`ops:${docId}`, 0, 99);
  await pg.query('INSERT INTO operations VALUES ...', ops);
  await redis.zrem(`ops:${docId}`, ...ops);
}, 1000);

• Trade-off: 1-second window of data in Redis only (mitigate with Redis persistence)

---

7. 2-Second Polling (MASSIVE WASTE)

Problem:

• N servers × M documents × 0.5 queries/sec = database overload
• 2-second latency unacceptable for real-time

Calculation:

10 servers × 1,000 active docs × 0.5 qps = 5,000 queries/sec
Just for polling! Actual useful work is extra.

Solution: Already covered in #2 (Redis Pub/Sub)

---

8. CDN Caching API Responses (DANGEROUS)

Problem:

• 5-minute cache on collaborative document API = stale data
• Users see outdated content
• Cache invalidation nightmare

Solution: Never cache document content

Cache-Control: no-store, must-revalidate  // Document endpoints
Cache-Control: public, max-age=31536000   // Static assets only

---

🔵 IMPORTANT: Security & Reliability

9. JWT in localStorage (XSS VULNERABILITY)

Problem:

• XSS attack steals token → full account compromise for 24 hours
• No way to revoke token

Solution: HttpOnly cookies + short-lived tokens + refresh tokens

// Access token: 15 minutes, HttpOnly cookie
res.cookie('accessToken', jwt.sign({...}, secret, {expiresIn: '15m'}), {
  httpOnly: true,
  secure: true,
  sameSite: 'strict'
});

// Refresh token: 7 days, stored in Redis for revocation
const refreshToken = crypto.randomBytes(32).toString('hex');
await redis.setex(`refresh:${userId}`, 7*24*60*60, refreshToken);

---

10. No WebSocket Authentication After Initial Connect

Problem:

• JWT expires after 24 hours but WebSocket stays open
• No re-authentication mechanism

Solution: Periodic token refresh over WebSocket

// Client
setInterval(() => {
  ws.send({type: 'REFRESH_TOKEN', token: getNewToken()});
}, 14 * 60 * 1000); // Every 14 minutes

// Server validates and updates connection auth

---

11. Single Point of Failure: PostgreSQL

Problem:

• Database down = entire system down
• No automatic failover mentioned

Solution: PostgreSQL with Patroni + HAProxy

┌─────────┐
│ HAProxy │ (virtual IP)
└────┬────┘
     ├──► [Primary] PostgreSQL + Patroni
     ├──► [Standby] PostgreSQL + Patroni  
     └──► [Standby] PostgreSQL + Patroni

• Automatic failover in <30 seconds
• Trade-off: 3x infrastructure cost

---

12. No Rate Limiting

Problem:

• Malicious user can spam operations
• DoS attack via WebSocket flooding

Solution: Token bucket rate limiter

const rateLimiter = new Map();

function checkRateLimit(userId) {
  const limit = rateLimiter.get(userId) || {tokens: 100, lastRefill: Date.now()};
  
  // Refill tokens (10 per second)
  const now = Date.now();
  const tokensToAdd = Math.floor((now - limit.lastRefill) / 100);
  limit.tokens = Math.min(100, limit.tokens + tokensToAdd);
  limit.lastRefill = now;
  
  if (limit.tokens < 1) return false;
  limit.tokens--;
  rateLimiter.set(userId, limit);
  return true;
}

---

📊 Revised Architecture

┌─────────────┐
│   Clients   │
└──────┬──────┘
       │ WSS (CRDT operations)
       ▼
┌──────────────────────────────┐
│  Load Balancer (Sticky)      │
└──────┬───────────────────────┘
       │
   ┌───┴────┬────────┬────────┐
   ▼        ▼        ▼        ▼
┌─────┐  ┌─────┐  ┌─────┐  ┌─────┐
│Node1│  │Node2│  │Node3│  │Node4│
└──┬──┘  └──┬──┘  └──┬──┘  └──┬──┘
   │        │        │        │
   └────────┴────────┴────────┘
            │
     ┌──────┴──────┐
     ▼             ▼
┌─────────┐   ┌──────────────┐
│  Redis  │   │  PostgreSQL  │
│ Pub/Sub │   │   + Patroni  │
│ + Cache │   │  (HA Cluster)│
└─────────┘   └──────────────┘

Key Changes:

1. CRDT-based operations (Yjs)
2. Redis Pub/Sub for real-time sync
3. Event sourcing with snapshots
4. Sticky sessions on load balancer
5. Redis write-through cache
6. PostgreSQL HA with Patroni
7. HttpOnly cookies for auth
8. No CDN caching of API responses

Performance Targets:

• Latency: <50ms for operations
• Throughput: 100k+ concurrent users
• Data loss: Zero (event sourcing)
• Availability: 99.95% (HA database)