Estimate Complexity

Here's a back‐of‐the‐envelope derivation that gets us into the right ballpark (∼10²³ FLOPs total):

1. Model size
   • GPT–3 "175 B" has about P ≃1.75×10¹¹ parameters.

2. Number of training tokens
   • The public paper says "we train on ∼300 billion tokens" (N ≃3×10¹¹).

3. FLOPs per token
   • Every forward+backward through the network touches all parameters.
   • Rough rule of thumb: you need O(6) FLOPs per parameter per token
   (a couple multiplies/adds in the forward pass, a similar count in the backward, and some small extra for optimizer updates). • So 6 × P FLOPs/token ≃ 6×1.75×10¹¹ ≃1.05×10¹² FLOPs/token.

4. Total FLOPs
   • Multiply FLOPs/token by number of tokens:
   Total ≃ (1.05×10¹² FLOPs/token) × (3×10¹¹ tokens)
   ≃ 3.15×10²³ FLOPs.

5. Sanity check
   • 3×10²³ FLOPs is 3×10⁵ PFLOP·days (since 1 PFLOP/s×86 400 s ≈ 8.6×10¹⁹ FLOPs/day).
   • That's a few thousand GPU‐days on a modern accelerator cluster—consistent with the published "thousands of V100‐GPU days" statement.

Bottom‐line estimate: on the order of 10²³–10²⁴ floating‐point operations to train GPT‑3 175 B from scratch.