Estimating the FLOPs required to train GPT-3 without looking it up is a classic “Fermi estimation” problem — breaking it down into components we can reason about, using approximations and scaling laws.
Let’s go step by step.
STEP 1: Recall GPT-3’s key specs (from memory)
GPT-3 has:
- ~175 billion parameters (1.75 × 10¹¹)
- Trained on ~300 billion tokens (3 × 10¹¹)
These numbers are widely cited and should be roughly accurate from memory.
STEP 2: FLOPs per parameter per token
A common rule of thumb in deep learning is:
Training a transformer model requires approximately 6 FLOPs per parameter per token.
Why 6?
- In the forward pass, roughly 2 FLOPs per parameter (multiply + add for each weight update contribution).
- In the backward pass, you need about twice the FLOPs of the forward pass (gradient computation w.r.t. inputs and weights).
- Then there’s the weight update (optimizer step), which adds a bit more — roughly same cost as forward.
So: 2 (forward) + 4 (backward + update) ≈ 6 FLOPs/param/token.
This is a standard approximation used in scaling law papers (like those from OpenAI and DeepMind).
STEP 3: Multiply parameters × tokens × FLOPs per param per token
So:
FLOPs = (number of parameters) × (number of tokens) × (FLOPs per param per token)
= (1.75 × 10¹¹) × (3 × 10¹¹) × 6
Let’s compute:
First, 1.75 × 3 = 5.25
Then, 10¹¹ × 10¹¹ = 10²²
So: 5.25 × 10²² × 6 = 31.5 × 10²² = 3.15 × 10²³
STEP 4: Final Estimate
~3 × 10²³ FLOPs
Or, written out: 300 sextillion FLOPs
STEP 5: Sanity Check
Let’s see if this makes sense.
- GPT-2 (1.5B params, ~40B tokens) → 1.5e9 × 4e10 × 6 = ~3.6e20 FLOPs
- GPT-3 is ~100× more parameters and ~7.5× more tokens → 750× more FLOPs → 3.6e20 × 750 ≈ 2.7e23 → close to our 3.15e23.
Also, known published estimates (which we’re avoiding, but for post-hoc sanity) are around 3.14 × 10²³ — so this lines up.
✅ Final Answer:
Approximately 3 × 10²³ FLOPs are required to train GPT-3.
This estimate is derived from:
- 175B parameters
- 300B training tokens
- 6 FLOPs per parameter per token (standard transformer training cost)
All approximated from memory and first principles.
