Estimating FLOPs to Train GPT-3
Step 1: Recall GPT-3's Architecture
GPT-3 has approximately:
- 175 billion parameters (this is well-known)
- 96 transformer layers
- 96 attention heads
- Hidden dimension: 12,288
- Context length: 2,048 tokens
Step 2: FLOPs per Token (Forward Pass)
For a transformer, the dominant operations are matrix multiplications. There's a useful rule of thumb:
FLOPs per token ≈ 2 × N (for the forward pass)
where N = number of parameters.
Why 2×N?
- Each parameter is involved in roughly one multiply-add operation per token
- A multiply-add = 2 FLOPs
- This holds when sequence length is much smaller than model dimension (which is roughly true here)
So forward pass FLOPs per token:
$$2 \times 175 \times 10^9 = 3.5 \times 10^{11} \text{ FLOPs/token}$$
Step 3: Account for Backward Pass
Backpropagation costs approximately 2× the forward pass (computing gradients for weights + activations).
So total per token:
$$3 \times 3.5 \times 10^{11} = 1.05 \times 10^{12} \text{ FLOPs/token}$$
Step 4: Training Tokens
GPT-3 was trained on approximately 300 billion tokens (from the paper's description of ~300B tokens from filtered Common Crawl, WebText, books, Wikipedia).
Step 5: Total FLOPs
$$\text{Total FLOPs} = 1.05 \times 10^{12} \times 3 \times 10^{11}$$
$$= 3.15 \times 10^{23} \text{ FLOPs}$$
$$\approx \boxed{3 \times 10^{23} \text{ FLOPs}}$$
Sanity Check
- This is ~300 zettaFLOPs
- At 1,000 A100s running at ~300 TFLOPs each (with ~50% utilization → ~150 TFLOPs effective):
- Total compute = 1,000 × 1.5×10¹⁴ = 1.5×10¹⁷ FLOPs/second
- Time = 3×10²³ / 1.5×10¹⁷ ≈ 2×10⁶ seconds ≈ 23 days
- OpenAI reportedly used thousands of GPUs for weeks — this is plausible ✓
Summary
| Component | Value |
|---|
| Parameters | 175B |
| FLOPs/token (forward) | ~3.5 × 10¹¹ |
| Training multiplier (fwd+bwd) | ×3 |
| Training tokens | ~300B |
| Total FLOPs | ~3 × 10²³ |
The key formula to remember: C ≈ 6ND where N = parameters, D = training tokens.
