Step-by-Step Reasoning for Estimating GPT-3 Training FLOPs
1. Recall GPT-3 Key Specs (from Memory)
- GPT-3 (175B parameter version): 175 billion parameters (1.75 × 10¹¹).
- Trained on 300 billion tokens (3 × 10¹¹ tokens). This is the standard figure from OpenAI's reports.
- Architecture: Standard Transformer decoder-only model.
- Batch size, sequence length, etc., aren't precisely recalled, but typical for such models: sequence length ~2K-4K tokens, large batch sizes (e.g., effective 1M+ tokens via gradient accumulation).
2. FLOPs Formula for Transformer Training
The standard empirical formula for total FLOPs in training autoregressive Transformers (from papers like Kaplan et al., Hoffmann et al., and used for GPT-3 estimates) is:
[
\text{Total FLOPs} \approx 6 \times N \times D
]
- N = number of parameters (1.75 × 10¹¹).
- D = number of tokens processed (3 × 10¹¹).
- Why 6?
- Forward pass: ~2 FLOPs per parameter per token (matrix multiplies dominate).
- Backward pass: ~4 FLOPs per parameter per token (gradients roughly double the forward cost).
- Total ~6× per token. This is a well-established rule-of-thumb validated across models like GPT-2/3, PaLM, etc. (ignores minor overheads like activations, which are ~20% but absorbed in the approximation).
This excludes pretraining optimizations (e.g., activations checkpointing saves ~30% memory but not FLOPs).
3. Plug in the Numbers
[
\text{Total FLOPs} = 6 \times (1.75 \times 10^{11}) \times (3 \times 10^{11})
]
- First, (1.75 \times 3 = 5.25).
- Then, (5.25 \times 10^{11} \times 10^{11} = 5.25 \times 10^{22}).
- Multiply by 6: (6 \times 5.25 = 31.5), so (3.15 \times 10^{23}) FLOPs.
4. Refinements and Validation
- Exact public figure: OpenAI and analyses (e.g., Epoch AI, Srivastava et al.) confirm ~3.14 × 10²³ FLOPs (or 314 sextillion). My calculation hits it spot-on.
- Sensitivity checks:
| Variable | Base Value | If 20% Lower | If 20% Higher | Impact on Total FLOPs |
|---|
| Parameters (N) | 175B | 140B | 210B | 2.51×10²³ / 3.78×10²³ |
| Tokens (D) | 300B | 240B | 360B | 2.51×10²³ / 3.78×10²³ |
| Multiplier | 6 | 5 (forward-heavy) | 7 (extra overhead) | 2.63×10²³ / 3.67×10²³ |
- Even with ±20% uncertainty, range is ~2.5–3.8 × 10²³. Base case is robust.
- Other factors: No major adjustments needed for GPT-3 (no MoE sparsity, standard dense training). Compute was on ~10K V100/A100 GPUs for ~1 month, consistent with this FLOP count (each A100 does ~300 TFLOPS effective).
5. Final Estimate
~3.1 × 10²³ FLOPs (or 310 sextillion floating-point operations).
This matches the canonical figure without any external lookup—purely from memorized specs and standard methodology.
