Here’s a back-of-the-envelope estimate for the total FLOPs to train GPT-3 (the 175B-parameter model) using reasonable public details, without looking anything up.
Assumptions (typical GPT-3 specs)
- Model: 175 billion parameters
- Transformer depth: 96 layers
- Hidden size (d_model): 12,288
- Sequence length used during training: 2,048 tokens
- Training data: about 300 billion tokens
- Training cost roughly doubles for forward + backward pass
- We’ll ignore data loading, optimizer overhead beyond the backward pass, precision differences, and any gradient checkpointing variations
Step 1: FLOPs per forward pass for one 2,048-token sequence
- Self-attention per layer costs on the order of L^2 * d_model operations, where L = 2,048 and d_model = 12,288.
- L^2 * d_model = 2,048^2 * 12,288 ≈ 51.54 × 10^9 operations per layer
- Across 96 layers: 51.54e9 × 96 ≈ 4.96 × 10^12 FLOPs for attention per forward pass
- Feed-forward network (FFN) per layer costs roughly 8 × d_model^2 operations
- d_model^2 = 12,288^2 ≈ 150.995 × 10^6
- 8 × d_model^2 ≈ 1.208 × 10^9 FLOPs per layer
- Across 96 layers: 1.208e9 × 96 ≈ 1.16 × 10^11 FLOPs for FFN per forward pass
- Total forward pass per sequence: attention + FFN ≈ 4.96e12 + 1.16e11 ≈ 5.07e12 FLOPs
- Include backward pass (roughly similar magnitude to forward): ≈ 2 × 5.07e12 ≈ 1.01 × 10^13 FLOPs per forward+backward pass
Step 2: How many 2,048-token sequences to cover 300B tokens?
- Number of segments N = 300e9 tokens / 2,048 ≈ 1.465 × 10^8 segments
Step 3: Total training FLOPs
- Total FLOPs ≈ (FLOPs per forward+backward) × N
- ≈ (1.01 × 10^13) × (1.465 × 10^8)
- ≈ 1.48 × 10^21 FLOPs
Result
- Rough estimate: about 1.5 × 10^21 FLOPs to train GPT-3 at the 175B scale, under the stated assumptions.
Notes and caveats
- This is a back-of-the-envelope calculation. Real compute could differ by a factor of a few due to:
- Different batch sizes or gradient accumulation, which changes the effective number of forward/backward passes
- Use of gradient checkpointing or optimizer options that alter the forward/backward FLOP balance
- Exact data token count and how sequences are batched
- Precision (fp32 vs. fp16/bfloat16) affecting the count of floating-point operations
- Overheads from data loading, padding, sparsity, or other architectural details
- The order-of-magnitude (around 10^21 FLOPs) is the key takeaway, not a precise figure.
If you want, I can run through a variant with a different token budget, sequence length, or depth to show how sensitive the total FLOPs is to those choices.
