I'll estimate the FLOPs required to train GPT-3 by breaking down the key components and making reasonable approximations.
Key Parameters of GPT-3
- Parameters: ~175 billion (1.75 × 10¹¹)
- Training tokens: ~300 billion (3 × 10¹¹)
- Context window: 2048 tokens
Step 1: FLOPs per Token Forward Pass
For a transformer model, the forward pass FLOPs are dominated by matrix multiplications in the attention and MLP layers.
Attention layers:
- Self-attention: ~4 × d_model × d_model × sequence_length per layer
- With d_model ≈ 12,288 (based on GPT-3 architecture) and 96 layers
- Attention FLOPs per token ≈ 96 × 4 × 12,288² ≈ 5.8 × 10¹⁰
MLP layers:
- Each MLP has ~8 × d_model² FLOPs per token (two linear transformations)
- MLP FLOPs per token ≈ 96 × 8 × 12,288² ≈ 1.16 × 10¹¹
Total forward pass FLOPs per token: ~1.7 × 10¹¹
Step 2: FLOPs per Token Backward Pass
Backward pass typically requires ~2× the forward pass FLOPs due to gradient computations.
- Backward pass FLOPs per token: ~3.4 × 10¹¹
Step 3: Total FLOPs per Token
Forward + Backward = ~5.1 × 10¹¹ FLOPs per token
Step 4: Training FLOPs
However, we need to account for multiple epochs and the fact that training typically processes sequences, not individual tokens.
Sequence-based calculation:
- Average sequence length: ~1024 tokens (conservative estimate)
- Sequences processed: 3 × 10¹¹ tokens ÷ 1024 ≈ 2.9 × 10⁸ sequences
- FLOPs per sequence: ~5.1 × 10¹¹ × 1024 ≈ 5.2 × 10¹⁴
- Total training FLOPs: 2.9 × 10⁸ × 5.2 × 10¹⁴ ≈ 1.5 × 10²³
Step 5: Additional Considerations
- Parameter updates and optimizer computations (Adam): ~2× the backward pass FLOPs
- Multiple epochs (effective training): roughly 2-3× the base calculation
Final Estimate
Total training FLOPs: ~3 × 10²³ to 5 × 10²³ FLOPs
This is equivalent to approximately 300 to 500 exaFLOPs.
This estimate aligns reasonably with published figures and captures the massive computational requirements of training such a large language model, representing one of the most computationally intensive AI training runs to date.
vs