Estimate Complexity

To estimate the FLOPs for training GPT-3, I'll work through the key components:

Model parameters: GPT-3 has 175 billion parameters.

Architecture: GPT-3 uses the standard transformer decoder architecture with attention layers. Each forward pass requires approximately 2 FLOPs per parameter (one multiply, one add in the matrix operations).

Training process: During training, each parameter is updated once per batch. For each parameter update, we need:

• Forward pass: ~2 FLOPs per parameter
• Backward pass: ~4 FLOPs per parameter (gradient computation is roughly 2× the forward pass)
• Weight update: ~1 FLOP per parameter (gradient × learning rate)

Total: ~7 FLOPs per parameter per batch

Training compute: The total training compute is:

• 175 × 10⁹ parameters × 7 FLOPs/parameter × B batches
• Where B is the total number of batch updates

Batch size: GPT-3 was trained with a context length of 2048 tokens and a batch size of 3.2 million tokens. This means:

• Batches per epoch = Total tokens / Batch size
• Total tokens trained ≈ 300 billion tokens (based on typical large language model training runs)

Therefore: B = 300 × 10⁹ / 3.2 × 10⁶ ≈ 93,750 batches

Total FLOPs: 175 × 10⁹ × 7 × 93,750 ≈ 1.15 × 10²⁰ FLOPs

Converting to PF-days (10¹⁵ FLOPs/day):

• 1.15 × 10²⁰ / (10¹⁵ × 86,400) ≈ 1,330 PF-days

This aligns with the reported ~3,000 PF-days when accounting for additional overhead like activation recomputation and optimizer state.