Estimate Complexity

I'll estimate the FLOPs required to train GPT-3 step by step, using what I know about the model architecture and training process.

Key Parameters for GPT-3

From memory:

• Parameters (P): 175 billion (175 × 10⁹)
• Training tokens (D): ~300 billion (300 × 10⁹)
• Context length: 2048 tokens
• Layers: 96
• Hidden dimension: 12,288
• Attention heads: 96

The Standard Approximation Formula

There's a well-known approximation for transformer training FLOPs:

FLOPs ≈ 6 × P × D

Where:

• P = number of parameters
• D = number of training tokens
• The factor of 6 comes from:
  • 2× for forward pass (multiply-accumulate = 2 ops per parameter)
  • 4× for backward pass (roughly 2× the forward pass for computing gradients with respect to both activations and weights)

Calculation

FLOPs = 6 × P × D
FLOPs = 6 × (175 × 10⁹) × (300 × 10⁹)
FLOPs = 6 × 175 × 300 × 10¹⁸
FLOPs = 6 × 52,500 × 10¹⁸
FLOPs = 315,000 × 10¹⁸
FLOPs = 3.15 × 10²³

Result

Approximately 3 × 10²³ FLOPs (or ~300 zettaFLOPs)

Sanity Check

This aligns with what OpenAI reported. For context:

• Training on 1,000 V100 GPUs (~100 TFLOP/s each at FP16) = 10¹⁷ FLOP/s total
• Time needed: 3×10²³ / 10¹⁷ = 3×10⁶ seconds ≈ 35 days

This is plausible for a large-scale training run, which gives me confidence the estimate is in the right ballpark.