Estimate Complexity

I'll estimate the FLOPs required to train GPT-3 by breaking down the key components and making reasonable approximations.

Key Parameters of GPT-3

• Parameters: ~175 billion (1.75 × 10¹¹)
• Training tokens: ~300 billion (3 × 10¹¹)
• Context window: 2048 tokens

Step 1: FLOPs per Token Forward Pass

For a transformer model, the forward pass FLOPs are dominated by matrix multiplications in the attention and MLP layers.

Attention layers:

• Self-attention: ~4 × d_model × d_model × sequence_length per layer
• With d_model ≈ 12,288 (based on GPT-3 architecture) and 96 layers
• Attention FLOPs per token ≈ 96 × 4 × 12,288² ≈ 5.8 × 10¹⁰

MLP layers:

• Each MLP has ~8 × d_model² FLOPs per token (two linear transformations)
• MLP FLOPs per token ≈ 96 × 8 × 12,288² ≈ 1.16 × 10¹¹

Total forward pass FLOPs per token: ~1.7 × 10¹¹

Step 2: FLOPs per Token Backward Pass

Backward pass typically requires ~2× the forward pass FLOPs due to gradient computations.

• Backward pass FLOPs per token: ~3.4 × 10¹¹

Step 3: Total FLOPs per Token

Forward + Backward = ~5.1 × 10¹¹ FLOPs per token

Step 4: Training FLOPs

However, we need to account for multiple epochs and the fact that training typically processes sequences, not individual tokens.

Sequence-based calculation:

• Average sequence length: ~1024 tokens (conservative estimate)
• Sequences processed: 3 × 10¹¹ tokens ÷ 1024 ≈ 2.9 × 10⁸ sequences
• FLOPs per sequence: ~5.1 × 10¹¹ × 1024 ≈ 5.2 × 10¹⁴
• Total training FLOPs: 2.9 × 10⁸ × 5.2 × 10¹⁴ ≈ 1.5 × 10²³

Step 5: Additional Considerations

• Parameter updates and optimizer computations (Adam): ~2× the backward pass FLOPs
• Multiple epochs (effective training): roughly 2-3× the base calculation

Final Estimate

Total training FLOPs: ~3 × 10²³ to 5 × 10²³ FLOPs

This is equivalent to approximately 300 to 500 exaFLOPs.

This estimate aligns reasonably with published figures and captures the massive computational requirements of training such a large language model, representing one of the most computationally intensive AI training runs to date.