Estimate Complexity
Estimate the total number of FLOPs required to train GPT-3 without searching online. Explain your reasoning step by step.
Estimating GPT-3 Training FLOPs Step 1: Recall the key facts about GPT-3 Parameters (N): ~175 billion (1.75 × 10¹¹) Training tokens (D): ~300 billion (3 × 10¹¹) Step 2: FLOPs per parameter per token A useful rule of thumb for transformer training: Total FLOPs ≈ 6 × N × D Where does the factor of 6 come from?Read the full answer
Assume “GPT-3” refers to the flagship 175-billion-parameter model. Use the standard transformer training-cost approximation For a dense transformer, the dominant cost is matrix multiplication involving the model weights: Forward pass: approximately (2) FLOPs per parameter per token one multiplication and one addition…Read the full answer