Estimate Complexity
Estimate the total number of FLOPs required to train GPT-3 without searching online. Explain your reasoning step by step.
Below is a “back-of-the-envelope” that stays within a factor of ~2 of the number most practitioners quote (≈3×10²³ FLOP). We will do it twice—once with the simplest possible formula (parameter count × 6 × tokens) and once with a slightly more careful treatment of the actual tensor shapes inside the transformer.Read the full answer
Estimating the FLOPs to Train GPT-3 Key Known Facts Property Value Parameters (N) 175 billion (1.75 × 10¹¹) Training tokens (D) ~300 billion (3 × 10¹¹) Architecture Decoder-only Transformer Step 1: FLOPs per Token in a Transformer The core computation in a transformer is dominated by linear (matrix multiplication)…Read the full answer