Estimate Complexity
Estimate the total number of FLOPs required to train GPT-3 without searching online. Explain your reasoning step by step.
A common back-of-the-envelope estimate for transformer training compute is: [ \text{Training FLOPs} \approx 6 \times N \times D ] where: (N) = number of model parameters (D) = number of training tokens For GPT-3 (175B), reasonable known assumptions are: Parameters: (N \approx 175 \text{ billion} = 1.75 \times 10^{11})…Read the full answer
Below is a step‑by‑step, “hand‑calculation” of how many floating‑point operations (FLOPs) GPT‑3 had to do to learn from its training data. I’ll show the logic, the numbers I use, and how they combine to give the widely‑quoted “≈ 300 exaFLOPs” figure. (All the math can be done in a spreadsheet or a simple calculator.)Read the full answer