To estimate the total floating-point operations (FLOPs) required to train GPT-3, we can use a standard heuristic derived from the architecture of Transformer models.
Here is the step-by-step reasoning.
1. Identify Key Constants
First, we need the two primary variables that determine training compute:
- Number of Parameters ($N$): GPT-3 (the full version) is known to have 175 billion parameters.
- Number of Training Tokens ($D$): GPT-3 was trained on a massive dataset. Based on the original research paper, the 175B model was trained on approximately 300 billion tokens.
2. The Heuristic Formula ($6ND$)
In deep learning, specifically for Transformers, there is a well-established "rule of thumb" for estimating training compute:
$$\text{Compute (FLOPs)} \approx 6 \times N \times D$$
Why 6?
The factor of 6 comes from the breakdown of the forward and backward passes:
- Forward Pass ($2N$): For every parameter in the model, there is roughly one multiply-accumulate operation (which counts as 2 FLOPs) per token.
- Backward Pass ($4N$): The backward pass (calculating gradients) is computationally more expensive. It typically requires roughly twice the compute of the forward pass ($2 \times 2N = 4N$).
- Total: $2N (\text{forward}) + 4N (\text{backward}) = 6N$ FLOPs per token.
3. The Calculation
Now, we plug in the values for GPT-3:
- $N = 1.75 \times 10^{11}$ (175 billion)
- $D = 3.0 \times 10^{11}$ (300 billion)
$$\text{Total FLOPs} = 6 \times (1.75 \times 10^{11}) \times (3.0 \times 10^{11})$$
Step-by-step multiplication:
- Multiply the coefficients: $6 \times 1.75 = 10.5$
- Multiply by the tokens: $10.5 \times 3.0 = 31.5$
- Combine the powers of ten: $10^{11} \times 10^{11} = 10^{22}$
$$\text{Total FLOPs} = 31.5 \times 10^{22}$$
$$\text{Total FLOPs} = 3.15 \times 10^{23}$$
4. Final Estimate
The total estimated compute for training GPT-3 is $3.15 \times 10^{23}$ FLOPs.
5. Contextualizing the Number
To put $3.15 \times 10^{23}$ into perspective:
- In scientific notation, this is 315 Sextillion FLOPs.
- In terms of "Zettaflops" (10²¹), it is 315 ZettaFLOPs.
- If you used a single high-end NVIDIA A100 GPU (which performs at roughly 312 TFLOPS of FP16 with Tensor Cores), and assuming perfect efficiency, it would take roughly 32 years to complete the training. (In reality, OpenAI used thousands of GPUs in parallel to finish in a few months).
Note on accuracy: This estimate ignores the overhead of the "attention" mechanism (which scales with sequence length), but for models as large as GPT-3, the feed-forward and projection layers (represented by $N$) dominate the compute cost, making $6ND$ a very accurate approximation.
