To estimate the total FLOPs (floating-point operations) required to train GPT-3, we can use a standard approximation formula derived from the architecture of Transformer models.
Here is the step-by-step breakdown of the estimation:
Step 1: Identify the Key Variables
To calculate training compute, we need two primary numbers: the number of parameters in the model ($P$) and the number of tokens the model was trained on ($D$).
- Model Size ($P$): GPT-3 is famously known to have 175 billion parameters.
$$P = 175 \times 10^9$$
- Dataset Size ($D$): According to the original GPT-3 paper ("Language Models are Few-Shot Learners"), the model was trained on a weighted mix of datasets (Common Crawl, WebText2, Books1, Books2, Wikipedia). The total number of tokens processed during training was approximately 300 billion tokens.
$$D = 300 \times 10^9$$
Step 2: Apply the Compute Approximation Formula
For dense Transformer models, a widely accepted heuristic (often cited in papers by Kaplan et al. and DeepMind regarding scaling laws) is that training requires approximately 6 FLOPs per parameter per training token.
- Forward Pass: Requires $\approx 2P$ FLOPs per token (matrix multiplication involves 1 multiply and 1 add, and we do this for the weights).
- Backward Pass (Gradient Calculation): Requires $\approx 4P$ FLOPs per token (roughly twice the cost of the forward pass).
- Total: $2P + 4P = 6P$ FLOPs per token.
Therefore, the formula for Total Training Compute ($C$) is:
$$C \approx 6 \times P \times D$$
Step 3: Perform the Calculation
Now, substitute the values into the formula:
$$C \approx 6 \times (175 \times 10^9) \times (300 \times 10^9)$$
First, group the integers and the powers of 10:
$$C \approx (6 \times 175 \times 300) \times (10^9 \times 10^9)$$
Calculate the integers:
- $6 \times 175 = 1,050$
- $1,050 \times 300 = 315,000$
Combine with the exponent ($10^{18}$):
$$C \approx 315,000 \times 10^{18} \text{ FLOPs}$$
Step 4: Convert to Scientific Notation
To make the number readable, we normalize it:
$$315,000 = 3.15 \times 10^5$$
Combine exponents:
$$3.15 \times 10^5 \times 10^{18} = 3.15 \times 10^{23}$$
Final Estimate
The estimated total compute required to train GPT-3 is approximately:
$$3.14 \times 10^{23} \text{ FLOPs}$$
(Note: In the AI literature, this is often converted to "Petaflop-days." One Petaflop-day is approximately $8.64 \times 10^{19}$ FLOPs. Dividing $3.14 \times 10^{23}$ by that number yields roughly 3,640 Petaflop-days, which aligns closely with OpenAI's published figures.)
