Here is a step-by-step estimate for the total training FLOPs of GPT-3.
Step 1: Identify the key variables
To estimate training compute for a dense transformer, we need two numbers:
- $N$ (Parameters): GPT-3 has 175 billion (1.75 × 10¹¹) parameters.
- $D$ (Tokens): GPT-3 was trained on roughly 300 billion (3 × 10¹¹) tokens. (This is the widely cited figure from the original paper for the final training corpus).
Step 2: Use the standard training FLOPs formula
For decoder-only transformers trained with backpropagation, the total floating-point operations (FLOPs) can be approximated as:
$$\text{FLOPs} \approx 6 \times N \times D$$
Why 6?
- Forward pass: Processing one token requires roughly 2 FLOPs per parameter (1 multiply + 1 accumulate for the matrix operations).
- Backward pass: Computing gradients requires roughly 4 FLOPs per parameter (approximately 2× the forward cost to calculate gradients with respect to both inputs and weights).
- Total: $2 + 4 = 6$ FLOPs per parameter per token.
(Note: This formula dominates the cost; the quadratic attention cost is negligible compared to the linear parameter cost for a model of this size).
Step 3: Calculate
$$\text{FLOPs} \approx 6 \times (1.75 \times 10^{11}) \times (3 \times 10^{11})$$
$$= 6 \times 1.75 \times 3 \times 10^{22}$$
$$= 6 \times 5.25 \times 10^{22}$$
$$= 31.5 \times 10^{22}$$
$$= \mathbf{3.15 \times 10^{23} \text{ FLOPs}}$$
Step 4: Account for uncertainty and overhead
The raw calculation gives ~3.15 × 10²³ FLOPs. However, we should adjust for real-world factors:
- Dataset size variance: If GPT-3 was trained on closer to 400–500B tokens (including potential epochs or different data mixes), the number could be 4–5 × 10²³ FLOPs.
- Implementation overhead: Data loading, checkpointing, communication in distributed training, and the Adam optimizer updates add 10–20% overhead.
Final Estimate
Approximately $3 \times 10^{23}$ to $5 \times 10^{23}$ FLOPs (roughly 300 to 500 sextillion FLOPs).
For context, this is equivalent to about 3,000 to 5,000 petaflop-seconds (or PFLOP-seconds), which aligns with the estimated thousands of petaflop-days often cited for GPT-3's training run on V100 GPUs.
