Inception: Mercury 2
Answer (order‑of‑magnitude estimate)
The training of the original GPT‑3 (the 175 billion‑parameter “GPT‑3‑175B” model) requires on the order of 10²³ floating‑point operations (FLOPs) – roughly 1 – 2 × 10²³ FLOPs.
Below is a step‑by‑step walk‑through of how that number is arrived at, using only the publicly‑known architecture details and a few well‑established approximations for transformer training cost.
1. What we need to know
| Quantity | Approximate value (publicly known) | Why it matters |
|---|---|---|
| Number of model parameters | (N_{\text{par}} \approx 175;\text{B} = 1.75\times10^{11}) | Each parameter is touched during forward and backward passes. |
| Training token count | (N_{\text{tok}} \approx 300;\text{B} = 3.0\times10^{11}) | GPT‑3 was trained on ~300 billion tokens (Common Crawl + WebText + Books + Wikipedia). |
| Model architecture | 96 transformer layers, hidden size (d = 12,288), feed‑forward size (4d) | Determines how many FLOPs per token per layer. |
| Optimizer | Adam (2‑momentum) | Adds a modest constant factor (≈2) to the raw forward‑backward count. |
| Training regime | 1 epoch over the data (the 300 B tokens) | No extra passes; the token count already incorporates the number of epochs. |
2. A simple “parameter‑times‑token” model
A widely‑used back‑of‑the‑envelope rule for transformer training is:
[ \text{FLOPs}{\text{train}} ;\approx; 2 \times N{\text{par}} \times N_{\text{tok}} ]
- Why the factor 2?
- Forward pass: each weight is multiplied by an activation and added → 2 FLOPs (multiply + add).
- Backward pass: the gradient w.r.t. each weight requires another multiply‑add → another 2 FLOPs.
- The total is therefore ≈ 4 FLOPs per weight per token, but the multiply‑add pair is usually counted as a single “FLOP” in deep‑learning literature, giving the factor 2.
Plugging in the numbers:
[ \begin{aligned} \text{FLOPs}_{\text{train}} &\approx 2 \times (1.75\times10^{11}) \times (3.0\times10^{11})\ &= 2 \times 5.25\times10^{22}\ &= 1.05\times10^{23};\text{FLOPs}. \end{aligned} ]
So ≈ 1 × 10²³ FLOPs is the baseline estimate.
3. Refinement: accounting for the internal structure of a transformer layer
If we want to see where the “2 × Npar × Ntok” figure comes from, we can break down a single transformer layer:
| Operation (per token) | FLOPs (approx.) | Explanation |
|---|---|---|
| Q, K, V projections (3 × (d\times d)) | (3d^{2}) | Each projection is a dense matrix multiply. |
| Attention scores (dot‑product over sequence length (L)) | (L d) | For a token we compute dot‑products with all (L) keys. |
| Attention output projection (1 × (d\times d)) | (d^{2}) | |
| Feed‑forward first linear ( (d \times 4d) ) | (4d^{2}) | |
| Activation (GELU) | ≈ (4d) (tiny) | |
| Feed‑forward second linear ( (4d \times d) ) | (4d^{2}) | |
| Total per layer (forward) | ≈ (12d^{2} + Ld) |
With (d = 12,288) we have (d^{2} \approx 1.5\times10^{8}).
Ignoring the sequence‑length term (it is ≈ (L d) ≈ (1024 \times 12,288 \approx 1.3\times10^{7}), an order of magnitude smaller than (d^{2})), the forward cost per layer per token is roughly:
[ 12 d^{2} \approx 12 \times 1.5\times10^{8} \approx 1.8\times10^{9};\text{FLOPs}. ]
Multiplying by the 96 layers:
[ \text{Forward per token} \approx 96 \times 1.8\times10^{9} \approx 1.7\times10^{11};\text{FLOPs}. ]
A backward pass roughly doubles this (gradient w.r.t. each weight), giving ≈ (3.4\times10^{11}) FLOPs per token for the whole model.
Now multiply by the token count:
[ 3.4\times10^{11} \times 3.0\times10^{11} \approx 1.0\times10^{23};\text{FLOPs}, ]
which coincides with the simpler 2 × Npar × Ntok estimate.
Thus the two viewpoints are consistent: the “parameter‑times‑token” rule already captures the dominant cost of the dense matrix multiplications inside each transformer layer.
4. Adding the optimizer overhead
Adam keeps two extra state tensors (first‑ and second‑moment estimates) per parameter, and each training step performs a few extra arithmetic operations per weight (e.g., bias‑corrected moments, division, sqrt). A common heuristic is to multiply the raw forward‑backward count by ≈ 2 to include Adam’s extra work.
[ \text{FLOPs}_{\text{with Adam}} \approx 2 \times 1.05\times10^{23} \approx 2.1\times10^{23};\text{FLOPs}. ]
So a realistic upper bound is ≈ 2 × 10²³ FLOPs.
5. Summary of the estimate
| Step | Value | Comment |
|---|---|---|
| Parameters | (1.75\times10^{11}) | 175 B |
| Training tokens | (3.0\times10^{11}) | 300 B |
| Base forward‑backward FLOPs | (1.0\times10^{23}) | 2 × Npar × Ntok |
| Adam optimizer factor | ≈ 2 | Gives ~(2\times10^{23}) |
| Final ball‑park | (1!-!2 \times 10^{23}) FLOPs | Equivalent to ~100 exaflop‑seconds, i.e. ~3 years of a 100 PFLOP/s machine. |
What does this number mean in practice?
- Energy: Assuming a modern GPU delivers ~10 GFLOP/s per watt, 2 × 10²³ FLOPs would consume on the order of 20 GWh of electricity (≈ 2 months of a typical data‑center’s power budget).
- Time: A 100 PFLOP/s training cluster (≈ 10 × A100 GPUs) would need ≈ 2 × 10⁶ seconds ≈ 23 days of continuous compute to finish the job, which matches the publicly reported training wall‑clock time (a few weeks).
6. Caveats & sources of uncertainty
| Source of uncertainty | Effect on estimate |
|---|---|
| Exact token count (some reports say 300 B, others 500 B) | Linear scaling – 500 B tokens would push the estimate to ~1.7 × 10²³ FLOPs. |
| Model‑specific tricks (e.g., mixed‑precision, gradient checkpointing) | Mixed‑precision reduces FLOPs per multiply‑add by ≈ 0.5×, but checkpointing adds extra recomputation; net effect is modest (≈ ± 20 %). |
| Additional regularization (e.g., dropout, layer‑norm) | Adds a few percent overhead. |
| Hardware‑specific optimizations (e.g., fused kernels) | May reduce the effective FLOP count, but the theoretical count stays the same. |
Given these uncertainties, the order‑of‑magnitude figure of 10²³ FLOPs is robust.
Bottom line
- Training GPT‑3 (175 B) ≈ 1 × 10²³ FLOPs (forward + backward).
- Including Adam optimizer ≈ 2 × 10²³ FLOPs.
This is the scale of compute that modern “large‑scale” language‑model training operates at.
