Below is a back-of-the-envelope estimate for the total training compute (in FLOPs) for GPT‑3, done from first principles and common Transformer training heuristics.
1) What we’re estimating
We want total FLOPs over the whole training run:
[
\text{Total FLOPs} \approx (\text{FLOPs per token}) \times (\text{# training tokens})
]
So we need:
- # parameters (P) (GPT‑3 “175B” ⇒ (P \approx 1.75\times 10^{11}))
- # training tokens (T) (we’ll estimate)
- A way to relate FLOPs/token to (P)
2) FLOPs per token ≈ a small multiple of parameter count
A commonly used rule for dense Transformers is:
[
\text{Training FLOPs} \approx 6 , P , T
]
Why “6”?
- A forward pass through a big dense network costs on the order of ~2 FLOPs per parameter (roughly one multiply-add per weight, counted as ~2 FLOPs).
- Backward pass (gradients wrt activations + wrt weights) costs about ~2× forward for dense layers.
- So forward + backward ≈ ~3× forward.
- That yields about:
[
\text{FLOPs/token} \approx 3 \times (2P) = 6P
]
This ignores some extras (attention score computations, softmax, layernorm, embedding lookups), but for very large models the big matrix multiplies dominate, and the (6PT) scaling is usually the right order of magnitude.
So we’ll use:
[
\text{Total FLOPs} \approx 6PT
]
3) Estimate the number of training tokens (T)
We need an assumption. GPT‑3 is widely described as trained on hundreds of billions of tokens. Without looking it up, a reasonable guess range is:
- Low: (T \approx 200\text{B} = 2\times 10^{11})
- Mid: (T \approx 300\text{B} = 3\times 10^{11})
- High: (T \approx 500\text{B} = 5\times 10^{11})
I’ll compute the mid estimate and show the range.
4) Compute total FLOPs
Given:
- (P = 1.75\times 10^{11})
- (T = 3\times 10^{11})
[
\text{Total FLOPs} \approx 6 \times (1.75\times 10^{11}) \times (3\times 10^{11})
]
Multiply constants:
- (6 \times 1.75 \times 3 = 31.5)
Multiply powers of ten:
- (10^{11}\times 10^{11} = 10^{22})
So:
[
\text{Total FLOPs} \approx 31.5 \times 10^{22} = 3.15\times 10^{23}\ \text{FLOPs}
]
Range using different token assumptions
- If (T=2\times 10^{11}):
[
6 \cdot 1.75 \cdot 2 \times 10^{22} = 21 \times 10^{22} = 2.1\times 10^{23}
]
- If (T=5\times 10^{11}):
[
6 \cdot 1.75 \cdot 5 \times 10^{22} = 52.5 \times 10^{22} = 5.25\times 10^{23}
]
5) Final estimate
Estimated total training compute for GPT‑3 (175B):
[
\boxed{\text{On the order of }10^{23}\text{ FLOPs, roughly }3\times 10^{23}\text{ FLOPs (with a plausible range }2\text{–}5\times 10^{23}\text{).}}
]
If you tell me what you want to assume for the training tokens (e.g., 300B vs 500B) or whether you want to include optimizer overhead (Adam adds extra memory ops; FLOP impact is smaller but nonzero), I can refine the estimate accordingly.
