GPT-5.2 Pro
Below is a back-of-the-envelope FLOP estimate for training GPT‑3 (the 175B dense Transformer), using only basic Transformer compute facts and a few widely known high-level quantities.
1) Start with the two key knobs: parameters and training tokens
Parameters (P)
GPT‑3 “175B” means:
- (P \approx 175 \times 10^9) parameters.
Training tokens (T)
A commonly cited order-of-magnitude for GPT‑3 training is a few hundred billion tokens. I’ll assume:
- (T \approx 300 \times 10^9) tokens processed during training.
(If you assumed 200B or 500B instead, the final answer scales linearly.)
2) FLOPs per token for a dense Transformer: why it’s (\approx 6P)
A Transformer layer is mostly matrix multiplies (linear layers in attention projections + MLP). For a dense model, the key scaling is:
- For a linear layer with weight matrix (W) containing (|W|) parameters, applying it to one token costs about (|W|) multiply-accumulates (MACs).
- If you count a MAC as 2 FLOPs (one multiply + one add), forward pass cost is about: [ \text{forward FLOPs per token} \approx 2P ]
Training requires backprop. For each linear layer, you effectively do ~3 matrix multiplies of comparable size:
- forward: (Y = XW)
- backward wrt weights: (\nabla W = X^T \nabla Y)
- backward wrt activations: (\nabla X = \nabla Y W^T)
So training compute is roughly 3× the forward compute: [ \text{train FLOPs per token} \approx 3 \times (2P) = 6P ]
This ignores smaller terms (layernorm, softmax, activations) and also ignores attention’s extra (L^2) work (addressed briefly below).
3) Total training FLOPs
Total FLOPs: [ \text{FLOPs} \approx (6P),T ]
Plug in numbers:
- (P = 175 \times 10^9)
- (T = 300 \times 10^9)
[ \text{FLOPs} \approx 6 \cdot (175 \times 10^9)\cdot(300 \times 10^9) ]
First multiply the plain numbers:
- (175 \cdot 300 = 52{,}500)
- (6 \cdot 52{,}500 = 315{,}000)
Track powers of ten:
- (10^9 \cdot 10^9 = 10^{18})
So: [ \text{FLOPs} \approx 315{,}000 \times 10^{18} = 3.15 \times 10^{23} ]
Estimated total:
[
\boxed{\approx 3 \times 10^{23}\ \text{FLOPs}}
]
4) Quick note: what about attention’s (L^2) cost?
Self-attention has extra compute from forming attention scores and applying them to values, scaling like (O(L^2 d)). For GPT‑3-sized models (large (d)) with a context length around a couple thousand, the dominant cost is still the dense matmuls tied to parameters ((O(L d^2))), and the (L^2) attention part is typically only a few percent of total FLOPs. So (\sim 6PT) is usually a decent first estimate.
If you add a modest overhead factor (say +5–15%), you’d still be in the same ballpark: [ \boxed{\text{rough range } \sim (3\text{ to }3.6)\times 10^{23}\ \text{FLOPs}} ]
5) Final estimate
GPT‑3 (175B) training compute:
[
\boxed{\approx 3.15 \times 10^{23}\ \text{FLOPs} ;;(\text{order of }10^{23})}
]
If you tell me what token count you want to assume (e.g., 200B vs 500B, or multiple epochs over a smaller dataset), I can rescale the estimate instantly.
