GPT-5.4 Nano
Below is a back-of-the-envelope FLOPs estimate for training something “GPT‑3–like” (the 175B parameter model described in the original GPT‑3 paper). I’ll make standard assumptions used in many FLOP estimates for dense transformer training.
1) What we mean by “training FLOPs”
For transformer pretraining with dense matrix multiplications, a common approximation is:
[ \text{FLOPs} \approx (\text{# parameters}) \times (\text{# tokens processed}) \times (\text{a factor for forward+backward}) ]
A widely used rule of thumb is that forward pass costs ~1 unit, and backward pass roughly 2× forward, so forward+backward ≈ 3× forward. Some estimates use ~6× depending on conventions (e.g., counting multiply-adds separately). I’ll be explicit.
2) Use the “Chinchilla”-style scaling FLOPs form
A commonly cited approximation for dense transformer training is:
[ \boxed{\text{FLOPs} \approx 6 , N , D} ]
Where:
- (N) = number of parameters
- (D) = total number of tokens seen during training
The factor 6 comes from accounting for:
- forward pass + backward pass,
- and multiply-add counting conventions.
This formula is popular because it matches typical FLOP counts for transformer training.
3) Plug in GPT‑3 numbers (model size and tokens)
For GPT‑3 (175B):
- (N \approx 1.75\times 10^{11}) parameters
GPT‑3 was trained on on the order of:
- (D \approx 3\times 10^{11}) tokens (hundreds of billions)
We can now compute:
[ \text{FLOPs} \approx 6 \times (1.75\times 10^{11}) \times (3\times 10^{11}) ]
First multiply parameters and tokens:
[ (1.75\times 10^{11}) (3\times 10^{11}) = 5.25\times 10^{22} ]
Now multiply by 6:
[ \text{FLOPs} \approx 6 \times 5.25\times 10^{22} = 3.15\times 10^{23} ]
So the estimate is:
[ \boxed{\text{FLOPs} \approx 3\times 10^{23}} ]
4) Sanity check / interpretation
- (3\times 10^{23}) FLOPs is the right order of magnitude reported in many discussions for training a 175B-class dense GPT model.
- If you change the token count or the constant factor (e.g., using 3 instead of 6, or a different tokens estimate), you can easily move by a factor of ~2–3.
5) Final answer (order-of-magnitude)
[ \boxed{\text{Estimated total training compute for GPT‑3 (175B): } \sim 3\times 10^{23}\ \text{FLOPs}} ]
Note
If you want, I can also produce an alternative derivation using transformer layer FLOPs (attention + MLP) in terms of (L) (sequence length), (d_{\text{model}}), number of layers, and heads—this gives a more “structural” estimate but requires more architectural details and still ends up in the same ballpark.
