Estimate Complexity

Below is a back-of-the-envelope FLOP estimate for training GPT‑3 (the 175B dense Transformer), using only basic Transformer compute facts and a few widely known high-level quantities.

---

1) Start with the two key knobs: parameters and training tokens

Parameters (P)

GPT‑3 “175B” means:

• (P \approx 175 \times 10^9) parameters.

Training tokens (T)

A commonly cited order-of-magnitude for GPT‑3 training is a few hundred billion tokens. I’ll assume:

• (T \approx 300 \times 10^9) tokens processed during training.

(If you assumed 200B or 500B instead, the final answer scales linearly.)

---

2) FLOPs per token for a dense Transformer: why it’s (\approx 6P)

A Transformer layer is mostly matrix multiplies (linear layers in attention projections + MLP). For a dense model, the key scaling is:

• For a linear layer with weight matrix (W) containing (|W|) parameters, applying it to one token costs about (|W|) multiply-accumulates (MACs).
• If you count a MAC as 2 FLOPs (one multiply + one add), forward pass cost is about: [ \text{forward FLOPs per token} \approx 2P ]

Training requires backprop. For each linear layer, you effectively do ~3 matrix multiplies of comparable size:

1. forward: (Y = XW)
2. backward wrt weights: (\nabla W = X^T \nabla Y)
3. backward wrt activations: (\nabla X = \nabla Y W^T)

So training compute is roughly 3× the forward compute: [ \text{train FLOPs per token} \approx 3 \times (2P) = 6P ]

This ignores smaller terms (layernorm, softmax, activations) and also ignores attention’s extra (L^2) work (addressed briefly below).

---

3) Total training FLOPs

Total FLOPs: [ \text{FLOPs} \approx (6P),T ]

Plug in numbers:

• (P = 175 \times 10^9)
• (T = 300 \times 10^9)

[ \text{FLOPs} \approx 6 \cdot (175 \times 10^9)\cdot(300 \times 10^9) ]

First multiply the plain numbers:

• (175 \cdot 300 = 52{,}500)
• (6 \cdot 52{,}500 = 315{,}000)

Track powers of ten:

• (10^9 \cdot 10^9 = 10^{18})

So: [ \text{FLOPs} \approx 315{,}000 \times 10^{18} = 3.15 \times 10^{23} ]

Estimated total:
[ \boxed{\approx 3 \times 10^{23}\ \text{FLOPs}} ]

---

4) Quick note: what about attention’s (L^2) cost?

Self-attention has extra compute from forming attention scores and applying them to values, scaling like (O(L^2 d)). For GPT‑3-sized models (large (d)) with a context length around a couple thousand, the dominant cost is still the dense matmuls tied to parameters ((O(L d^2))), and the (L^2) attention part is typically only a few percent of total FLOPs. So (\sim 6PT) is usually a decent first estimate.

If you add a modest overhead factor (say +5–15%), you’d still be in the same ballpark: [ \boxed{\text{rough range } \sim (3\text{ to }3.6)\times 10^{23}\ \text{FLOPs}} ]

---

5) Final estimate

GPT‑3 (175B) training compute:
[ \boxed{\approx 3.15 \times 10^{23}\ \text{FLOPs} ;;(\text{order of }10^{23})} ]

If you tell me what token count you want to assume (e.g., 200B vs 500B, or multiple epochs over a smaller dataset), I can rescale the estimate instantly.