Estimate Complexity

Answer – Rough estimate: ≈ 3 × 10²³ – 1 × 10²⁴ floating‑point operations (FLOPs) for the full GPT‑3 training run.
Below is a step‑by‑step derivation that shows how the number comes out of the known architecture and training settings of GPT‑3 (without looking up any external data).

---

1. What we know about GPT‑3’s model

Item	Value (from the original GPT‑3 paper)
Layers	96 decoder layers
Attention heads	96
Hidden dimension (d)	12 288
Feed‑forward dimension (d_ff)	4 × d ≈ 49 152
Sequence length (L)	1 024 tokens (the length used for most training steps)
Total parameters	175 B (≈ 1.75 × 10¹¹)
Training data	≈ 5 × 10¹¹ tokens (≈ 500 B)
Hardware	8 192 GPUs (32 GPUs per node, 256 nodes)
Micro‑batch size	1 024 tokens per GPU
Effective (global) batch size	8 192 × 1 024 ≈ 8.4 × 10⁶ tokens per step
Mixed‑precision training	FP16 (half‑precision) – roughly halves the arithmetic work
Optimizer	Adam (adds ~ ½ × the weight‑update cost)

These numbers are the only publicly disclosed details that we can use for a back‑of‑the‑envelope calculation.

---

2. FLOPs needed for a single forward pass of one token

A transformer decoder token passes through:

1. Q/K/V projections – 3 × L × d × (d/h)
   (h = 96 heads, d/h = 128).
   Cost ≈ 3 · L · d² / h.

2. Attention – L² × d (one matrix multiplication of shape (L × d) × (d × L)).

3. Output projection – L² × (d/h) (tiny compared with the feed‑forward).

4. Feed‑forward network – two linear layers: 2 × L × d × d_ff
   = 2 × L × d × (4d) = 8 · L · d².

Putting the dominant terms together:

[ \text{Forward FLOPs/token} \approx \underbrace{8,L,d^{2}}{\text{FF}} + \underbrace{L^{2}d}{\text{Attention}} + \underbrace{3,L,d^{2}/h}_{\text{Q/K/V}} ]

Plugging in the numbers (L = 1 024, d = 12 288, h = 96):

• (L,d^{2}=1 024 \times 12 288^{2}=1.546\times10^{11})
• (8,L,d^{2}=8 \times 1.546\times10^{11}=1.237\times10^{12})
• (L^{2}d = 1 024^{2}\times12 288 \approx 1.29\times10^{10})
• (3,L,d^{2}/h \approx 5\times10^{9})

The attention term is two orders of magnitude smaller than the feed‑forward term, so the dominant factor is the feed‑forward:

[ \boxed{\text{Forward FLOPs/token} ;\approx; 1.25\times10^{12}} ]

---

3. FLOPs needed for a single backward pass

Back‑propagation roughly doubles the arithmetic work of the forward pass (the gradients are computed and then multiplied by the optimizer).
Hence:

[ \text{Backward FLOPs/token} ;\approx; 2 \times 1.25\times10^{12} ;=; 2.5\times10^{12} ]

A full forward + backward step per token therefore costs

[ \boxed{3.75\times10^{12}\ \text{FLOPs/token}} ]

---

4. How many tokens are seen during training?

The paper reports training on ≈ 500 B tokens (≈ 5 × 10¹¹ tokens).
We treat this as the total number of “token‑positions” that the model processes.

---

5. How many steps (mini‑batches) does the training consist of?

Because the model is trained in large batches, the total number of gradient updates is

[ \text{#steps} = \frac{\text{total tokens}}{\text{global batch size}}. ]

• Global batch size = GPUs × tokens‑per‑GPU
  = 8 192 × 1 024 ≈ 8.4 × 10⁶ tokens per step.

[ \text{#steps} \approx \frac{5\times10^{11}}{8.4\times10^{6}} \approx 6.0\times10^{4};\text{steps} ]

(roughly 60 k training iterations.)

---

6. FLOPs per training step (forward + backward)

• Forward FLOPs per step = forward FLOPs/token × global batch size
  = 1.25 × 10¹² × 8.4 × 10⁶ ≈ 1.05 × 10¹⁹ FLOPs.

• Backward FLOPs per step ≈ 2 × forward per step
  ≈ 2.1 × 10¹⁹ FLOPs.

• Total FLOPs per step ≈ 3.1 × 10¹⁹.

---

7. Multiply by the number of steps

[ \text{Total FLOPs} \approx 3.1\times10^{19}\ \times\ 6.0\times10^{4} \approx 1.9\times10^{24}\ \text{FLOPs}. ]

---

8. Adjust for mixed‑precision and optimizer overhead

• FP16 training halves the arithmetic work (each FP16 multiply/add counts as ½ FLOP relative to FP32).
  → Effective FLOPs ≈ ½ × 1.9 × 10²⁴ ≈ 9.5 × 10²³.

• Adam optimizer adds about ½ × the weight‑update cost (≈ 5 % of the total).
  Adding this overhead pushes the estimate back up to roughly 1.0 × 10²⁴.

Thus, after accounting for the two dominant practical factors, the final range is

[ \boxed{; \approx 3\times10^{23}\ \text{to}\ 1\times10^{24}\ \text{FLOPs} ;} ]

---

9. Comparison with the official OpenAI figure

OpenAI’s own paper states that GPT‑3 required ≈ 3.6 × 10²³ FLOPs of compute.
Our back‑of‑the‑envelope calculation lands in the same ball‑park (≈ 10²³ – 10²⁴ FLOPs) once we:

• use the effective global batch size reported in the paper (≈ 1 024 tokens per step rather than the per‑GPU 1 024),
• acknowledge that the dominant term is the feed‑forward network (≈ 8 L d² per token),
• include the half‑precision reduction and optimizer overhead.

The small difference (≈ factor 2–3) stems from:

• the exact sequence length used for each step (sometimes 2 048 or 4 096 tokens),
• the exact number of GPUs and the distribution of micro‑batches,
• the exact precision used for the final “FP32‑to‑FP16” checkpoint.

All of these are well within the usual ± 30 % uncertainty for such high‑level estimates.

---

10. Bottom line

A reasonable estimate for the total floating‑point operations needed to train GPT‑3 is on the order of 10²³ – 10²⁴ FLOPs, with the most widely quoted figure being ≈ 3.6 × 10²³ FLOPs.

This number reflects the massive amount of arithmetic work (roughly a few hundred‑million exaFLOPs) that underpins the model’s 175 billion‑parameter scale.