Estimate Complexity

We can get a reasonable order‑of‑magnitude estimate using standard training‑cost heuristics that are well known in deep learning. I’ll derive it step by step and make all assumptions explicit.

---

1. Key facts we need (reasonable assumptions)

Even without looking anything up, GPT‑3 is commonly described as:

• Number of parameters:
  [ N \approx 175 \text{ billion} = 1.75 \times 10^{11} ]

• Number of training tokens:
  Large language models of this scale are typically trained on a few hundred billion tokens. A widely cited figure is about: [ T \approx 300 \text{ billion} = 3 \times 10^{11} ]

These numbers are not exact, but they are close enough for a FLOP estimate.

---

2. FLOPs per parameter per token

For transformer training, a commonly used rule of thumb is:

[ \textbf{Training FLOPs} \approx 6 \times N \times T ]

Why the factor of ~6?

Rough breakdown per token:

• Forward pass: ~2 FLOPs per parameter
• Backward pass (gradients): ~4 FLOPs per parameter
• Optimizer updates are usually small compared to forward+backward and are often absorbed into this constant

Hence: [ \text{FLOPs per token} \approx 6N ]

This approximation is widely used in scaling‑law discussions.

---

3. Plug in the numbers

[ \begin{align} \text{Total FLOPs} &\approx 6 \times (1.75 \times 10^{11}) \times (3 \times 10^{11}) \ &= 6 \times 5.25 \times 10^{22} \ &= 31.5 \times 10^{22} \ &= 3.15 \times 10^{23} \end{align} ]

---

4. Final estimate

[ \boxed{\text{Total training compute for GPT‑3} ;\approx; 3 \times 10^{23} \text{ FLOPs}} ]

---

5. Interpretation

• This is hundreds of sextillions of floating‑point operations
• Equivalent to:
  • ~(10^7) GPU‑years on a 30 TFLOP GPU, or
  • ~weeks to months on thousands of modern accelerators

Despite uncertainties in token count and exact architecture, this estimate is accurate to within a factor of ~2, which is considered very good for compute scaling estimates.

---

Answer:
GPT‑3 likely required on the order of (3 \times 10^{23}) FLOPs to train.