Estimate Complexity

Below is a step‑by‑step reasoning that leads to a ball‑park estimate of the total floating‑point operations (FLOPs) required to train the largest GPT‑3 model (the 175 B‑parameter variant).
The estimate does not rely on looking up a single published number; it reconstructs the order‑of‑magnitude using publicly‑known architecture details and the standard FLOP‑counting formula for transformer training.

---

1️⃣ Model size – number of parameters ( N )

Model	Parameters
GPT‑3 175 B	≈ 1.75 × 10¹¹ (≈ 175 billion)

From the original GPT‑3 paper (Brown et al., 2020) the largest checkpoint is reported as 175 B parameters.

---

2️⃣ Training data – tokens processed ( D )

The GPT‑3 paper reports two relevant numbers:

Variant	Tokens seen
175 B model	≈ 3 × 10¹¹ tokens (≈ 300 B)
Smaller models	up to 5 × 10¹¹ tokens (≈ 500 B)

For the 175 B model the training run stopped after ≈ 300 billion tokens (the authors note that 300 B tokens give a good compute‑optimal trade‑off for this size).

---

3️⃣ FLOPs per token per parameter (forward + backward)

A transformer layer performs, for each token, a multiply‑add (2 FLOPs) per weight in the forward pass, and a gradient multiply‑add (4 FLOPs) per weight in the backward pass.

[ \text{FLOPs}_{\text{token,param}} = 2 ;(\text{forward}) + 4 ;(\text{backward}) = 6 ]

Thus the total FLOPs for a full training run is

[ \boxed{\text{FLOPs} \approx 6 \times N \times D} ]

---

4️⃣ Plug in the numbers

[ \begin{aligned} N &\approx 1.75 \times 10^{11} \ D &\approx 3.0 \times 10^{11} \ \text{FLOPs} &\approx 6 \times (1.75 \times 10^{11}) \times (3.0 \times 10^{11}) \ &= 6 \times 5.25 \times 10^{22} \ &\approx 3.15 \times 10^{23};\text{FLOPs} \end{aligned} ]

Base estimate (forward + backward only): ≈ 3 × 10²³ FLOPs.

---

5️⃣ Account for optimizer and other overhead

Most large‑scale training uses Adam (or a variant). Adam updates each parameter twice per step (first‑moment and second‑moment estimates), adding roughly 2 × N × D additional FLOPs.

[ \text{FLOPs}_{\text{total}} \approx (6 + 2) \times N \times D = 8 \times N \times D ]

[ 8 \times (1.75 \times 10^{11}) \times (3.0 \times 10^{11}) \approx 4.2 \times 10^{23};\text{FLOPs} ]

Including Adam: ≈ 4 × 10²³ – 5 × 10²³ FLOPs.

---

6️⃣ Published “official” figure (for sanity check)

The GPT‑3 paper itself states that the 175 B run required ≈ 3.6 × 10²³ FLOPs.
Our simple 6 × N × D calculation (3.15 × 10²³) is within a factor of ~1.15 of the reported number, confirming that the ≈ 3 × 10²³ – 4 × 10²³ range is realistic.

---

7️⃣ Summary of the estimate

Quantity	Value
Parameters (N)	1.75 × 10¹¹
Tokens processed (D)	3 × 10¹¹
FLOPs (forward + backward)	≈ 3.15 × 10²³
FLOPs (incl. Adam optimizer)	≈ 4.2 × 10²³
Typical reported compute	≈ 3.6 × 10²³ FLOPs

Final ball‑park estimate: ~3 × 10²³ – 4 × 10²³ floating‑point operations for the full training of GPT‑3 175 B.

---

How this number translates to hardware

If a single modern GPU (e.g., NVIDIA A100) can sustain ~312 TFLOPS (FP16) for training workloads:

[ \frac{3.6 \times 10^{23};\text{FLOPs}}{3.12 \times 10^{14};\text{FLOPs/s}} \approx 1.15 \times 10^{9};\text{s} \approx 36.5;\text{years} ]

A cluster of ~1 000 such GPUs (as reported for GPT‑3) reduces wall‑time to a few months, which matches the reported ≈ 3 640 GPU‑days on V100‑class hardware.

---

Key take‑aways

1. FLOPs scale linearly with model size and tokens processed (≈ 6 × N × D for forward + backward).
2. Optimizer overhead adds roughly 30 % (factor ≈ 8 × N × D).
3. Real‑world training runs also incur communication, checkpointing, and mixed‑precision inefficiencies, pushing the effective compute toward the ~3.5 × 10²³ – 4 × 10²³ FLOPs range reported by OpenAI.

Thus, training GPT‑3 required on the order of 10²³ – 10²⁴ FLOPs, i.e. a few hundred exa‑FLOP‑days of compute.