Estimate Complexity

Below is one way to arrive at a back-of-the-envelope estimate for the total floating-point operations (FLOPs) needed to train GPT-3 (the 175B-parameter model). Note that different sources use slightly different assumptions, so you may see figures that differ by factors of 2–3; however, the overall scale (10^23 FLOPs) remains consistent.

---

1. Key Quantities

• Number of parameters (Nₚ): 175 billion (1.75 × 10^11)
• Number of training tokens (Nₜ): ~300 billion (3.0 × 10^11)
• Rule-of-thumb for FLOPs per parameter per token: A common rough estimate is 6 FLOPs per parameter per token for a full forward-and-backward pass with Adam-style updates.

Why 6?

• Forward pass often taken as ~2 × Nₚ FLOPs.
• Backward pass can be roughly twice the cost of the forward pass (~4 × Nₚ).
• Together → 2 + 4 = 6 × Nₚ FLOPs per token.

(Exact counts differ depending on the details of the architecture, hardware efficiency, and whether you count a multiply-add as one or two FLOPs, but 6 is a convenient ballpark multiplier.)

---

2. Core Calculation

Using the 6 FLOPs rule of thumb:

[ ext{Total FLOPs} ;approx; 6 ; imes; N_p ; imes; N_t. ]

Plug in the numbers:

1. (N_p = 1.75 imes 10^{11}) (175B parameters).
2. (N_t = 3.0 imes 10^{11}) (300B tokens).
3. Multiplier = 6.

So,

[ ext{Total FLOPs} ;=; 6 ; imes; (1.75 imes 10^{11}) ; imes; (3.0 imes 10^{11}). ]

First, multiply the “nice” numbers:

• (6 imes 1.75 = 10.5)
• (10.5 imes 3.0 = 31.5)

So the coefficient is 31.5. Now include the powers of ten:

• (31.5 imes 10^{11} imes 10^{11} = 31.5 imes 10^{22} = 3.15 imes 10^{23}.)

Hence,

[ ext{Total FLOPs} ;approx; 3.15 imes 10^{23}. ]

---

3. Result and Interpretation

Therefore, a rough estimate for the total training cost of GPT-3 (175B) is on the order of
[ mathbf{3 imes 10^{23}} ext{ FLOPs.} ]

Depending on exact architectural details, optimizer choices, hardware efficiency, and definitions of FLOPs vs. MACs (multiply-accumulate operations), public estimates generally range from around (2 imes 10^{23}) to (4 imes 10^{23}). But the important takeaway is that it is a few times (10^{23}) FLOPs.

---

4. Summary

1. Identify parameters and tokens: 175B parameters, 300B tokens.
2. Adopt the “6 FLOPs per parameter per token” rule for forward + backward + update.
3. Multiply: (6 imes 1.75 imes 10^{11} imes 3.0 imes 10^{11} approx 3.15 imes 10^{23}).

This simple calculation gives a ballpark figure without delving into the fine details of the GPT-3 architecture or specifics of training hardware.