Estimate Complexity

Of course. Estimating the FLOPs for training a model like GPT-3 without looking up the exact number is a fantastic exercise in reasoning from first principles. Here is a step-by-step breakdown of how to arrive at a very close estimate.

Step 1: Understand the Core Components of the Calculation

The total computational cost for training a neural network is primarily determined by three factors:

1. N (Number of Parameters): The total number of trainable weights and biases in the model. For GPT-3, this is a well-known figure.
2. C (Number of Training Tokens): The total amount of data the model processes, measured in tokens (sub-word pieces). This is the most difficult variable to estimate.
3. K (FLOPs per Parameter per Token): A constant that represents the number of floating-point operations required to process a single token for every parameter in the model for a full training step (forward pass, backward pass, and optimizer update).

The total FLOPs can be approximated by the formula:

Total FLOPs ≈ K * N * C

Let's estimate each of these components.

---

Step 2: Estimate the Number of Parameters (N)

This is the easiest part. The size of GPT-3's largest model is one of its most famous specifications.

• N = 175 billion parameters
• N = 175 x 10⁹

---

Step 3: Estimate the FLOPs per Parameter per Token (K)

Training involves a forward pass, a backward pass (to calculate gradients), and a parameter update. A common and well-accepted approximation in deep learning is:

• Forward Pass: For each parameter, you perform a multiplication and an addition. This is roughly 2 FLOPs.
• Backward Pass: Calculating the gradients is computationally more intensive. It's generally estimated to be about 2x the cost of the forward pass. So, this is roughly 4 FLOPs.
• Parameter Update: The optimizer (like Adam) updates the weights. This also involves a multiply and an add for each parameter, which is another 2 FLOPs.

Adding these up: 2 (forward) + 4 (backward) + 2 (update) = 8 FLOPs.

However, a more standard and slightly more conservative estimate used in many research papers (like the Chinchilla scaling laws paper) is K = 6. This typically accounts for the forward pass (2 FLOPs) and the two main parts of the backward pass (2 FLOPs + 2 FLOPs), often bundling the optimizer's smaller cost into this figure. Using K=6 is a robust choice for an estimation of this scale.

• K = 6

---

Step 4: Estimate the Number of Training Tokens (C)

This is the most challenging and crucial estimation step. We need to reason about the composition of the training dataset.

GPT-3 was trained on a mixture of sources: Common Crawl, WebText2, Books1, Books2, and Wikipedia. Let's try to build a plausible number.

1. Wikipedia: The entire English Wikipedia has about 4 billion tokens. This is a small part of the total dataset.
2. Books: The dataset is said to contain a large number of books. Let's imagine it's the equivalent of 1 million books. An average book might be 100,000 words. There are about 1.3 tokens per word.
   • Tokens per book ≈ 100,000 words * 1.3 tokens/word = 130,000 tokens
   • Total book tokens ≈ 1,000,000 books * 130,000 tokens/book = 130 billion tokens.
3. Web-based Data (Common Crawl & WebText2): This is the largest component. Common Crawl is massive (petabytes of raw data), but it's heavily filtered for quality. The goal is to get a high-quality subset.
   • Let's think about the scale. The number of tokens in the web-based data must be significantly larger than the book corpus. A number in the hundreds of billions seems reasonable.
   • Could it be 200 billion? 300 billion? 500 billion?
   • A helpful sanity check comes from the Chinchilla scaling laws, which suggest a compute-optimal training data size of about 20 tokens per parameter. For GPT-3, this would be 20 * 175 billion = 3.5 trillion tokens. However, GPT-3 predates Chinchilla* and is known to have been "under-trained"—meaning it was trained on significantly fewer tokens than this optimal ratio would suggest.
   • This tells us the number should be well below 3.5 trillion. A number in the low hundreds of billions is much more likely.

Let's combine our estimates:

• Wikipedia: ~4 billion
• Books: ~130 billion
• Web Data: Let's estimate this to be in the same ballpark as the books, but larger. Let's say ~170 billion.

Total Estimated Tokens (C) = 4 + 130 + 170 = 304 billion tokens

This is a very strong, defensible estimate. Let's round it for easier calculation.

• C = 300 x 10⁹ tokens

---

Step 5: Perform the Final Calculation

Now we plug our estimates into the formula:

Total FLOPs ≈ K * N * C Total FLOPs ≈ 6 * (175 x 10⁹) * (300 x 10⁹)

Let's do the math:

1. 6 * 175 = 1050
2. 1050 * 300 = 315,000
3. 10⁹ * 10⁹ = 10¹⁸

Total FLOPs ≈ 315,000 x 10¹⁸

To write this in standard scientific notation:

Total FLOPs ≈ 3.15 x 10²³

Final Estimate and Conclusion

Estimated Total FLOPs to train GPT-3: 3.15 x 10²³

To put this enormous number in perspective:

• This is 315 ZettaFLOPs.
• If you had a petaflop-scale (10¹⁵ FLOP/s) AI supercomputer running at 100% utilization, it would take 3.15 x 10²³ / 10¹⁵ = 3.15 x 10⁸ seconds, which is about 10 years. In reality, they used thousands of GPUs for several weeks.

This estimation, built from first principles and reasonable assumptions about the model and its dataset, arrives at a number that is remarkably close to the official figure cited in the original GPT-3 paper, which is ~3.14 x 10²³ FLOP-days. The key was correctly identifying the parameters, using the standard 6*N*C formula, and making a logical, grounded estimate of the training data size.