The total number of FLOPs required to train GPT‑3 is approximately 3.15 × 10²³.

Step‑by‑step reasoning:

1. Model size: GPT‑3 has 175 billion (1.75 × 10¹¹) trainable parameters.

2. Training data: According to the paper abstract, the model was trained on 300 billion (3 × 10¹¹) tokens.

3. FLOPs per token for training a transformer:

   • Forward pass: each parameter is used in one multiply‑add operation → ~2 FLOPs per parameter → 2 N FLOPs per token.
   • Backward pass (gradient computation): roughly twice the forward cost → ~4 N FLOPs per token.
   • Total: ~6 N FLOPs per token.
     This approximation is standard for large transformer models and neglects small overheads like attention quadratic terms (which add only ~1–2% for GPT‑3’s sequence length).

4. Calculation:
   Total FLOPs ≈ 6 × (1.75 × 10¹¹) × (3 × 10¹¹)
   = 6 × 5.25 × 10²²
   = 3.15 × 10²³.

5. Consistency check: The GPT‑3 paper reports a training compute of 3,640 PF‑days. Converting:
   1 PF‑day = 10¹⁵ FLOP/s × 86400 s = 8.64 × 10¹⁹ FLOPs.
   3,640 PF‑days = 3,640 × 8.64 × 10¹⁹ ≈ 3.14 × 10²³ FLOPs, matching the estimate.

Thus, the total FLOPs required to train GPT‑3 is about 3.15 × 10²³.

Of course. Here is an estimate of the total number of FLOPs required to train GPT-3, based on my internal kn owledge and first principles, without searching online.

The Logic and Formula

To estimate the computational cost of training a large language model like GPT-3, we can use a well-establis hed formula. The total number of Floating Point Operations (FLOPs) is approximately:

Total FLOPs ≈ 6 × P × D

Let's break down what each part of this formula means:

• P: The number of parameters in the model. This is the total count of all the weights and biases that the model learns during training.
• D: The total number of tokens in the training dataset. This represents the volume of data the mo del processes.
• 6: This is a constant factor that provides a good approximation for transformer-based models. It com es from:
  • Forward Pass (1 × P × D): For each token processed, the number of operations is roughly proporti onal to the number of parameters. A common rule of thumb is 2 * P FLOPs per token for a forward pass (one multiplication and one addition for each parameter). So the total for one pass over the data is 2 * P * D.
  • Backward Pass (2 × Forward Pass): The backward pass (backpropagation), where the model calculate s gradients to update its parameters, is computationally more intensive. It's typically estimated to require about twice the operations of the forward pass. So, 2 * (2 * P * D) = 4 * P * D.
  • Total: Adding the forward and backward pass gives us 2PD + 4PD = 6PD.

Now, let's estimate the values for P and D for GPT-3.

---

Step-by-Step Estimation

Step 1: Estimate the Number of Parameters (P)

My internal knowledge base indicates that the largest version of GPT-3, which is the one most people refer t o, has a specific number of parameters.

• P ≈ 175 billion parameters
• In scientific notation, this is 175 x 10⁹

Step 2: Estimate the Training Dataset Size (D)

GPT-3 was trained on a massive corpus of text data scraped from the internet (like the Common Crawl dataset) , books, and Wikipedia. The key is not the raw size of the dataset, but the number of tokens actually used f or training. My recollection is that the model was trained for a single epoch on a curated dataset.

• D ≈ 300 billion tokens
• In scientific notation, this is 300 x 10⁹

Step 3: Perform the Calculation

Now we can plug these estimated values into our formula:

Total FLOPs ≈ 6 × P × D

1. Substitute the values for P and D: Total FLOPs ≈ 6 × (175 × 10⁹) × (300 × 10⁹)

2. Multiply the constants first: 6 × 175 = 1050 1050 × 300 = 315,000

3. Multiply the powers of 10: 10⁹ × 10⁹ = 10¹⁸

4. Combine the results: Total FLOPs ≈ 315,000 × 10¹⁸

5. Express this in standard scientific notation: Total FLOPs ≈ 3.15 × 10⁵ × 10¹⁸ = 3.15 × 10²³

---

Final Estimate and Conclusion

Based on my reasoning and the standard formula for calculating training costs, the estimated number of FLOPs required to train the 175B parameter GPT-3 model is approximately:

3.15 x 10²³ FLOPs

To put this astronomical number into perspective:

• A petaFLOP is 10¹⁵ FLOPs. Our estimate is 315,000,000 petaFLOPs.
• If you had a supercomputer that could perform one petaFLOP per second (10¹⁵ FLOPs/sec), it would take ** 3.15 x 10⁸ seconds** to complete the training.
• There are about 31.5 million seconds in a year, so this is equivalent to roughly 10 years of continu ous computation on that machine.

This calculation demonstrates why training foundational models like GPT-3 requires massive-scale data center s with thousands of GPUs running in parallel for months and is an undertaking that costs millions of dollars . The actual paper from OpenAI stated the training took approximately 3.6 x 10²³ FLOPs, so my estimate i s very close to the reported figure.

Here are 5 jokes:

1. Why did the scarecrow win an award? Because he was outstanding in his field.
2. I'm reading a book about anti-gravity. It's impossible to put down.
3. What do you call a fake noodle? An impasta.
4. Why don't skeletons fight each other? They don't have the guts.
5. I used to hate facial hair, but then it grew on me.

All 5 are unique.

Here are 5 jokes:

1. Why did the scarecrow win an award? Because he was outstanding in his field.

2. I have a joke about construction... But I'm still working on it.

3. What do you call a fake noodle? An Impasta.

4. I told my wife she was drawing her eyebrows too high. She looked surprised.

5. Why don't scientists trust atoms? Because they make up everything.

---

How many are unique?

Zero. The real joke is asking an AI for original content.