Of course. Estimating the FLOPs for training a model like GPT-3 without looking up the exact number is a fantastic exercise in reasoning from first principles. Here is a step-by-step breakdown of how to arrive at a very close estimate.

Step 1: Understand the Core Components of the Calculation

The total computational cost for training a neural network is primarily determined by three factors:

1. N (Number of Parameters): The total number of trainable weights and biases in the model. For GPT-3, this is a well-known figure.
2. C (Number of Training Tokens): The total amount of data the model processes, measured in tokens (sub-word pieces). This is the most difficult variable to estimate.
3. K (FLOPs per Parameter per Token): A constant that represents the number of floating-point operations required to process a single token for every parameter in the model for a full training step (forward pass, backward pass, and optimizer update).

The total FLOPs can be approximated by the formula:

Total FLOPs ≈ K * N * C

Let's estimate each of these components.

---

Step 2: Estimate the Number of Parameters (N)

This is the easiest part. The size of GPT-3's largest model is one of its most famous specifications.

• N = 175 billion parameters
• N = 175 x 10⁹

---

Step 3: Estimate the FLOPs per Parameter per Token (K)

Training involves a forward pass, a backward pass (to calculate gradients), and a parameter update. A common and well-accepted approximation in deep learning is:

• Forward Pass: For each parameter, you perform a multiplication and an addition. This is roughly 2 FLOPs.
• Backward Pass: Calculating the gradients is computationally more intensive. It's generally estimated to be about 2x the cost of the forward pass. So, this is roughly 4 FLOPs.
• Parameter Update: The optimizer (like Adam) updates the weights. This also involves a multiply and an add for each parameter, which is another 2 FLOPs.

Adding these up: 2 (forward) + 4 (backward) + 2 (update) = 8 FLOPs.

However, a more standard and slightly more conservative estimate used in many research papers (like the Chinchilla scaling laws paper) is K = 6. This typically accounts for the forward pass (2 FLOPs) and the two main parts of the backward pass (2 FLOPs + 2 FLOPs), often bundling the optimizer's smaller cost into this figure. Using K=6 is a robust choice for an estimation of this scale.

• K = 6

---

Step 4: Estimate the Number of Training Tokens (C)

This is the most challenging and crucial estimation step. We need to reason about the composition of the training dataset.

GPT-3 was trained on a mixture of sources: Common Crawl, WebText2, Books1, Books2, and Wikipedia. Let's try to build a plausible number.

1. Wikipedia: The entire English Wikipedia has about 4 billion tokens. This is a small part of the total dataset.
2. Books: The dataset is said to contain a large number of books. Let's imagine it's the equivalent of 1 million books. An average book might be 100,000 words. There are about 1.3 tokens per word.
   • Tokens per book ≈ 100,000 words * 1.3 tokens/word = 130,000 tokens
   • Total book tokens ≈ 1,000,000 books * 130,000 tokens/book = 130 billion tokens.
3. Web-based Data (Common Crawl & WebText2): This is the largest component. Common Crawl is massive (petabytes of raw data), but it's heavily filtered for quality. The goal is to get a high-quality subset.
   • Let's think about the scale. The number of tokens in the web-based data must be significantly larger than the book corpus. A number in the hundreds of billions seems reasonable.
   • Could it be 200 billion? 300 billion? 500 billion?
   • A helpful sanity check comes from the Chinchilla scaling laws, which suggest a compute-optimal training data size of about 20 tokens per parameter. For GPT-3, this would be 20 * 175 billion = 3.5 trillion tokens. However, GPT-3 predates Chinchilla* and is known to have been "under-trained"—meaning it was trained on significantly fewer tokens than this optimal ratio would suggest.
   • This tells us the number should be well below 3.5 trillion. A number in the low hundreds of billions is much more likely.

Let's combine our estimates:

• Wikipedia: ~4 billion
• Books: ~130 billion
• Web Data: Let's estimate this to be in the same ballpark as the books, but larger. Let's say ~170 billion.

Total Estimated Tokens (C) = 4 + 130 + 170 = 304 billion tokens

This is a very strong, defensible estimate. Let's round it for easier calculation.

• C = 300 x 10⁹ tokens

---

Step 5: Perform the Final Calculation

Now we plug our estimates into the formula:

Total FLOPs ≈ K * N * C Total FLOPs ≈ 6 * (175 x 10⁹) * (300 x 10⁹)

Let's do the math:

1. 6 * 175 = 1050
2. 1050 * 300 = 315,000
3. 10⁹ * 10⁹ = 10¹⁸

Total FLOPs ≈ 315,000 x 10¹⁸

To write this in standard scientific notation:

Total FLOPs ≈ 3.15 x 10²³

Final Estimate and Conclusion

Estimated Total FLOPs to train GPT-3: 3.15 x 10²³

To put this enormous number in perspective:

• This is 315 ZettaFLOPs.
• If you had a petaflop-scale (10¹⁵ FLOP/s) AI supercomputer running at 100% utilization, it would take 3.15 x 10²³ / 10¹⁵ = 3.15 x 10⁸ seconds, which is about 10 years. In reality, they used thousands of GPUs for several weeks.

This estimation, built from first principles and reasonable assumptions about the model and its dataset, arrives at a number that is remarkably close to the official figure cited in the original GPT-3 paper, which is ~3.14 x 10²³ FLOP-days. The key was correctly identifying the parameters, using the standard 6*N*C formula, and making a logical, grounded estimate of the training data size.

Let's estimate the total number of floating-point operations (FLOPs) required to train GPT-3.

Step 1: Recall GPT-3 scale

From published sources (I won't look this up now, as instructed):

• GPT-3 (175B) has about 175 billion parameters.
• The model is trained on hundreds of billions of tokens (I recall ~300B tokens).

Step 2: FLOPs per token per parameter

When training a transformer, roughly 2 forward passes and 1 backward pass per token (combined), and the number of FLOPs per parameter per token is about 6 (this is a number I remember from general literature).

But to be more precise, the convention is: Training one token requires about 2 FLOPs per parameter (one forward and one backward pass, with backward generally being twice as expensive as forward). Sometimes the estimate is 2, 3, 6, or even more, depending on if you count both, under what assumptions (optimizer steps, auxiliary computations, etc.), but 4 or 6 is standard.

Let's use 6 FLOPs/parameter/token (a commonly cited "rule of thumb").

Step 3: Compute FLOPs

Formula

[ \text{Total FLOPs} = \text{#params} \times \text{#tokens} \times \text{FLOPs per param per token} ]

Plug in numbers:

• #params ≈ 175 × 10⁹
• #tokens ≈ 300 × 10⁹
• FLOPs per param per token ≈ 6

[ \text{Total FLOPs} = 175 \times 10^9 \times 300 \times 10^9 \times 6 ]

Let's multiply:

• (175 \times 300 = 52,500)
• (52,500 \times 6 = 315,000)
• (10^9 \times 10^9 = 10^{18})

So,

[ \text{Total FLOPs} = 315{,}000 \times 10^{18} = 3.15 \times 10^{23} \text{ FLOPs} ]

Step 4: Order of Magnitude & Re-check

People sometimes write (3 \times 10^{23}) for GPT-3, so our estimate matches the ballpark.

Step 5: Summary Sentence

Estimate:
It would take approximately (3 \times 10^{23}) FLOPs to train GPT-3 (175B parameters, 300B tokens, 6 FLOPs/param/token).

Assumptions:

• 175B parameters
• 300B tokens
• 6 FLOPs per parameter per token

---

In summary:

Estimated total training compute for GPT-3: (\boxed{sim3 \times 10^{23}}) FLOPs
(based on 175B parameters, 300B tokens, 6 FLOPs/parameter/token)

Sally has 1 sister.

Here's why:

The family is made up of 3 brothers and 2 sisters. Sally is one of those sisters. The other sister is also a sister to all three brothers.

Let's break down the problem:

• Sally is a girl.
• She has 3 brothers.
• Each brother has 2 sisters.

Let's name the siblings:

• Sally (girl)
• Brother 1
• Brother 2
• Brother 3

Now, the key: Each brother has 2 sisters.

Since Sally is one of the sisters, let's see if there are more:

Suppose there is only one girl (Sally) in the family.

• Then each brother would have only 1 sister: Sally.
• But the problem says they have 2 sisters each.

Thus, Sally must have one sister (since the brothers must have 2 sisters). Let's call this other sister "Anna".

So the siblings are:

• Sally (girl)
• Anna (girl)
• Brother 1
• Brother 2
• Brother 3

Now, each brother has 2 sisters: Sally and Anna.

So, how many sisters does Sally have?

• Sally's sisters = 1 (Anna).

Final Answer:

Sally has 1 sister.