Below is a simple "back-of-the-envelope" estimate that uses only a few public facts about GPT-3 and very generic rules of thumb for counting operations.

Step 1. Number of learned parameters
GPT-3's largest version contains ≈175 billion (1.75 × 10¹¹) trainable parameters.

Step 2. How much data was seen during training
OpenAI reports that the model was trained on about 300 billion tokens (3 × 10¹¹).

Step 3. How many floating-point operations does one parameter cost per token?
A practical rule of thumb for Transformer language models is
 Training FLOPs ≈ 6 × (# parameters) × (# training tokens)

Where does the factor 6 come from?

• Forward pass: every parameter is involved in one multiply-accumulate → 2 FLOPs
• Back-prop gradient computation: roughly another 2 FLOPs
• Parameter-update (optimizer) and miscellaneous overhead: ≈2 FLOPs
Total ≈6 FLOPs per parameter per token.

Step 4. Plug in the numbers
Total FLOPs = 6 × (1.75 × 10¹¹) × (3 × 10¹¹)
      = 6 × 5.25 × 10²²
      ≈ 3.15 × 10²³ FLOPs.

Step 5. Express the answer in more familiar units
3.15 × 10²³ ≈ 315 zettaFLOPs (1 ZFLOP = 10²¹ FLOPs).

Step 6. Sanity check with hardware-time intuition
An NVIDIA A100 delivers ≈ 312 teraFLOP/s (0.312 PFLOP/s) in mixed precision.
Wall-clock time on a single A100:
 3.15 × 10²³ / 3.12 × 10¹⁴ ≈ 1.0 × 10⁹ s ≈ 32 years.
Running on 1,000 A100s in parallel would take ≈12 days, which is the right ballpark for a large industrial training run.

Final estimate
Training GPT-3 (175 B parameters, 300 B tokens) requires on the order of 3 × 10²³ floating-point operations, i.e. roughly 300 zettaFLOPs.

To estimate the total number of FLOPs required to train GPT-3 without searching online, we can make several assumptions and perform calculations based on the information available about GPT-3's architecture and training process. Here's a step-by-step explanation of the reasoning:
Step 1: Determine the number of parameters in GPT-3

GPT-3 is a massive language model with 175 billion parameters. Each parameter represents a trainable weight in the neural network, and updating these weights during training requires floating-point operations (FLOPs).

Step 2: Estimate the number of training iterations

The number of training iterations refers to how many times the entire training dataset is passed through the neural network during the training process. Although the exact number may vary, let's assume that GPT-3 was trained for 100 epochs.

Step 3: Calculate the FLOPs per iteration

During each training iteration, the neural network performs forward and backward passes. The forward pass involves computing the output of the network given the input, while the backward pass involves calculating the gradients of the loss function with respect to the network's weights.
For simplicity, we can assume that the forward and backward passes have roughly the same number of FLOPs. Therefore, the total FLOPs per iteration can be estimated by multiplying the number of parameters by 2 (forward and backward passes).

Total FLOPs per iteration = (Number of parameters) x 2

(175 billion parameters) x 2 = 350 billion FLOPs per iteration

Step 4: Calculate the total FLOPs for training

Multiplying the FLOPs per iteration by the number of training iterations gives us the total FLOPs required to train GPT-3.

Total FLOPs for training = (FLOPs per iteration) x (Number of iterations)

(350 billion FLOPs per iteration) x (100 iterations) = 35 trillion FLOPs for training

Therefore, based on these assumptions, we estimate that training GPT-3 without searching online would require approximately 35 trillion FLOPs.

Let
• S = number of sisters in the family (girls)
• B = number of brothers in the family (boys)

Given

1. Sally has 3 brothers ⇒ B = 3.
2. Each brother has 2 sisters ⇒ every boy sees exactly 2 girls in the sibling group.
   Therefore S = 2.

Since Sally herself is one of those 2 sisters, the number of sisters Sally has (other than herself) is

2 – 1 = 1.

Sally has 1 sister.

Sally has 2 sisters.

No.
Think of 9.9 as 9.90; since 0.11 < 0.90, we have 9.11 < 9.9.

9.11 is less than 9.9.