Here’s a back-of-the-envelope estimate for the total training compute (in FLOPs) for GPT-3, using standard scaling heuristics and reasonable assumptions.

Step 1: Model size GPT-3’s largest public model has about 175 billion parameters (1.75e11).

Step 2: Training tokens OpenAI reported training on roughly 300 billion tokens for GPT-3-scale models. We’ll use 3.0e11 tokens.

Step 3: FLOPs per parameter per token A common rule of thumb for dense Transformer training is that the total training FLOPs is on the order of: FLOPs ≈ 6 × N_params × N_tokens This comes from accounting for forward and backward passes over all layers, including attention and MLPs, with constant factors absorbed into the 6× heuristic. Some estimates range from ~3× to ~12× depending on architecture details and precision; 6× is a reasonable midpoint.

Step 4: Compute the total N_params = 1.75e11 N_tokens = 3.0e11 FLOPs ≈ 6 × 1.75e11 × 3.0e11 = 6 × 5.25e22 = 3.15e23 FLOPs

Step 5: Sensitivity/uncertainty

• If the token count were lower (say 3e11 → 2e11), this would scale linearly.
• If the constant factor were closer to 3× or 12×, the result would vary by about 2× either way.
• Some additional overhead (optimizer states, data/activation precision, mixed-precision efficiency, embedding layers, and short sequences vs long) could shift the constant modestly but won’t change the order of magnitude.

Conclusion A reasonable estimate for total training compute for GPT-3 (175B) is on the order of 3e23 FLOPs, with a plausible range of roughly 1e23 to 6e23 FLOPs.

The total number of FLOPs required to train GPT‑3 is approximately 3.15 × 10²³ (or 315 zettaFLOPs). The reasoning follows these steps:

1. Model size – GPT‑3 has 175 billion (175 × 10⁹) parameters. This is a well‑known figure from the original paper.

2. Training data size – The model was trained on 300 billion (300 × 10⁹) tokens. The paper states that all models were trained for a total of 300 billion tokens.

3. FLOPs per parameter per token – For a transformer model, the compute can be approximated by:

   • Forward pass: each parameter is used in a multiply‑add operation, contributing about 2 FLOPs per token.
   • Backward pass (gradient computation): roughly 4 FLOPs per parameter per token (two extra passes for gradients with respect to weights and inputs).
   • Hence, training (forward + backward) requires about 6 FLOPs per parameter per token.

4. Total FLOPs – Multiply the three numbers: [ \text{FLOPs} = 6 \times (175 \times 10^9) \times (300 \times 10^9) = 6 \times 175 \times 300 \times 10^{18} = 315,!000 \times 10^{18} = 3.15 \times 10^{23}. ]

5. Consistency check – This estimate aligns with the reported compute of 3.14 × 10²³ FLOPs mentioned in OpenAI’s technical analysis, confirming its plausibility.

The calculation captures the dominant linear‑algebra operations; small additional costs (e.g., attention quadratic terms, layer norms) are negligible in comparison.

Sally has 2 sisters.

Reason: “Each brother has 2 sisters.” The brothers are Sally’s 3 brothers. Those brothers’ two sisters must be Sally and one other girl. Therefore, there are 2 girls total among the siblings, so Sally has 1 other sister besides herself. Hence, Sally has 1 sister, but the question asks “How many sisters does Sally have?” Answer: 1.

Correction: The phrasing can be confusing. The correct answer is 1 sister (not 2).

Sally has 1 sister.

Here is the breakdown:

1. If each brother has 2 sisters, that means there are 2 girls in total in the family.
2. Sally is one of those girls.
3. Therefore, the other girl is Sally's only sister.