Below is a “back-of-the-envelope” that stays within a factor of ~2 of the number most practitioners quote (≈3×10²³ FLOP).
We will do it twice—once with the simplest possible formula (parameter count × 6 × tokens) and once with a slightly more careful treatment of the actual tensor shapes inside the transformer. Both converge to the same ball-park.

1. What we know publicly (no web look-ups)

• OpenAI said GPT-3 has 175 B parameters.
• They also said it was trained on “about 300 B tokens” (that figure is in the original paper).
• The model is a dense, decoder-only transformer trained with vanilla autoregressive language-modeling (next-token prediction).
• Training used the usual Adam optimiser with weight-decay; no fancy MoE or pipeline-parallel tricks that would change the arithmetic intensity.

2. Quick first cut: the “6×” rule

For any dense transformer that is trained with the standard forward+backward pass, the total floating-point operations are very close to

    FLOP ≈ 6 × N × D

where N = number of non-embedding parameters
D = number of tokens processed during the whole run.

The factor “6” comes from:

• 2 mul–adds per parameter for the forward pass (weight multiply + bias add)
• 4 mul–adds per parameter for the backward pass (grad w.r.t. weights and activations).

Plugging in the public numbers:

    N = 1.75 × 10¹¹
    D = 3 × 10¹¹

    FLOP = 6 × 1.75e11 × 3e11
          ≈ 3.15 × 10²³

That is already within the canonical range.

3. Second cut: walk through the transformer dimensions

We can sanity-check the “6×” rule by actually counting the multiply–adds inside one forward pass and then doubling for the backward.

3.1 Model shape

• d_model = 12288 (12 k) ← well-known from the paper
• n_layers = 96
• n_heads = 96
• Vocab = 50 k (we’ll keep it symbolic, V = 5e4)
• Intermediate FFN multiplier = 4 → 4d_model = 49 k

3.2 Parameters per layer

• QKV projection: 3 matrices of shape (d_model, d_model) → 3 d_model²
• Output projection: 1 matrix (d_model, d_model) → d_model²
• FFN: 2 matrices (d_model, 4d_model) and (4d_model, d_model) → 8 d_model²
  Total per layer = (3+1+8) d_model² = 12 d_model²
  Embedding matrix (shared input/output) = V d_model

Total non-embedding parameters
N_layer = 96 × 12 d_model² = 1152 d_model²
N_emb = V d_model
Plug d_model = 12288
d_model² ≈ 1.51 × 10⁸
N_layer ≈ 1152 × 1.51e8 ≈ 1.74 × 10¹¹
N_emb ≈ 5e4 × 1.22e4 ≈ 6.1 × 10⁸
So N ≈ 1.75 × 10¹¹, matching the headline number.

3.3 FLOPs per token in the forward pass
For every layer we do one matrix multiply for each of the four attention projections, one for the attention output, and two for the FFN. All of them are of the form

    Y = X W      (batch×seq, d) × (d, d′) → (batch×seq, d′)

with d′ either d_model or 4d_model. Counting multiply–adds as 2 FLOPs per scalar:

• Attention QKV: 3 × (seq·d_model²) × 2
• Attention out: 1 × (seq·d_model²) × 2
• FFN: 2 × (seq·4d_model²) × 2
  Total per layer = 2 seq d_model² (3 + 1 + 8) = 24 seq d_model²
  Over 96 layers: 96 × 24 seq d_model² = 2304 seq d_model²

Add the embedding look-up (sparse, but we usually treat it as dense for an upper bound): seq V d_model × 2
Total forward FLOPs per token = 2304 d_model² + 2 V d_model
Numerically:

    2304 × 1.51e8 ≈ 3.48 × 10¹¹  
    2 V d_model ≈ 1.2 × 10⁹  (negligible)

So ≈ 3.5 × 10¹¹ FLOP per token forward.

Backward pass is again ~2× forward (same multiplies, but both grad-w and grad-x), so total 3× forward = 1.05 × 10¹² FLOP per token.

3.4 Scale to 300 B tokens
1.05e12 × 3e11 = 3.15 × 10²³ FLOP

4. Conclusion

Both the ultra-simple “6 N D” rule and the detailed per-layer count land at

    ≈ 3 × 10²³ floating-point operations

for the entire GPT-3 training run.

Below, I'll provide a step-by-step estimation of the total floating-point operations (FLOPs) required to train GPT-3, based on my built-in knowledge of the model (from its original 2020 paper and related analyses). I'll avoid any external searches or real-time data lookups, relying solely on established facts and standard approximation methods for large language model (LLM) training. This is an estimate, as exact figures can vary slightly based on implementation details (e.g., optimizer choices, mixed-precision training, or hardware efficiencies), but it should be reasonably accurate.

Step 1: Recall Key Specifications of GPT-3

• Number of parameters (N): GPT-3 is a transformer-based model with 175 billion parameters. This is the total number of trainable weights in the model (often denoted as 175 × 10^9 or 1.75 × 10^11).
• Training dataset size (T): GPT-3 was trained on a massive dataset filtered from sources like Common Crawl, WebText2, Books1, Books2, and Wikipedia. Based on the original paper, this amounts to approximately 300 billion tokens (3 × 10^11 tokens). Tokens here refer to subword units (e.g., via Byte-Pair Encoding), which is the standard way to measure LLM training data volume.
  • Note: This is a one-pass training setup (the model sees the data once), without heavy data augmentation or multiple epochs, which is typical for such large-scale pre-training.

These figures are foundational and well-documented in the literature, so they're a solid starting point for estimation.

Step 2: Understand the FLOPs Calculation for Transformer Training

To estimate training compute, we use a well-established rule of thumb for transformer models like GPT-3. The total FLOPs is approximated as:

Total FLOPs ≈ 6 × N × T

• Why this formula?
  • In transformer architectures (which GPT-3 uses), training involves processing the input sequence token by token.
  • For each token:
    • Forward pass: Involves matrix multiplications and activations across all layers, costing roughly 2 FLOPs per parameter (one for the multiply, one for the add in GEMM operations, scaled by the model's dense structure).
    • Backward pass (gradient computation): This is roughly twice as expensive as the forward pass due to backpropagation through the chain rule, adding about 4 FLOPs per parameter.
    • Total per token: ~6 FLOPs per parameter.
  • This 6× factor is a standard approximation derived from detailed analyses (e.g., by Kaplan et al. in their scaling laws work and confirmed in the GPT-3 paper). It assumes full-precision floating-point ops and ignores minor overheads like activations or embeddings (which are small relative to the dominant matrix multiplies).
  • The formula scales linearly with N (more parameters mean more ops per token) and T (more data means more tokens processed).
  • Caveats: Real-world training might be 10-20% more or less efficient due to optimizations (e.g., FP16 precision reduces FLOPs but increases memory use, or sparsity isn't used in base GPT-3). But for estimation, 6×N×T is widely accepted and matches reported figures for models like GPT-3.

This doesn't include pre-training setup costs (e.g., data curation) or fine-tuning, just the core pre-training phase, which dominates the total compute.

Step 3: Plug in the Numbers

Now, let's compute it explicitly:

• N = 175 × 10^9 = 1.75 × 10^11
• T = 300 × 10^9 = 3 × 10^11
• First, compute N × T = (1.75 × 10^11) × (3 × 10^11) = 5.25 × 10^22
• Then, multiply by 6: 6 × 5.25 × 10^22 = 3.15 × 10^23

So, the estimated total FLOPs is 3.15 × 10^23 (or 315 septillion FLOPs, in plain terms).

Step 4: Validate the Estimate with Context

• This aligns closely with the figure cited in the GPT-3 paper and follow-up analyses (around 3.14 × 10^23 FLOPs), confirming the approximation's accuracy.
• For perspective:
  • Training GPT-3 took about 1 month on a cluster of thousands of GPUs (e.g., NVIDIA A100s or V100s), each capable of ~10^14-10^15 FLOPs per second. At scale, this adds up to the estimated total.
  • Smaller models provide a sanity check: For example, GPT-2 (1.5 billion params, ~40 billion tokens) uses ~6 × 1.5e9 × 4e10 ≈ 3.6 × 10^20 FLOPs, which is ~1,000× less than GPT-3, matching the scaling.
• Uncertainties: If the token count was slightly higher (e.g., 410 billion in some filtered subsets), it could push the estimate to ~4 × 10^23. But 300 billion is the conservative, paper-stated value.

Step 5: Implications and Limitations

This massive compute requirement highlights why training models like GPT-3 is resource-intensive—it's equivalent to billions of human-years of calculation if done serially. In practice, it's distributed across supercomputers. If you're interested in breakdowns for inference FLOPs, comparisons to other models (e.g., GPT-4 estimates), or ways to optimize, let me know!

This estimation is based purely on my training data up to my last update, without any online queries. If new details have emerged since then, they wouldn't affect this historical estimate.

1

Explanation

The reasoning starts by establishing the family structure based on the given information. Sally, as a girl, has 3 brothers, indicating at least 4 children in the family: Sally and her 3 brothers.

Next, the statement that each brother has 2 sisters is analyzed. Since Sally is one sister to each brother, an additional sister is required to account for the second sister per brother.

This leads to a total of 5 children: Sally, 1 other sister, and 3 brothers. In this configuration, each brother has exactly 2 sisters (Sally and the other sister).

From Sally's viewpoint, she has 3 brothers and shares the family with 1 other sister, confirming she has 1 sister. This structure satisfies all conditions without requiring additional siblings.

Below, I'll generate 5 jokes as requested. They're a mix of AI-themed and nature-inspired ones (fitting for an AI built by Oak AI, with a nod to Sonoma's oak-filled vibes). I'll number them for clarity. After that, I'll answer your question about uniqueness.

1. Why did the AI go to therapy? It had too many unresolved issues!

2. Why did the AI go to therapy? It had too many unresolved issues!

3. Why don't oak trees use computers? They have too many branches to handle!

4. What do you call an AI that's always late? A procrastin-bot!

5. Why don't oak trees use computers? They have too many branches to handle!

How many are unique?

Out of these 5 jokes, there are 3 unique ones. (Jokes 1 and 2 are identical, and jokes 3 and 5 are identical, leaving three distinct jokes overall.) If you'd like more jokes or a different twist, just let me know! 😊