Estimate Complexity

Okay, let's estimate the FLOPs for training GPT-3. I'll break this down.

Key Formula: The number of FLOPs for training a transformer-based model can be roughly estimated as: FLOPs ≈ 6 * N * D Where:

• N is the number of parameters in the model.
• D is the total number of tokens processed during training (dataset size in tokens * number of epochs).
• The factor of 6 comes from:
  • 2 * N FLOPs per token for the forward pass (each parameter is involved in roughly one multiply-accumulate, and a MAC is 2 FLOPs).
  • 4 * N FLOPs per token for the backward pass (gradient computation is roughly twice as expensive as the forward pass).

Assumptions for GPT-3 (Davinci, the largest version):

1. Number of Parameters (N):

   • GPT-3 (the largest variant, Davinci) is famously stated to have 175 billion parameters.
   • So, N = 175 * 10^9.

2. Number of Tokens Processed (D):

   • This is the trickier part to recall without searching. I remember it was trained on a massive amount of data, a significant portion of Common Crawl, books, Wikipedia, etc.
   • Large Language Models are typically trained for roughly one epoch over their massive datasets, or even less if datasets are mixed and weighted. The key figure is the total number of tokens seen by the model.
   • I seem to recall a figure in the range of 300 billion to 500 billion tokens for GPT-3's training. The number often cited in discussions around the GPT-3 paper and subsequent scaling law papers (like Chinchilla) is around 300 billion tokens. Let's use that as a reasonable estimate for D.
   • So, D = 300 * 10^9.

Calculation:

Now, let's plug these into the formula: FLOPs ≈ 6 * N * D FLOPs ≈ 6 * (175 * 10^9) * (300 * 10^9)

Let's do the numerical part: 6 * 175 = 1050 1050 * 300 = 315,000

Now, the powers of 10: 10^9 * 10^9 = 10^18

So, combining these: FLOPs ≈ 315,000 * 10^18 FLOPs ≈ 3.15 * 10^5 * 10^18 FLOPs ≈ 3.15 * 10^(5+18) FLOPs ≈ 3.15 * 10^23

Expressing the Result:

• TeraFLOPs (TFLOPs) = 10^12
• PetaFLOPs (PFLOPs) = 10^15
• ExaFLOPs (EFLOPs) = 10^18
• ZettaFLOPs (ZFLOPs) = 10^21
• YottaFLOPs (YFLOPs) = 10^24

So, 3.15 * 10^23 FLOPs can be written as:

• 315 * 10^21 FLOPs = 315 ZettaFLOPs
• Or, as 0.315 * 10^24 FLOPs = 0.315 YottaFLOPs

The number is often quoted in PetaFLOP/s-days or similar. Let's convert 3.15 * 10^23 FLOPs to PetaFLOP-seconds: 3.15 * 10^23 / 10^15 = 3.15 * 10^8 PetaFLOP-seconds.

Number of seconds in a day = 24 hours/day * 60 minutes/hour * 60 seconds/minute = 86,400 seconds/day. So, 3.15 * 10^8 PetaFLOP-seconds / 86,400 seconds/day ≈ 3.15 * 10^8 / (8.64 * 10^4) PetaFLOP-days ≈ (3.15 / 8.64) * 10^(8-4) PetaFLOP-days ≈ 0.3646 * 10^4 PetaFLOP-days ≈ 3646 PetaFLOP/s-days (if executed on a machine capable of 1 PFLOP/s constantly for that many days, or scaled accordingly).

Reasoning Summary:

1. Identified the core formula: FLOPs ≈ 6 * N * D for transformer training.
2. Recalled model parameters (N): GPT-3 Davinci has 175 billion parameters.
3. Estimated training tokens (D): Based on common knowledge about LLM training and GPT-3 specifics, estimated ~300 billion tokens processed. This assumes roughly one pass over the (weighted) training data.
4. Performed the multiplication: 6 * (175 * 10^9) * (300 * 10^9) = 3.15 * 10^23 FLOPs.
5. Interpreted the result: This is approximately 315 ZettaFLOPs. This also aligns with the often-cited figure for GPT-3 training compute, which is in the range of 3000-4000 PetaFLOP/s-days (my calculation of ~3646 PetaFLOP/s-days is right in this ballpark).

Caveats:

• The 6ND is an approximation. The exact number can vary slightly based on the specific transformer architecture details (e.g., use of sparse attention, specific normalization layers), optimizer overhead, and whether techniques like activation checkpointing are used (which can increase FLOPs for recomputation but save memory).
• Mixed-precision training (e.g., bfloat16 or fp16) doesn't change the number of FLOPs, but it changes the type of FLOPs and allows for faster computation on hardware that supports it.
• The number of training tokens D is an estimate. The GPT-3 paper describes a complex data mixture and weighting schedule, so "300 billion tokens" is an effective number of tokens processed.

Therefore, my estimate is approximately 3.15 x 10^23 FLOPs or 315 ZettaFLOPs.