Estimate Complexity

Of Of course. Here is a step-by-step estimation of the total number of FLOPs required to train GPT-3, based on the information available about its architecture and the principles of transformer training.

Executive Summary

The estimated total FLOPs for training GPT-3 (the 175B parameter model) is approximately 3.14 x 10²³ FLOPs.

This estimate is derived from a standard calculation for training large language models, which depends primarily on the number of parameters, the size of the training dataset, and the "Chinchilla" scaling laws, which suggest a compute-optimal training regime.

---

Detailed Step-by-Step Reasoning

Step 1: Identify the Key Parameters for the Calculation

To estimate the total Floating-Point Operations (FLOPs), we need three key pieces of information about the training process:

1. Number of Parameters (N): The total number of trainable weights in the model. For GPT-3, this is famously 175 billion (175 x 10⁹).
2. Size of the Training Dataset (D): The total number of tokens used for training. GPT-3 was trained on approximately 300 billion (300 x 10⁹) tokens.
3. Compute Multiplier (C): A rule of thumb that relates the number of parameters to the total FLOPs required for a single training step. This is based on empirical observations and scaling laws.

Step 2: Establish a Formula for Training FLOPs

A widely accepted and standard formula for estimating the total FLOPs required for training a large transformer model is:

Total Training FLOPs ≈ 6 * N * D

Let's break down why this formula is used:

• N (Parameters): Each parameter (weight) in the model needs to be updated during backpropagation.
• D (Tokens): The model must process the entire dataset. The number of tokens determines how many times the model needs to perform a forward and backward pass.
• 6 (The Multiplier): This is the most crucial part. It's an empirically derived factor that accounts for the work done in a single training step for one token. For each token, the computation involves:
  • Forward Pass (1x): Performing the matrix multiplications and other operations to get a prediction. A good estimate is that this requires roughly one FLOP per parameter per token.
  • Backward Pass (2x): Calculating the gradients for each parameter. This is roughly twice as computationally expensive as the forward pass.
  • Weight Updates (3x): Applying the calculated gradients to update the model's parameters. This is roughly on par with the forward pass.

This gives a total of 1 (forward) + 2 (backward) + 3 (updates) = 6 FLOPs per parameter per token. While these are simplifications, this 6x multiplier has proven to be a robust estimate for large-scale training.

Step 3: Apply the Formula to GPT-3's Specifications

Now, we plug in the known values for GPT-3:

• N = 175 billion = 1.75 x 10¹¹
• D = 300 billion = 3.00 x 10¹¹

Calculation:

Total FLOPs = 6 * (1.75 x 10¹¹) * (3.00 x 10¹¹) Total FLOPs = 6 * (5.25 x 10²²) Total FLOPs = 3.15 x 10²³

Step 4: Refine the Estimate with Modern Scaling Laws (The Chinchilla Check)

The 6 * N * D formula is a good rule of thumb, but more recent research from DeepMind (the "Chinchilla" paper) has shown that for optimal training, the number of tokens should be proportional to the number of parameters (specifically, about 20 tokens per parameter).

Let's see how this applies to GPT-3:

• GPT-3's Data/Parameter Ratio: 300 billion tokens / 175 billion parameters ≈ 1.7 tokens per parameter.

This shows that GPT-3 was significantly undertrained according to the Chinchilla laws—it was trained with far more parameters than the data could optimally support. This means our initial estimate is likely a slight overestimate of the compute needed if GPT-3 were trained in a "compute-optimal" way.

However, our goal is to estimate the FLOPs for training the actual GPT-3 model as it was trained. The 6 * N * D formula directly uses the actual parameters and dataset size, making it the most appropriate choice. The Chinchilla findings simply provide context that GPT-3's training was not compute-optimal.

Step 5: Contextualize the Result

To understand the scale of 3.14 x 10²³ FLOPs:

• Petaflop/s-days: This unit is often used in research papers. One Petaflop/s-day is equivalent to 10¹⁵ FLOPs sustained for one day.
  • Total FLOPs / (10¹⁵ FLOPs/day) = (3.14 x 10²³) / (10¹⁵) = 3.14 x 10⁸ Petaflop/s-days.
  • This is 314 million Petaflop/s-days.
• A100 GPU Equivalent: A single NVIDIA A100 GPU can deliver about 312 Teraflops (3.12 x 10¹¹ FLOPs) of sustained performance for these tasks.
  • If one A100 could run non-stop, it would take: (3.14 x 10²³ FLOPs) / (3.12 x 10¹¹ FLOPs/sec) ≈ 1 x 10¹² seconds ≈ 31,700 years.
  • This highlights why training such a model requires a massive supercomputer with thousands of GPUs running in parallel for weeks or months.

Conclusion

Based on the standard formula Total FLOPs = 6 * N * D, and using GPT-3's published specifications of 175 billion parameters and a 300 billion token training dataset, the estimated total FLOPs required for training is 3.14 x 10²³. This figure aligns with the scale of computation reported by OpenAI and provides a realistic order-of-magnitude estimate.