Estimate Complexity

Estimating the Total Number of FLOPs Required to Train GPT-3

Based on publicly available information about large language models (LLMs) and the specific parameters of GPT-3, we can derive a highly accurate estimate of the total FLOPs required for training.

Estimated Total FLOPs: $\approx 3.14 \times 10^{23}$

---

Step-by-Step Reasoning

The estimation relies on a well-established scaling law for Transformer models, often referred to as the "Chinchilla Law" (though applicable to GPT-3's architecture).

1. The Core Scaling Law

The total number of floating-point operations (FLOPs) required to train a dense Transformer model is approximated by the following formula:

$$\text{FLOPs} \approx 6 \times P \times D$$

Where:

• $P$ (Parameters): The total number of non-embedding parameters in the model.
• $D$ (Dataset Size/Tokens): The total number of training tokens the model processes.
• $6$ (The Multiplier): This factor accounts for the complexity of the forward and backward passes during training. Specifically:
  • 2: For the forward pass (matrix multiplications involved in attention and feed-forward layers).
  • 4: For the backward pass (calculating gradients for the weights and inputs).

2. Identifying GPT-3's Key Parameters ($P$)

We know from the GPT-3 paper ("Language Models are Few-Shot Learners") that the largest model variant, which we refer to as GPT-3, has:

$$P = 175 \text{ Billion Parameters} = 1.75 \times 10^{11}$$

3. Identifying GPT-3's Training Token Count ($D$)

The paper also specifies the total volume of data used for training the 175B model:

$$D = 300 \text{ Billion Tokens} = 3.0 \times 10^{11}$$

(Note: While some training runs use more tokens, the original GPT-3 paper specifies 300 billion tokens for the final 175B model train. This contrasts with modern models like Chinchilla, which use significantly more tokens relative to their parameter count.)

4. Calculation

Now we substitute these values into the scaling law:

$$\text{FLOPs} = 6 \times P \times D$$

$$\text{FLOPs} = 6 \times (1.75 \times 10^{11}) \times (3.0 \times 10^{11})$$

Intermediate Calculations:

1. Multiply the numerical coefficients: $$6 \times 1.75 \times 3.0 = 10.5 \times 3.0 = 31.5$$

2. Multiply the powers of 10: $$10^{11} \times 10^{11} = 10^{(11+11)} = 10^{22}$$

Final Result:

$$\text{FLOPs} = 31.5 \times 10^{22}$$

5. Standardizing the Notation

To express the result in standard scientific notation (where the coefficient is between 1 and 10):

$$\text{FLOPs} = 3.15 \times 10^{23}$$

---

Summary of Estimate

The total estimated number of FLOPs required to train the 175-billion parameter GPT-3 model on 300 billion tokens is:

$$\approx 3.15 \times 10^{23} \text{ FLOPs}$$

(Self-Correction/Refinement: Often, due to various overheads (like non-matrix multiplication operations, which are minor but present), the actual figure cited by OpenAI is slightly lower or rounded, often cited as $3.14 \times 10^{23}$ or $3.2 \times 10^{23}$. Our direct calculation of $3.15 \times 10^{23}$ is accurate based on the standard scaling law.)