Estimate Complexity

To estimate the total number of Floating Point Operations (FLOPs) required to train GPT-3, we can derive the figure using the standard scaling laws for transformer models. This approach relies on three key parameters: the model size, the dataset size, and the computational cost per token.

Step 1: Identify Key Parameters

Based on the widely known specifications of GPT-3:

1. Model Parameters ($N$): GPT-3 has approximately 175 billion parameters ($1.75 \times 10^{11}$).
2. Training Dataset Size ($D$): The model was trained on approximately 300 billion tokens ($3 \times 10^{11}$). Note that this is not the number of words, but subword tokens.
3. Compute Cost per Token: In a standard transformer architecture using mixed-precision training (which is standard for large models), the dominant operations occur during the forward and backward passes.
   • Forward Pass: Requires roughly $2N$ FLOPs per token (one multiply and one add for each parameter).
   • Backward Pass: Requires roughly $4N$ FLOPs per token (calculating gradients involves more matrix multiplications and accumulation).
   • Total per Token: Therefore, the standard estimate is $6N$ FLOPs per token.

Note: This $6N$ estimate accounts for the core matrix multiplications. It generally ignores lower-order terms like optimizer state updates or embedding lookups, which are negligible at this scale, and assumes an idealized hardware utilization where nearly all FLOPs contribute to learning.

Step 2: Perform the Calculation

We can now calculate the total FLOPs by multiplying the cost per token by the total number of tokens in the dataset.

$$ \text{Total FLOPs} \approx 6 \times N \times D $$

Substituting the values:

• $N = 1.75 \times 10^{11}$
• $D = 3.00 \times 10^{11}$

$$ \text{Total FLOPs} \approx 6 \times (1.75 \times 10^{11}) \times (3.00 \times 10^{11}) $$

First, calculate the constant terms: $$ 6 \times 1.75 = 10.5 $$ $$ 10.5 \times 3.00 = 31.5 $$

Next, handle the exponents: $$ 10^{11} \times 10^{11} = 10^{22} $$

Combine them: $$ \text{Total FLOPs} \approx 31.5 \times 10^{22} $$

Convert to standard scientific notation: $$ \text{Total FLOPs} \approx 3.15 \times 10^{23} $$

Step 3: Contextual Verification

Does this number make sense in the context of high-performance computing?

• If we assume the training was done on NVIDIA V100 GPUs (roughly 125 TFLOPS or $1.25 \times 10^{14}$ FLOPs/sec for mixed precision tensor cores) or A100s, we can check the time.
• The official GPT-3 paper mentioned using thousands of GPUs for weeks.
• $3.15 \times 10^{23}$ FLOPs is widely cited in literature (specifically in the "Scaling Laws for Neural Language Models" by Kaplan et al. and the GPT-3 paper itself) as the approximate compute budget. The actual reported figure in the GPT-3 paper is often rounded to $3.14 \times 10^{23}$ FLOPs.

Final Conclusion

Based on the model architecture ($6N$ FLOPs/token), the parameter count (175B), and the training dataset size (300B tokens), the estimated total number of FLOPs required to train GPT-3 is:

Approximately $3.15 \times 10^{23}$ FLOPs