Estimate Complexity

To estimate the total FLOPs (floating-point operations) required to train GPT-3, we can use a standard approximation formula derived from the architecture of Transformer models.

Here is the step-by-step breakdown of the estimation:

Step 1: Identify the Key Variables

To calculate training compute, we need two primary numbers: the number of parameters in the model ($P$) and the number of tokens the model was trained on ($D$).

1. Model Size ($P$): GPT-3 is famously known to have 175 billion parameters. $$P = 175 \times 10^9$$
2. Dataset Size ($D$): According to the original GPT-3 paper ("Language Models are Few-Shot Learners"), the model was trained on a weighted mix of datasets (Common Crawl, WebText2, Books1, Books2, Wikipedia). The total number of tokens processed during training was approximately 300 billion tokens. $$D = 300 \times 10^9$$

Step 2: Apply the Compute Approximation Formula

For dense Transformer models, a widely accepted heuristic (often cited in papers by Kaplan et al. and DeepMind regarding scaling laws) is that training requires approximately 6 FLOPs per parameter per training token.

• Forward Pass: Requires $\approx 2P$ FLOPs per token (matrix multiplication involves 1 multiply and 1 add, and we do this for the weights).
• Backward Pass (Gradient Calculation): Requires $\approx 4P$ FLOPs per token (roughly twice the cost of the forward pass).
• Total: $2P + 4P = 6P$ FLOPs per token.

Therefore, the formula for Total Training Compute ($C$) is: $$C \approx 6 \times P \times D$$

Step 3: Perform the Calculation

Now, substitute the values into the formula:

$$C \approx 6 \times (175 \times 10^9) \times (300 \times 10^9)$$

First, group the integers and the powers of 10: $$C \approx (6 \times 175 \times 300) \times (10^9 \times 10^9)$$

Calculate the integers:

1. $6 \times 175 = 1,050$
2. $1,050 \times 300 = 315,000$

Combine with the exponent ($10^{18}$): $$C \approx 315,000 \times 10^{18} \text{ FLOPs}$$

Step 4: Convert to Scientific Notation

To make the number readable, we normalize it:

$$315,000 = 3.15 \times 10^5$$

Combine exponents: $$3.15 \times 10^5 \times 10^{18} = 3.15 \times 10^{23}$$

Final Estimate

The estimated total compute required to train GPT-3 is approximately:

$$3.14 \times 10^{23} \text{ FLOPs}$$

(Note: In the AI literature, this is often converted to "Petaflop-days." One Petaflop-day is approximately $8.64 \times 10^{19}$ FLOPs. Dividing $3.14 \times 10^{23}$ by that number yields roughly 3,640 Petaflop-days, which aligns closely with OpenAI's published figures.)