Estimate Complexity

To estimate the computing power required to train GPT-3, we can use the standard approximation formula derived from scaling laws for Large Language Models (LLMs).

Here is the step-by-step breakdown:

1. The Compute Formula

The standard heuristic for estimating the floating-point operations (FLOPs) required to train a Transformer-based language model is:

$$C \approx 6ND$$

Where:

• $C$ is the total compute in FLOPs.
• $N$ is the number of model parameters.
• $D$ is the number of tokens in the training dataset.

Why 6? In a Transformer, for every token processed:

• The forward pass requires roughly $2N$ FLOPs (multiply-accumulate operations for matrix multiplications).
• The backward pass (calculating gradients) requires roughly $4N$ FLOPs (it is generally twice as expensive as the forward pass).
• $2N + 4N = 6N$ FLOPs per token.

2. Estimating the Variables

$N$ (Parameters): GPT-3 is famously known for having 175 billion parameters. $$N = 175 \times 10^9$$

$D$ (Training Tokens): According to the GPT-3 paper ("Language Models are Few-Shot Learners"), the model was trained on a weighted mix of datasets (Common Crawl, WebText2, Books1, Books2, Wikipedia). The total number of tokens processed during the training run is widely cited as 300 billion tokens. $$D = 300 \times 10^9$$ (Note: Modern models are often trained on trillions of tokens, but GPT-3 was trained on relatively few by today's "Chinchilla optimal" standards.)

3. The Calculation

Now, we plug the numbers into the formula:

$$C \approx 6 \times (175 \times 10^9) \times (300 \times 10^9)$$

Step A: Group the coefficients and powers of 10 $$C \approx (6 \times 175 \times 300) \times (10^9 \times 10^9)$$

Step B: Simplify the coefficients First, multiply 6 by 300: $$6 \times 300 = 1,800$$

Next, multiply 1,800 by 175: $$1,800 \times 175$$ $$(18 \times 100) \times 175$$ $$18 \times 17,500$$

Calculation: $$10 \times 17,500 = 175,000$$ $$8 \times 17,500 = 140,000$$ $$175,000 + 140,000 = 315,000$$

So the coefficient is 315,000.

Step C: Combine with powers of 10 $$C \approx 315,000 \times 10^{18}$$

Step D: Convert to scientific notation Move the decimal point 5 places to the left to normalize the coefficient: $$3.15 \times 10^5 \times 10^{18}$$ $$3.15 \times 10^{23}$$

Final Estimate

The total estimated compute to train GPT-3 is:

$3.15 \times 10^{23}$ FLOPs

---

Contextual Check

In the AI industry, this is often converted into Petaflop-days to make the number more intuitive.

• 1 Petaflop/s = $10^{15}$ FLOPs/s.
• Seconds in a day $\approx 86,400$.
• 1 Petaflop-day $\approx 8.64 \times 10^{19}$ FLOPs.

$$\frac{3.15 \times 10^{23}}{8.64 \times 10^{19}} \approx 3,645 \text{ Petaflop-days}$$

This aligns with the widely accepted figure that GPT-3 required several thousand Petaflop-days of compute.