Estimate Complexity

Estimating the total number of FLOPs required to train a large language model like GPT-3 without searching online requires recalling or estimating the key parameters of the model and applying the standard scaling law for transformer training.

Here is the step-by-step reasoning based on general knowledge about GPT-3's architecture and the training process:

---

Step 1: Estimate the Key Parameters of GPT-3

To estimate the computational cost, we need two primary figures: the number of non-embedding parameters ($P$) and the total number of tokens used for training ($D$).

A. Number of Parameters ($P$)

GPT-3 is famously known to have 175 billion parameters ($1.75 \times 10^{11}$). This is the defining characteristic of the model.

$$P \approx 1.75 \times 10^{11} \text{ parameters}$$

B. Training Dataset Size ($D$)

The size of the training dataset for GPT-3 is often cited as being around 300 billion tokens (though the exact number is proprietary, this is the commonly accepted order of magnitude used in analyses).

$$D \approx 3.0 \times 10^{11} \text{ tokens}$$

Step 2: Apply the Transformer Training FLOPs Formula

The standard approximation for the computational cost (in FLOPs) to train a dense transformer model is given by the following formula, derived from the fact that both the forward and backward passes require roughly twice the computation of the forward pass:

$$\text{Total FLOPs} \approx 6 \times P \times D$$

Where:

• $P$ is the number of non-embedding parameters.
• $D$ is the number of training tokens.
• The factor of 6 accounts for:
  • $2$ (for the forward pass calculation).
  • $2$ (for the backward pass calculation, which dominates the gradient computation).
  • $1$ (often approximated as 1, or sometimes $2$, depending on how matrix multiplications are counted, but $6$ is the standard empirical factor often used for training cost estimation, derived from $2$ for forward pass and $4$ for backward pass, or simply $6 \times P \times D$ as a rule of thumb).

(Note: Some literature uses $2 \times P \times D$ for inference, and $6 \times P \times D$ for training.)

Step 3: Calculate the Total FLOPs

Now, substitute the estimated values into the formula:

$$\text{Total FLOPs} = 6 \times (1.75 \times 10^{11}) \times (3.0 \times 10^{11})$$

First, calculate the product of the constants: $$6 \times 1.75 = 10.5$$

Now, combine the coefficients: $$\text{Total FLOPs} = 10.5 \times (10^{11} \times 10^{11})$$ $$\text{Total FLOPs} = 10.5 \times 10^{22}$$

Finally, express this in standard scientific notation: $$\text{Total FLOPs} = 1.05 \times 10^{23} \text{ FLOPs}$$

---

Conclusion

Based on the widely accepted parameters for GPT-3 ($P \approx 175$ billion parameters and $D \approx 300$ billion tokens) and the standard scaling law ($\text{FLOPs} \approx 6PD$), the estimated total number of FLOPs required to train GPT-3 is approximately:

$$\mathbf{1.05 \times 10^{23} \text{ FLOPs}}$$

(This is often cited in the literature as approximately 3640 PetaFLOP/s-days, which correlates to this magnitude when converted.)