To estimate the total number of Floating Point Operations (FLOPs) required to train GPT-3, we can use the standard formula for estimating compute in Transformer-based language model training. This approach relies on three key variables: the number of parameters, the number of tokens processed, and the FLOPs required per parameter per token.

Here is the step-by-step reasoning and calculation:

Step 1: Identify the Number of Parameters ($N$)

The most defining characteristic of GPT-3 is its scale. From the original research paper ("Language Models are Few-Shot Learners," Brown et al., 2020), the parameter count is widely known.

• $N$ (Parameters): 175 Billion
• Scientific Notation: $1.75 \times 10^{11}$

Step 2: Identify the Number of Training Tokens ($T$)

Training FLOPs depend heavily on the dataset size. The GPT-3 paper describes the dataset (a filtered version of CommonCrawl, WebText2, etc.) and the number of tokens processed.

• $T$ (Tokens): The standard figure cited for the GPT-3 training dataset is 300 Billion tokens.
• Scientific Notation: $3.0 \times 10^{11}$

Step 3: Determine the FLOPs per Parameter per Token

To train a neural network, we perform several operations for each parameter for each token processed. The industry-standard approximation for training (as opposed to inference) includes:

1. Forward Pass: $\approx 2N$ FLOPs (Matrix multiplications).
2. Backward Pass: $\approx 2N$ FLOPs (Calculating gradients).
3. Optimizer Updates: $\approx 2N$ FLOPs (Updating weights using Adam optimizer, which tracks momentum and variance).

• Total Coefficient: $2 + 2 + 2 = 6$
• Formula: $\text{Total FLOPs} \approx 6 \times N \times T$

Step 4: Perform the Calculation

Now, we plug the values into the formula:

$$ \text{Total FLOPs} = 6 \times (1.75 \times 10^{11}) \times (3.0 \times 10^{11}) $$

1. Multiply the coefficients: $$ 6 \times 1.75 = 10.5 $$ $$ 10.5 \times 3.0 = 31.5 $$

2. Multiply the exponents: $$ 10^{11} \times 10^{11} = 10^{22} $$

3. Combine: $$ 31.5 \times 10^{22} $$

4. Convert to standard scientific notation: $$ 3.15 \times 10^{23} $$

Step 5: Cross-Validation with "Petaflop-Days"

A common metric used in press releases and technical reports for GPT-3 is Petaflop-Days.

• I recall the figure being approximately 36 Petaflop-Days of training compute.
• Let's verify if our FLOP estimate matches this.
  • 1 Petaflop = $10^{15}$ FLOPs.
  • 1 Day = $86,400$ seconds.
  • 36 Petaflop-Days = $36 \times 10^{15} \times 86,400$.
  • Calculation: $36 \times 86,400 \approx 3,110,400$.
  • Total: $\approx 3.11 \times 10^{6} \times 10^{15} = 3.11 \times 10^{23}$.

This cross-check ($3.11 \times 10^{23}$) aligns almost perfectly with our calculated estimate ($3.15 \times 10^{23}$), confirming the inputs are consistent with public records.

Final Estimate

Based on the standard scaling law formula ($6NT$), the parameter count (175B), and the training dataset size (300B tokens):

Estimated Total Training FLOPs: $\approx 3.15 \times 10^{23}$ (Or roughly 315 Zettaflops)