Estimating FLOPs to Train GPT-3

Key Parameters

Let me recall the key specs of GPT-3:

• Parameters: 175 billion (1.75 × 10¹¹)
• Training tokens: ~300 billion (3 × 10¹¹)
• Architecture: Decoder-only Transformer

The Standard Approximation Rule

There's a well-known approximation for Transformer training FLOPs:

$$C ≈ 6 \times N \times D$$

where:

• N = number of parameters
• D = number of training tokens
• 6 comes from the factor of 2 for each of three operations:
  • Forward pass: ~2 FLOPs per parameter per token (one multiply + one add per parameter for matrix operations)
  • Backward pass (activation gradients): ~2 FLOPs per parameter per token
  • Backward pass (weight gradients): ~2 FLOPs per parameter per token

Why 2 FLOPs per parameter per token for the forward pass?

A Transformer is dominated by matrix multiplications. For a matrix multiplication of shape (m × k) times (k × n), the FLOPs are ~2mkn (k multiplies and k-1 adds per output element ≈ 2k ops × mn outputs). When you sum across all the weight matrices in the model, the total forward-pass FLOPs work out to approximately 2 × N per token, where N is the total parameter count.

Calculation

$$C ≈ 6 \times N \times D$$

$$C ≈ 6 \times (1.75 \times 10^{11}) \times (3 \times 10^{11})$$

$$C ≈ 6 \times 5.25 \times 10^{22}$$

$$C ≈ 3.15 \times 10^{23} \text{ FLOPs}$$

Cross-check with reported values

The OpenAI paper actually reported 3.14 × 10²³ FLOPs, so this estimate is essentially exact — which makes sense because the 6ND formula is precisely the approximation the community (and OpenAI themselves) use.

Final Answer

$$\boxed{C \approx 3.15 \times 10^{23} \text{ FLOPs}}$$

This is about 315 ZettaFLOPs — roughly equivalent to running thousands of V100 GPUs for weeks-to-months, which aligns with what we know about the training infrastructure used.