Estimate Complexity

To estimate the total FLOPs (floating-point operations) required to train GPT-3, we can use a common approximation for transformer models:

Total FLOPs ≈ 6 × N_params × N_tokens

Where:

• N_params is the number of model parameters.
• N_tokens is the number of training tokens.
• The factor of 6 accounts for forward and backward passes, along with other computations involved in training.

For GPT-3:

• N_params ≈ 175 billion = 1.75 × 10¹¹
• N_tokens ≈ 300 billion = 3.0 × 10¹¹

Plugging in:

6 × (1.75 × 10¹¹) × (3.0 × 10¹¹)
= 6 × 5.25 × 10²²
= 3.15 × 10²³ FLOPs

So the total estimated compute to train GPT-3 is ~3.15 × 10²³ FLOPs.

This is a rough but widely used approximation for large-scale transformer training.