Estimating the total number of floating-point operations (FLOPs) required to train GPT-3 involves reasoning through several key parameters: model size (number of parameters), the number of training tokens, and the relationship between these quantities and FLOPs.
Let’s break this down step by step using known architectural principles of transformer models like GPT-3 and standard training practices.
Step 1: Understand the relationship between parameters, data, and FLOPs
For transformer-based language models, a widely accepted approximation is that:
Total FLOPs ≈ 6 × (number of parameters) × (number of training tokens)
This factor of 6 comes from the observation that each parameter update during training involves a certain number of floating-point operations across forward pass, backward pass, and parameter update. More specifically:
- Forward pass: ~2 × N × D FLOPs (where N is parameters, D is data)
- Backward pass: ~4 × N × D FLOPs
- So total ≈ 6 × N × D
This approximation is commonly used in the literature (e.g., in "Scaling Laws for Neural Language Models" by Kaplan et al.) and has become a standard heuristic.
Step 2: Estimate the number of parameters in GPT-3
GPT-3 has multiple versions, but the largest version has 175 billion parameters, or 1.75 × 10¹¹.
We’ll use this value.
N = 1.75 × 10¹¹
Step 3: Estimate the number of training tokens
GPT-3 was trained on a dataset that includes a large portion of the internet, such as books, websites, and other text. While the exact number isn’t recalled perfectly, we can estimate based on typical corpus sizes.
- The Common Crawl dataset is on the order of trillions of tokens.
- GPT-3 was trained on about 300 billion tokens, or 3 × 10¹¹.
This number is consistent with public reports and scaling laws papers.
D = 3 × 10¹¹ tokens
Step 4: Apply the FLOPs formula
Now plug into the formula:
Total FLOPs ≈ 6 × N × D
= 6 × (1.75 × 10¹¹) × (3 × 10¹¹)
First, multiply the constants:
6 × 1.75 × 3 = 6 × 5.25 = 31.5
Now the powers of 10:
10¹¹ × 10¹¹ = 10²²
So:
Total FLOPs ≈ 3.15 × 10²³
Final Estimate
Approximately 3 × 10²³ FLOPs
Summary of Reasoning
- Used the standard FLOPs estimation formula: 6 × parameters × tokens
- GPT-3 has 175B parameters (~1.75 × 10¹¹)
- Trained on approximately 300B tokens (~3 × 10¹¹)
- Computed: 6 × 1.75e11 × 3e11 = 3.15e23
- Final estimate: ~3 × 10²³ FLOPs
This is in line with widely cited figures (e.g., OpenAI's blog and subsequent analyses), even without looking up the exact number.
