Certainly! Let's estimate the total number of floating-point operations (FLOPs) required to train GPT-3 by breaking down the problem step-by-step. We won't rely on online data but will reason from first principles and known architectural characteristics.
Step 1: Understand GPT-3's basic specifications
GPT-3, the largest model version, has approximately:
- Number of parameters (weights): 175 billion (1.75 × 10¹¹)
- Number of layers: about 96 transformer layers
- Model size: 175B parameters is widely known
Step 2: Estimate FLOPs per forward pass per token
A transformer forward pass consists mainly of:
- Matrix multiplications in self-attention
- Matrix multiplications in feedforward network (FFN)
Two main components dominate FLOPs:
- Self-attention: roughly proportional to ( d_{model}^2 ) per token (where ( d_{model} ) is hidden size)
- Feedforward layer: 2 linear layers per transformer layer, roughly (4 d_{model}^2) multiplications per token (due to expanding dimension by factor 4 in FFN)
Step 3: Extract hidden size and sequence length
From papers and common configurations:
- Hidden size ( d_{model} approx 12,288 ) (for 175B model)
- Sequence length used during training ( L approx 2048 ) tokens (max context length)
Step 4: Calculate approximate FLOPs per token per forward pass
Using formulas from transformer FLOPs estimation:
- For each transformer layer, per token, forward pass FLOPs roughly:
[
ext{FLOPs}_ ext{layer/token} approx 2 imes d_{model}^2
]
(for attention and FFN combined, includes multiplications and additions)
Because the FFN (feedforward network) is the heaviest part, it dominates, so let's assume:
- ( sim 8 imes d_{model}^2 ) FLOPs per token per layer (this is a common estimate for one forward pass)
Substituting:
[
8 imes (12,288)^2 approx 8 imes 1.5 imes 10^8 = 1.2 imes 10^9 ext{ FLOPs per token per layer}
]
For 96 layers:
[
1.2 imes 10^9 imes 96 = 1.15 imes 10^{11} ext{ FLOPs per token (forward pass)}
]
Step 5: Account for backward pass
Backpropagation generally costs about 2× to 3× the forward pass FLOPs because computing gradients involves forward pass and backward pass computations.
Use factor 3 to be safe:
[
ext{FLOPs} = 3 imes 1.15 imes 10^{11} = 3.45 imes 10^{11} ext{ FLOPs per token (forward + backward)}
]
Step 6: Calculate total number of tokens used in training
From publicly known info, GPT-3 was trained on about 300 billion tokens (around 3 × 10¹¹ tokens) of text.
Step 7: Calculate total FLOPs for full training
Multiply FLOPs per token by total tokens:
[
3.45 imes 10^{11} imes 3 imes 10^{11} = 1.035 imes 10^{23} ext{ FLOPs}
]
Final estimation:
[
oxed{
ext{Total FLOPs to train GPT-3} approx 10^{23} quad ext{FLOPs}
}
]
Summary:
- Per token forward + backward FLOPs: (sim 3.5 imes 10^{11})
- Number of training tokens: (sim 3 imes 10^{11})
- Total training FLOPs: (sim 10^{23})
Additional notes:
- This estimate matches the order of magnitude widely discussed in ML circles.
- Variations depend on exact training steps, optimizer, precision (FP16/32), etc.
- Might be a bit lower or higher depending on exact implementation.
So, the training of GPT-3 requires on the order of ( 10^{23} ) floating point operations.
