DeepSeek V3 (March 2024)
Estimating the total number of FLOPs (floating-point operations) required to train GPT-3 involves several steps and assumptions. Here's a step-by-step breakdown:
1. Understand GPT-3's Architecture
GPT-3 is a transformer-based model with the following key parameters (for the largest version, GPT-3 175B):
- Parameters (N): 175 billion (1.75 × 10¹¹)
- Layers (L): 96
- Hidden dimension (d): 12,288
- Context length (sequence length, T): 2,048 tokens
2. FLOPs per Forward Pass
For a transformer model, the FLOPs for one forward pass per token can be approximated as: [ \text{FLOPs}_{\text{forward}} \approx 2 \cdot N \cdot T ] This accounts for the matrix multiplications (2N FLOPs per token due to multiply-add operations) across all layers for a sequence of length ( T ).
For GPT-3: [ \text{FLOPs}_{\text{forward}} \approx 2 \cdot 1.75 \times 10^{11} \cdot 2048 \approx 7.17 \times 10^{14} \text{ FLOPs} ]
3. FLOPs per Backward Pass
Backpropagation roughly requires twice the FLOPs of a forward pass (gradient computation + weight updates). Thus: [ \text{FLOPs}{\text{backward}} \approx 2 \cdot \text{FLOPs}{\text{forward}} \approx 1.43 \times 10^{15} \text{ FLOPs} ]
Total FLOPs per token (forward + backward): [ \text{FLOPs}{\text{token}} \approx 3 \cdot \text{FLOPs}{\text{forward}} \approx 2.15 \times 10^{15} \text{ FLOPs} ]
4. Total Tokens Processed
GPT-3 was trained on:
- Training tokens (D): ~300 billion tokens (3 × 10¹¹)
However, the actual number of tokens processed during training is: [ \text{Total tokens} = \text{Batch size} \cdot \text{Iterations} \cdot \text{Sequence length} ]
Assuming:
- Batch size (B): 3.2 million tokens (as used in similar models)
- Iterations (I): ( \frac{D}{B} = \frac{3 \times 10^{11}}{3.2 \times 10^6} \approx 94,000 )
Total tokens: [ \text{Total tokens} = B \cdot I \cdot T = 3.2 \times 10^6 \cdot 94,000 \cdot 2048 \approx 6 \times 10^{14} \text{ tokens} ]
(Note: This aligns with the ~300B tokens over 94,000 steps, as each step processes ( B \cdot T ) tokens.)
5. Total Training FLOPs
Multiply FLOPs per token by total tokens: [ \text{Total FLOPs} \approx \text{FLOPs}_{\text{token}} \cdot \text{Total tokens} ] [ \approx 2.15 \times 10^{15} \cdot 6 \times 10^{14} ] [ \approx 1.29 \times 10^{30} \text{ FLOPs} ]
This is a rough estimate. The actual value is often cited as ~3.14 × 10²³ FLOPs, but this discrepancy arises from:
- Simplified assumptions (e.g., ignoring attention FLOPs, which add ~2N²/T per layer).
- The exact batch size/sequence length may differ.
6. Refined Estimate
A more accurate formula for transformer FLOPs is: [ \text{FLOPs} \approx 6 \cdot N \cdot D ] (Accounts for forward/backward passes and attention.)
For GPT-3: [ \text{FLOPs} \approx 6 \cdot 1.75 \times 10^{11} \cdot 3 \times 10^{11} ] [ \approx 3.15 \times 10^{23} \text{ FLOPs} ]
This aligns with published estimates (~3.14 × 10²³ FLOPs).
Final Answer:
The total training FLOPs for GPT-3 are approximately 3.1 × 10²³ FLOPs. This is derived from: [ \text{FLOPs} \approx 6 \cdot \text{Parameters} \cdot \text{Tokens} ] [ = 6 \cdot 175 \text{B} \cdot 300 \text{B} ] [ = 3.15 \times 10^{23} \text{ FLOPs} ]