Estimate Complexity

Step-by-Step Reasoning for Estimating GPT-3 Training FLOPs

1. Recall GPT-3 key specs from memory:

   • GPT-3 (davinci variant) has N = 175 billion parameters (1.75 × 10¹¹).
   • It was trained on approximately D = 300 billion tokens (3 × 10¹¹). This is the total number of byte-pair encoded (BPE) tokens processed during pre-training, as stated in the GPT-3 paper ("Language Models are Few-Shot Learners"). The dataset included filtered Common Crawl (~250B tokens), WebText2, Books1/Books2, and Wikipedia, totaling ~300B tokens for the largest model.

2. Understand the standard FLOPs formula for transformer training:

   • For autoregressive transformers like GPT, the total training compute C ≈ 6 × N × D floating-point operations (FLOPs).
   • Why 6? This is an established approximation from OpenAI's scaling laws papers (e.g., Kaplan et al., 2020):

     Pass	FLOPs per token	Rationale
     Forward	~2N	Two main matrix multiplications per layer (attention + FFN), summed over layers ≈ 2N ops/token.
     Backward	~4N	Gradient computation is roughly 2× forward pass FLOPs.
     Total	6N	Neglects minor overheads like activations/softmax (~10-20% but often ignored for estimates).

   • This assumes dense training (no sparsity), BF16/FP16 precision, and standard optimizer (AdamW). It's accurate within ~10-20% for LMs like GPT.

3. Plug in the numbers:

   C ≈ 6 × (1.75 × 10¹¹) × (3 × 10¹¹)
     = 6 × 1.75 × 3 × 10^(11+11)
     = 6 × 5.25 × 10²²
     = 31.5 × 10²²
     = 3.15 × 10²³ FLOPs
   

   • In scientific notation: ~3.1 × 10²³ FLOPs (rounded for estimation; exact calc is 3.15 × 10²³).

4. Validation and caveats:

   • This matches widely cited figures (e.g., ~3 × 10²³ FLOPs) from OpenAI reports and papers like Hoffmann et al. (Chinchilla, 2022), which retroactively analyzed GPT-3.
   • Uncertainties (±20%): Exact D might be 300-410B tokens (some sources say 410B BPE); activations/optimizer overhead; mixture-of-experts (none in GPT-3); hardware efficiency losses.
   • Not included: Preprocessing, fine-tuning, or inference FLOPs—just core pre-training.

Final estimate: ~3 × 10²³ FLOPs (or 300 sextillion FLOPs). This was feasible on ~10,000 V100/A100 GPUs over months, costing ~$4-12M in cloud compute.