Estimating GPT-3 Training FLOPs

I'll work through this from first principles using publicly known GPT-3 specs.

Known Parameters

• Model size: 175 billion parameters
• Training tokens: ~300 billion tokens (estimated from OpenAI's hints about compute)
• Training approach: Standard transformer with dense attention

Step 1: FLOPs per Token per Parameter

For a forward pass through a transformer:

• Each token attends to all previous tokens (roughly)
• Each parameter is involved in roughly 2 operations per token (multiply-accumulate = 2 FLOPs)

Rough estimate: ~2 FLOPs per parameter per token for forward pass

Step 2: Account for Backward Pass

Standard backpropagation requires:

• Forward pass: 1x
• Backward pass: ~2x (gradients for weights and activations)

Total: ~6 FLOPs per parameter per token (accounting for forward + backward)

Some literature suggests 3-6x multiplier; I'll use 6 as a reasonable middle-ground estimate.

Step 3: Calculate Total

Total FLOPs = Parameters × Tokens × FLOPs-per-token-per-param
            = 175 × 10^9 × 300 × 10^9 × 6
            = 175 × 300 × 6 × 10^18
            = 315,000 × 10^18
            ≈ 3.15 × 10^23 FLOPs

Final Estimate

~3 × 10^23 FLOPs (order of magnitude: 300 exaFLOPs-seconds)

This aligns with OpenAI's reported ~3,640 petaflop/s-days, which converts to roughly 3 × 10^23 FLOPs.

Reasoning Check

• This represents ~10,000 GPU-years of A100 compute (rough validation)
• Cost would be $10-15M at 2020 cloud prices
• Publicly consistent with OpenAI's disclosed estimates

Confidence: Moderate — the 2-6x multiplier for backprop is the biggest uncertainty, but order of magnitude should be solid.