Estimate Complexity

To estimate the total FLOPs (Floating Point Operations) required to train GPT-3 without external research, I'll rely on fundamental principles of transformer-based language model training, established scaling laws, and publicly known (but not "searched") specifications of GPT-3 from common knowledge in the ML community. I'll explain my reasoning step by step, highlighting assumptions and simplifications. The goal is a reasonable order-of-magnitude estimate, not an exact figure.

---

Step 1: Key Known Specifications of GPT-3 (Common Knowledge)

From widespread reports (e.g., OpenAI's 2020 paper, widely discussed in AI circles):

• Parameters (N): 175 billion (175 × 10⁹).
• Training tokens (D): 300 billion (300 × 10⁹).
• Sequence length (T): 2,048 tokens (standard for GPT-3).

These values are foundational and frequently cited in ML literature, so I treat them as given.

---

Step 2: Core Principle – FLOPs per Token in Transformer Training

For transformer models, the dominant FLOPs come from matrix multiplications in attention and feed-forward layers. The standard approximation for training FLOPs is derived as follows:

1. Forward pass FLOPs per token:

   • A forward pass requires ~2 FLOPs per parameter per token (1 for multiplication, 1 for addition in multiply-accumulate operations).
   • Thus: Forward FLOPs ≈ 2 × N per token.

2. Backward pass FLOPs per token:

   • Backpropagation requires ~3× the FLOPs of the forward pass (due to gradient computations for weights, activations, and chain rule overhead).
   • Thus: Backward FLOPs ≈ 6 × N per token (3 × 2N).

3. Total training FLOPs per token:

   • Forward + Backward = 2N + 6N = 8N per token.
   • However, empirical scaling laws (e.g., Kaplan et al. 2020) simplify this to 6N per token for large models. This accounts for:
     • Optimizer overhead (e.g., Adam adds ~20–30% FLOPs, but is often folded into the constant).
     • Minor operations (layer norms, softmax) being negligible at scale.
     • Real-world efficiency (e.g., memory bottlenecks reduce theoretical peak FLOPs).
   • I adopt the widely accepted approximation: Total FLOPs per token ≈ 6 × N.

Why not include sequence length (T)?

• The 6N approximation already incorporates T implicitly. Per-token FLOPs scale with N (parameters), not T (sequence length), because:
  • For large N (e.g., 175B), the cost of linear layers (O(N)) dominates over attention (O(T²)). With T=2,048, T² ≈ 4M, while N=175B ≫ T², so T-dependent costs are negligible in the leading term.
  • Scaling laws (e.g., Chinchilla, Kaplan) confirm total FLOPs ≈ 6 × N × D, where D is total tokens. T is absorbed into the constant.

---

Step 3: Total FLOPs Calculation

Using the formula:
Total FLOPs = 6 × N × D

Substitute known values:

• N = 175 × 10⁹
• D = 300 × 10⁹

Calculation:

1. N × D = (175 × 10⁹) × (300 × 10⁹) = 5.25 × 10²²
2. Total FLOPs = 6 × 5.25 × 10²² = 3.15 × 10²³

---

Step 4: Addressing Key Uncertainties and Refinements

This is a simplification. I consider adjustments to ensure realism:

1. Is 6N per token accurate?

   • The factor of 6 comes from:
     • Forward: 2N (standard for dense layers).
     • Backward: 4N (some studies use 2× forward for backward, but 3× is conservative; 2N × 3 = 6N total).
   • Optimizers (e.g., Adam) add ~20–30% FLOPs for momentum/variance updates. Including this:
     Adjusted factor = 6 × 1.25 ≈ 7.5.
     → Total FLOPs ≈ 7.5 × 5.25 × 10²² ≈ 3.94 × 10²³.
   • However, hardware inefficiencies (e.g., memory bottlenecks, non-peak utilization) reduce effective FLOPs. The 6N rule is a "theoretical maximum" often used in estimates, so I stick with 6N for consistency with literature.

2. Sequence length (T) impact:

   • Strictly, per-token FLOPs = 2N × (1 + αT), where α is small (e.g., α ≈ 0.01 for attention). For T=2,048:
     2N × (1 + 0.01×2048) ≈ 2N × 21.5 ≈ 43N per token (forward).
     This would imply ~129N per token (backward ≈ 3× forward), leading to 129 × 5.25 × 10²² ≈ 6.77 × 10²⁴ FLOPs – 10× higher than before.
   • Why this is incorrect for large N:
     The αT term assumes attention dominates, but for N ≫ T² (175B ≫ 4M), linear layers dominate. Thus, α is tiny (e.g., α ≈ 10⁻⁵), making T negligible. Scaling laws confirm N × D is the primary driver.

3. Data efficiency:

   • Not all tokens contribute equally (e.g., padding, short sequences). GPT-3 used dynamic batching, but D=300B is the effective token count after preprocessing, so no adjustment needed.

4. Model sparsity:

   • GPT-3 is dense (no MoE), so no FLOPs reduction.

---

Step 5: Final Estimate and Confidence

• Base estimate: 6 × N × D = 6 × 175e9 × 300e9 = 3.15 × 10²³ FLOPs.
• Realistic range: Accounting for optimizer overhead (+25%) and hardware inefficiency (-10%), the true value likely falls between 2.8 × 10²³ and 3.9 × 10²³ FLOPs.
• Order of magnitude: ~10²³ FLOPs (3 × 10²³ is the anchor).

Why this is plausible:

• This aligns with the "Chinchilla scaling law" (DeepMind 2022), which cites GPT-3 training at ~3.14 × 10²³ FLOPs (note: 3.14 × 10²³ ≈ π × 10²³, a common meme-adjusted figure).
• For context:
  • Training GPT-3 on 1,000 V100 GPUs (15,000 TFLOPS each) would take ~36 days at 100% utilization:
    (3.15e23 FLOPs) / (1,000 × 15e12 FLOPS) ≈ 2.1e7 seconds ≈ 243 days.
    Actual training took ~34 days, implying ~7× hardware efficiency (realistic for distributed training), confirming the estimate's magnitude.

---

Conclusion

Total FLOPs to train GPT-3 ≈ 3.15 × 10²³ (315 sextillion).

This estimate is derived from:

1. Universal transformer FLOPs scaling: 6 × parameters × training tokens.
2. GPT-3 specs: 175B parameters, 300B tokens.
3. Conservative adjustments for real-world factors (optimizer, hardware), keeping the result within 10% of established literature values.

While minor details (e.g., exact optimizer FLOPs) could shift this by ±20%, the 10²³ order of magnitude is robust and consistent with foundational ML scaling principles. For reference, this is equivalent to ~300 million GPU-years of V100 compute, explaining why training GPT-3 required a massive cluster.