Estimate Complexity

Estimating the total number of floating-point operations (FLOPs) required to train GPT-3 involves reasoning through several key parameters: model size (number of parameters), the number of training tokens, and the relationship between these quantities and FLOPs.

Let’s break this down step by step using known architectural principles of transformer models like GPT-3 and standard training practices.

---

Step 1: Understand the relationship between parameters, data, and FLOPs

For transformer-based language models, a widely accepted approximation is that:

Total FLOPs ≈ 6 × (number of parameters) × (number of training tokens)

This factor of 6 comes from the observation that each parameter update during training involves a certain number of floating-point operations across forward pass, backward pass, and parameter update. More specifically:

• Forward pass: ~2 × N × D FLOPs (where N is parameters, D is data)
• Backward pass: ~4 × N × D FLOPs
• So total ≈ 6 × N × D

This approximation is commonly used in the literature (e.g., in "Scaling Laws for Neural Language Models" by Kaplan et al.) and has become a standard heuristic.

---

Step 2: Estimate the number of parameters in GPT-3

GPT-3 has multiple versions, but the largest version has 175 billion parameters, or 1.75 × 10¹¹.

We’ll use this value.

N = 1.75 × 10¹¹

---

Step 3: Estimate the number of training tokens

GPT-3 was trained on a dataset that includes a large portion of the internet, such as books, websites, and other text. While the exact number isn’t recalled perfectly, we can estimate based on typical corpus sizes.

• The Common Crawl dataset is on the order of trillions of tokens.
• GPT-3 was trained on about 300 billion tokens, or 3 × 10¹¹.

This number is consistent with public reports and scaling laws papers.

D = 3 × 10¹¹ tokens

---

Step 4: Apply the FLOPs formula

Now plug into the formula:

Total FLOPs ≈ 6 × N × D
= 6 × (1.75 × 10¹¹) × (3 × 10¹¹)

First, multiply the constants:

6 × 1.75 × 3 = 6 × 5.25 = 31.5

Now the powers of 10:

10¹¹ × 10¹¹ = 10²²

So:

Total FLOPs ≈ 3.15 × 10²³

---

Final Estimate

Approximately 3 × 10²³ FLOPs

---

Summary of Reasoning

• Used the standard FLOPs estimation formula: 6 × parameters × tokens
• GPT-3 has 175B parameters (~1.75 × 10¹¹)
• Trained on approximately 300B tokens (~3 × 10¹¹)
• Computed: 6 × 1.75e11 × 3e11 = 3.15e23
• Final estimate: ~3 × 10²³ FLOPs

This is in line with widely cited figures (e.g., OpenAI's blog and subsequent analyses), even without looking up the exact number.