Estimate Complexity

Certainly! Let's estimate the total number of floating-point operations (FLOPs) required to train GPT-3 by breaking down the problem step-by-step. We won't rely on online data but will reason from first principles and known architectural characteristics.

---

Step 1: Understand GPT-3's basic specifications

GPT-3, the largest model version, has approximately:

• Number of parameters (weights): 175 billion (1.75 × 10¹¹)
• Number of layers: about 96 transformer layers
• Model size: 175B parameters is widely known

---

Step 2: Estimate FLOPs per forward pass per token

A transformer forward pass consists mainly of:

• Matrix multiplications in self-attention
• Matrix multiplications in feedforward network (FFN)

Two main components dominate FLOPs:

1. Self-attention: roughly proportional to ( d_{model}^2 ) per token (where ( d_{model} ) is hidden size)
2. Feedforward layer: 2 linear layers per transformer layer, roughly (4 d_{model}^2) multiplications per token (due to expanding dimension by factor 4 in FFN)

---

Step 3: Extract hidden size and sequence length

From papers and common configurations:

• Hidden size ( d_{model} approx 12,288 ) (for 175B model)
• Sequence length used during training ( L approx 2048 ) tokens (max context length)

---

Step 4: Calculate approximate FLOPs per token per forward pass

Using formulas from transformer FLOPs estimation:

• For each transformer layer, per token, forward pass FLOPs roughly:

[ ext{FLOPs}_ ext{layer/token} approx 2 imes d_{model}^2 ]

(for attention and FFN combined, includes multiplications and additions)

Because the FFN (feedforward network) is the heaviest part, it dominates, so let's assume:

• ( sim 8 imes d_{model}^2 ) FLOPs per token per layer (this is a common estimate for one forward pass)

Substituting:

[ 8 imes (12,288)^2 approx 8 imes 1.5 imes 10^8 = 1.2 imes 10^9 ext{ FLOPs per token per layer} ]

For 96 layers:

[ 1.2 imes 10^9 imes 96 = 1.15 imes 10^{11} ext{ FLOPs per token (forward pass)} ]

---

Step 5: Account for backward pass

Backpropagation generally costs about 2× to 3× the forward pass FLOPs because computing gradients involves forward pass and backward pass computations.

Use factor 3 to be safe:

[ ext{FLOPs} = 3 imes 1.15 imes 10^{11} = 3.45 imes 10^{11} ext{ FLOPs per token (forward + backward)} ]

---

Step 6: Calculate total number of tokens used in training

From publicly known info, GPT-3 was trained on about 300 billion tokens (around 3 × 10¹¹ tokens) of text.

---

Step 7: Calculate total FLOPs for full training

Multiply FLOPs per token by total tokens:

[ 3.45 imes 10^{11} imes 3 imes 10^{11} = 1.035 imes 10^{23} ext{ FLOPs} ]

---

Final estimation:

[ oxed{ ext{Total FLOPs to train GPT-3} approx 10^{23} quad ext{FLOPs} } ]

---

Summary:

• Per token forward + backward FLOPs: (sim 3.5 imes 10^{11})
• Number of training tokens: (sim 3 imes 10^{11})
• Total training FLOPs: (sim 10^{23})

---

Additional notes:

• This estimate matches the order of magnitude widely discussed in ML circles.
• Variations depend on exact training steps, optimizer, precision (FP16/32), etc.
• Might be a bit lower or higher depending on exact implementation.

---

So, the training of GPT-3 requires on the order of ( 10^{23} ) floating point operations.