Estimate Complexity

I'll work through this estimation step by step, using what I know about GPT-3's architecture and training.

Known/Estimated Parameters

Model size:

• GPT-3 (175B version) has ~175 billion parameters

Training data:

• Trained on ~300 billion tokens (this was mentioned in the paper)

Architecture:

• Transformer decoder-only model

FLOPs Calculation

For transformer models, there's an approximate rule for FLOPs per token:

FLOPs per token ≈ 6P (for forward pass) + 6P (for backward pass) = 12P total

Where P = number of parameters

This factor of 6 comes from:

• 2 FLOPs per multiply-accumulate operation
• Each parameter is involved in approximately 3 operations per forward pass (attention, FFN layers, etc.)

Calculation:

• Parameters (P): 175 × 10⁹
• Tokens (T): 300 × 10⁹
• FLOPs per token: 12P = 12 × 175 × 10⁹ = 2.1 × 10¹²

Total training FLOPs:

• Total = FLOPs per token × number of tokens
• Total = 2.1 × 10¹² × 300 × 10⁹
• Total = 6.3 × 10²³ FLOPs

Final Estimate

~3-6 × 10²³ FLOPs (or ~300-600 zettaFLOPs)

This is roughly equivalent to running a GPU capable of 100 TFLOPS continuously for about 75,000 years, or more practically, running thousands of GPUs for several weeks.