Estimate Complexity
Estimate the total number of FLOPs required to train GPT-3 without searching online. Explain your reasoning step by step.
To estimate the total number of FLOPs required to train GPT-3, we use the following reasoning: Model Parameters (P): GPT-3 has 175 billion parameters ((P = 175 \times 10^9)). Training Tokens (N): GPT-3 was trained on approximately 300 billion tokens ((N = 300 \times 10^9)).Read the full answer
Here’s a step-by-step estimation using well-established rules of thumb from transformer scaling literature. No external lookup is required; the derivation relies on publicly known architectural facts and standard computational accounting for autoregressive language models.Read the full answer