Estimate Complexity
Estimate the total number of FLOPs required to train GPT-3 without searching online. Explain your reasoning step by step.
Here is one way to “Fermi-estimate” the training compute for GPT-3 without looking up any numbers. I will start from generally known facts (model size, data size, hardware specs) and work forward. Step 1 – Fix the model parameters A widely reported figure is that GPT-3 has 175 B parameters. N ≈ 1.75 × 10¹¹.Read the full answer
To estimate the total number of FLOPs (Floating Point Operations) required to train GPT-3, we can rely on the model's configuration details which were published in the original paper ("Language Models are Few-Shot Learners", Brown et al., 2020) and the general scaling laws for Transformer models (Kaplan et al., 2020).Read the full answer