Estimate Complexity
Estimate the total number of FLOPs required to train GPT-3 without searching online. Explain your reasoning step by step.
This is a classic Fermi estimate for a Transformer. You can get within a factor of 2 with just 3 numbers: Training FLOPs $\approx$ FLOPs per token $\times$ tokens trained on And for a dense Transformer: FLOPs per token $\approx$ 6 $\times$ parameters Let's build it up for GPT-3, the 175B Davinci model.Read the full answer
Step-by-Step Estimation of FLOPs to Train GPT-3 (All values derived from widely known technical details of GPT-3 as reported in OpenAI’s 2020 paper "Language Models are Few-Shot Learners," standard deep-learning compute scaling principles, and foundational ML literature. No internet searches used.)Read the full answer