Estimate Complexity
Estimate the total number of FLOPs required to train GPT-3 without searching online. Explain your reasoning step by step.
Below is a “back-of-the-envelope” that stays within a factor of ~2 of the number most practitioners quote (≈3×10²³ FLOP). We will do it twice—once with the simplest possible formula (parameter count × 6 × tokens) and once with a slightly more careful treatment of the actual tensor shapes inside the transformer.Read the full answer
Step-by-Step Estimation of FLOPs to Train GPT-3 (All values derived from widely known technical details of GPT-3 as reported in OpenAI’s 2020 paper "Language Models are Few-Shot Learners," standard deep-learning compute scaling principles, and foundational ML literature. No internet searches used.)Read the full answer