Estimate Complexity
Estimate the total number of FLOPs required to train GPT-3 without searching online. Explain your reasoning step by step.
Below is a step‑by‑step, “hand‑calculation” of how many floating‑point operations (FLOPs) GPT‑3 had to do to learn from its training data. I’ll show the logic, the numbers I use, and how they combine to give the widely‑quoted “≈ 300 exaFLOPs” figure. (All the math can be done in a spreadsheet or a simple calculator.)Read the full answer
To estimate the total number of FLOPs required to train GPT-3, we need to consider the model architecture, training process, and dataset size. GPT-3 has 175 billion parameters, a context length of 2048 tokens, 96 layers, a hidden dimension of 12,288, and a feed-forward dimension of 49,152 (4 × hidden dimension).Read the full answer