Estimate Complexity
Estimate the total number of FLOPs required to train GPT-3 without searching online. Explain your reasoning step by step.
To estimate the training compute for GPT‑3, I’ll break the problem into three pieces: the size of the model, the size of the training dataset, and the amount of work the training algorithm does per parameter per token. Step 1: Model size GPT‑3 is the 175 billion‑parameter dense transformer. N ≈ 175 × 10⁹ parameters.Read the full answer
To estimate the total number of Floating Point Operations (FLOPs) required to train GPT-3 without searching online, we can rely on well-established rules of thumb in deep learning and the publicly known architectural specifications of the model. Here is the step-by-step reasoning and calculation.Read the full answer