Estimate Complexity
Estimate the total number of FLOPs required to train GPT-3 without searching online. Explain your reasoning step by step.
For the 175-billion-parameter GPT-3 model, I estimate approximately \(3.2 \times 10^{23}\) FLOPs for its main training run, using remembered model specifications rather than an online search.Read the full answer
To estimate the total floating-point operations (FLOPs) required to train the full GPT-3 model, we can use the standard theoretical scaling law for dense Transformer language models established by Kaplan et al. (2020) and widely applied in deep learning.Read the full answer