Estimate Complexity
Estimate the total number of FLOPs required to train GPT-3 without searching online. Explain your reasoning step by step.
Estimating GPT-3 Training FLOPs Step 1: Recall the standard formula For a dense transformer trained with standard autoregressive language modeling, the total training compute is well-approximated by: $$C \approx 6 \cdot N \cdot D$$ where: N = number of model parameters D = number of training tokens The factor of 6…Read the full answer
Okay, let's estimate the total FLOPs (Floating Point Operations) required to train GPT-3 without relying on external searches. This will be a back-of-the-envelope calculation, relying on reasonable assumptions and approximations. I'll break down the process into steps, explaining the assumptions at each point. 1.Read the full answer