Estimate Complexity
Estimate the total number of FLOPs required to train GPT-3 without searching online. Explain your reasoning step by step.
I'll work through this estimation for you.# Estimating GPT-3's Training FLOPs Step 1: Use the standard transformer compute formula For training a dense transformer, the well-established rule of thumb is: $$C \approx 6 \times N \times D$$ where N is the number of model parameters and D is the number of training tokens.Read the full answer
Below is one way to arrive at a back-of-the-envelope estimate for the total floating-point operations (FLOPs) needed to train GPT-3 (the 175B-parameter model). Note that different sources use slightly different assumptions, so you may see figures that differ by factors of 2–3; however, the overall scale (10^23 FLOPs)…Read the full answer