Estimate Complexity
Estimate the total number of FLOPs required to train GPT-3 without searching online. Explain your reasoning step by step.
Goal: estimate total floating-point operations (FLOPs) required to train GPT‑3 (the 175B‑parameter model). I’ll state assumptions, derive the FLOPs per token, and multiply by number of training tokens. Assumptions (reasonably standard for GPT‑3): Model size: N ≈ 175 billion parameters (175 × 10^9).Read the full answer
3.15 \times 10^{23} FLOPs GPT-3 (the 175B-parameter version) has N ≈ 1.75 × 10^{11} parameters. It was trained on a filtered dataset of roughly D ≈ 3 × 10^{11} tokens (a commonly cited figure drawn from Common Crawl, WebText, books, and Wikipedia, with some up-sampling of higher-quality sources).Read the full answer