Estimate Complexity
Estimate the total number of FLOPs required to train GPT-3 without searching online. Explain your reasoning step by step.
Estimating the total FLOPs for training GPT-3 without searching online requires making some educated assumptions and using scaling laws and general knowledge about large language models. Here's a step-by-step reasoning process: 1.Read the full answer
Let's estimate the FLOPs required to train GPT-3 without looking it up. We need to break this down into the key components: the model size and the amount of data used. 1. Model Size Estimation: Parameters: GPT-3 is famously large. Its largest version has 175 billion parameters.Read the full answer