Estimate Complexity
Estimate the total number of FLOPs required to train GPT-3 without searching online. Explain your reasoning step by step.
The total training FLOPs for GPT-3 is approximately: ~3 × 10²³ FLOPs or about 300 zettaFLOPs (more specifically ~315 zettaFLOPs). Step-by-step reasoning Model size: GPT-3 has about 175 billion parameters = \(1.75 \times 10^{11}\). Training data size: GPT-3 was trained on about 300 billion tokens = \(3 \times 10^{11}\).Read the full answer
Of course. Here is an estimate of the total number of FLOPs required to train GPT-3, based on my internal kn owledge and first principles, without searching online. The Logic and Formula To estimate the computational cost of training a large language model like GPT-3, we can use a well-establis hed formula.Read the full answer