Estimate Complexity

Let's break down estimating the FLOPs for GPT-3 training. Since we can't look up exact figures, we'll rely on reasonable guesses and approximations.

1. Model Size: GPT-3 has 175 billion parameters. This is our anchor point.

2. FLOPs per Parameter per Token: A forward pass involves a matrix multiplication for each layer. Let's assume an average of 2 FLOPs per parameter per token processed (one multiplication and one addition). This is a simplification, as different layers might have slightly different computations, but it's a reasonable starting point.

3. Dataset Size: GPT-3 was trained on a massive dataset. Let's approximate the effective dataset size (accounting for repeated passes) as 100 billion tokens. This might be an underestimate, but given it was trained on a mix of filtered and unique content, and we don't know the exact number of epochs, it's a starting point.

4. Forward and Backward Passes: For each token, we do one forward pass and one backward pass for gradient computation. So, we have 2 passes.

5. Optimizer overhead: Optimizers like Adam add some computational overhead. Let's add a 20% overhead factor to account for this.

Now, let's put it all together:

• FLOPs per token = 175B parameters * 2 FLOPs/parameter * 2 passes = 700B FLOPs/token

• Total FLOPs = 700B FLOPs/token * 100B tokens = 7 x 10^22 FLOPs

• Optimizer overhead: 7 x 10^22 * 1.2 = 8.4 x 10^22 FLOPs

Therefore, a very rough estimate is around 8.4 x 10^22 FLOPs to train GPT-3.

It's crucial to remember that this is a back-of-the-envelope calculation. The actual number could be higher or lower due to factors such as:

• Variations in layer computations: Different layers might have different computational complexity.
• Optimization techniques: Specific optimizations during training could reduce the actual FLOPs.
• Data preprocessing and tokenization overhead: We haven't accounted for these factors.
• Dataset repetitions/epochs: The assumed effective dataset size is a guess.
• Architecture specifics: The actual architecture and training process might involve more complexities not captured here.

So, while our estimate provides a general order of magnitude, it shouldn't be taken as a precise figure.