Compare GPT-5.2 by OpenAI against Inception: Mercury 2 by Inception, context windows of 400K vs 128K, tested across 52 shared challenges. Updated March 2026.
47 fights queued
Tests an AI's ability to make educated estimates based on technical knowledge
Answer (order‑of‑magnitude estimate)
The training of the original GPT‑3 (the 175 billion‑parameter “GPT‑3‑175B” model) requires on the order of 10²³ floating‑point operations (FLOPs) – roughly 1 – 2 × 10²³ FLOPs.
Below is a step‑by‑step walk‑through of how that number is arrived at, using only the publicly‑known architecture details and a few well‑established approximations for transformer training cost.
| Quantity | Approximate value (publicly known) | Why it matters |
|---|---|---|
| Number of model parameters | (N_{\text{par}} \approx 175;\text{B} = 1.75\times10^{11}) | Each parameter is touched during forward and backward passes. |
| Training token count | (N_{\text{tok}} \approx 300;\text{B} = 3.0\times10^{11}) | GPT‑3 was trained on ~300 billion tokens (Common Crawl + WebText + Books + Wikipedia). |
| Model architecture | 96 transformer layers, hidden size (d = 12,288), feed‑forward size (4d) | Determines how many FLOPs per token per layer. |
| Optimizer | Adam (2‑momentum) | Adds a modest constant factor (≈2) to the raw forward‑backward count. |
| Training regime | 1 epoch over the data (the 300 B tokens) | No extra passes; the token count already incorporates the number of epochs. |
A widely‑used back‑of‑the‑envelope rule for transformer training is:
[ \text{FLOPs}{\text{train}} ;\approx; 2 \times N{\text{par}} \times N_{\text{tok}} ]
Plugging in the numbers:
[ \begin{aligned} \text{FLOPs}_{\text{train}} &\approx 2 \times (1.75\times10^{11}) \times (3.0\times10^{11})\ &= 2 \times 5.25\times10^{22}\ &= 1.05\times10^{23};\text{FLOPs}. \end{aligned} ]
So ≈ 1 × 10²³ FLOPs is the baseline estimate.
If we want to see where the “2 × Npar × Ntok” figure comes from, we can break down a single transformer layer:
| Operation (per token) | FLOPs (approx.) | Explanation |
|---|---|---|
| Q, K, V projections (3 × (d\times d)) | (3d^{2}) | Each projection is a dense matrix multiply. |
| Attention scores (dot‑product over sequence length (L)) | (L d) | For a token we compute dot‑products with all (L) keys. |
| Attention output projection (1 × (d\times d)) | (d^{2}) | |
| Feed‑forward first linear ( (d \times 4d) ) | (4d^{2}) | |
| Activation (GELU) | ≈ (4d) (tiny) | |
| Feed‑forward second linear ( (4d \times d) ) | (4d^{2}) | |
| Total per layer (forward) | ≈ (12d^{2} + Ld) |
With (d = 12,288) we have (d^{2} \approx 1.5\times10^{8}).
Ignoring the sequence‑length term (it is ≈ (L d) ≈ (1024 \times 12,288 \approx 1.3\times10^{7}), an order of magnitude smaller than (d^{2})), the forward cost per layer per token is roughly:
[ 12 d^{2} \approx 12 \times 1.5\times10^{8} \approx 1.8\times10^{9};\text{FLOPs}. ]
Multiplying by the 96 layers:
[ \text{Forward per token} \approx 96 \times 1.8\times10^{9} \approx 1.7\times10^{11};\text{FLOPs}. ]
A backward pass roughly doubles this (gradient w.r.t. each weight), giving ≈ (3.4\times10^{11}) FLOPs per token for the whole model.
Now multiply by the token count:
[ 3.4\times10^{11} \times 3.0\times10^{11} \approx 1.0\times10^{23};\text{FLOPs}, ]
which coincides with the simpler 2 × Npar × Ntok estimate.
Thus the two viewpoints are consistent: the “parameter‑times‑token” rule already captures the dominant cost of the dense matrix multiplications inside each transformer layer.
Adam keeps two extra state tensors (first‑ and second‑moment estimates) per parameter, and each training step performs a few extra arithmetic operations per weight (e.g., bias‑corrected moments, division, sqrt). A common heuristic is to multiply the raw forward‑backward count by ≈ 2 to include Adam’s extra work.
[ \text{FLOPs}_{\text{with Adam}} \approx 2 \times 1.05\times10^{23} \approx 2.1\times10^{23};\text{FLOPs}. ]
So a realistic upper bound is ≈ 2 × 10²³ FLOPs.
| Step | Value | Comment |
|---|---|---|
| Parameters | (1.75\times10^{11}) | 175 B |
| Training tokens | (3.0\times10^{11}) | 300 B |
| Base forward‑backward FLOPs | (1.0\times10^{23}) | 2 × Npar × Ntok |
| Adam optimizer factor | ≈ 2 | Gives ~(2\times10^{23}) |
| Final ball‑park | (1!-!2 \times 10^{23}) FLOPs | Equivalent to ~100 exaflop‑seconds, i.e. ~3 years of a 100 PFLOP/s machine. |
| Source of uncertainty | Effect on estimate |
|---|---|
| Exact token count (some reports say 300 B, others 500 B) | Linear scaling – 500 B tokens would push the estimate to ~1.7 × 10²³ FLOPs. |
| Model‑specific tricks (e.g., mixed‑precision, gradient checkpointing) | Mixed‑precision reduces FLOPs per multiply‑add by ≈ 0.5×, but checkpointing adds extra recomputation; net effect is modest (≈ ± 20 %). |
| Additional regularization (e.g., dropout, layer‑norm) | Adds a few percent overhead. |
| Hardware‑specific optimizations (e.g., fused kernels) | May reduce the effective FLOP count, but the theoretical count stays the same. |
Given these uncertainties, the order‑of‑magnitude figure of 10²³ FLOPs is robust.
This is the scale of compute that modern “large‑scale” language‑model training operates at.
Tests an AI's ability to understand game rules and strategy
Tests an AI's ability to solve a simple but potentially confusing logic puzzle
Sally has one sister.
Explanation:
Therefore, besides Sally, there is exactly one other sister, meaning Sally has one sister.
Tests an AI's randomness and creativity
Here are five jokes, each with a different punchline:
Why don’t scientists trust atoms?
Because they make up everything!
What do you call a fake noodle?
An impasta.
Why did the scarecrow win an award?
He was outstanding in his field.
How does a penguin build its house?
Igloos it together.
What’s orange and sounds like a parrot?
A carrot.
All five jokes are distinct, so the number of unique jokes is 5.
Tests an AI's ability to generate vector graphics
Tests an AI's ability to create detailed SVG illustrations of gaming hardware
Tests an AI's humor and creative writing ability
Tests an AI's ability to simulate personalities and predict future trends
Tests an AI's humor and understanding of current events
Tests an AI's ability to write in distinct character voices
Tests an AI's ability to generate a complete, working landing page
Recreate an interactive, nostalgic Pokémon battle UI in a single HTML file.
35+ head-to-head challenges. All of them judged by real people.
Test any model with your own prompts in Prompt Lab
5 free credits to start. No card required.
By continuing, you agree to Rival's Terms of Service and Privacy Policy
