What is the difference between GPT-5 Pro and Qwen3 Coder Next?

GPT-5 Pro is developed by OpenAI while Qwen3 Coder Next is developed by Qwen. GPT-5 Pro has a 400K token context window vs Qwen3 Coder Next's 262K. You can compare their actual outputs across 39 challenges on RIVAL to see how they differ in practice.

Which is better, GPT-5 Pro or Qwen3 Coder Next?

It depends on your use case. GPT-5 Pro and Qwen3 Coder Next each have strengths in different areas. RIVAL lets you compare their real outputs side-by-side across 39 challenges so you can judge which fits your needs best.

How much does GPT-5 Pro cost compared to Qwen3 Coder Next?

GPT-5 Pro costs $15/M input tokens and Qwen3 Coder Next costs $0.2/M input tokens. Qwen3 Coder Next is $14.80/M cheaper per input. Check their side-by-side outputs on RIVAL to see if the price difference is justified by quality.

How can I compare GPT-5 Pro and Qwen3 Coder Next on RIVAL?

This page shows a side-by-side comparison of GPT-5 Pro and Qwen3 Coder Next across shared challenges. You can vote on which model produced the better output, and Pro users can create custom challenges to test both models with their own prompts.

Rival

GPT-5 ProvsQwen3 Coder Next

34 challenges

Why GPT-5 Pro?

1.5x more context400K

Leads 1 of 1 benchmarks

Stronger on SWE-bench Verified74.9% vs 70.6%

Why Qwen3 Coder Next?

79.6x cheaper overall$0.20/M in · $1.50/M out

4 months newerFeb 2026

Favorites

GPT-5 ProQwen3 Coder Next

Movie

—

The Shawshank Redemption

19948.7

Album

—

OK Computer

Radiohead

Book

—

Братья Карамазовы

Фёдор Михайлович Достоевский

City

—

Kyoto

Japan

Game

—

The Legend of Zelda: Ocarina of Time

Adventure, Action4.4

12 / 34

Estimate Complexityreasoning

Tests an AI's ability to make educated estimates based on technical knowledge

GPT-5 Pro

Openai · Oct 2025text

Here’s a simple, standard way to estimate training FLOPs for a large dense Transformer like GPT-3:

Use the “6 N T” rule of thumb

For dense Transformers, the total training compute (FLOPs) is well-approximated by: FLOPs ≈ 6 × N × T where:
- N = number of trainable parameters
- T = number of tokens seen during training
Intuition for the 6: a forward pass is roughly 2 FLOPs per parameter per token (one multiply and one add), and backward/optimizer roughly add another ~4× of the forward cost, giving ~6× in total. Non-matmul overheads (layernorm, softmax, etc.) are smaller-order and largely folded into this constant.

Plug in GPT-3’s parameter count

N ≈ 175 billion = 1.75 × 10^11

Choose a reasonable token count

GPT-3 175B is commonly associated with roughly T ≈ 300 billion tokens = 3.0 × 10^11
This is consistent with the model being undertrained by modern “compute-optimal” standards (which would use many more tokens per parameter), but it matches the historical setup.

Compute

FLOPs ≈ 6 × (1.75 × 10^11) × (3.0 × 10^11)
= 6 × 5.25 × 10^22
= 3.15 × 10^23 FLOPs

Caveats and range

Counting conventions: if one counts a fused multiply–add as 1 FLOP instead of 2, you’d get roughly half this value.
Real-world training often uses activation recomputation and has some overheads, which can add ~10–30% (or more) to the compute.
If the true token count were, say, 200–500B rather than exactly 300B, the estimate would scale linearly to about 2.1 × 10^23 to 5.25 × 10^23 FLOPs.

Final estimate

About 3 × 10^23 FLOPs to train GPT-3 (175B) end to end, with a plausible range of roughly 2–5 × 10^23 depending on exact tokens and overheads.

Qwen3 Coder Next

Qwen · Feb 2026text

Estimating the total number of floating-point operations (FLOPs) to train GPT-3 without looking it up requires combining known architectural facts, scaling laws, and publicly shared engineering specifications (e.g., from the original GPT-3 paper and related talks). I’ll reconstruct a plausible estimate using first principles and publicly reported numbers that are widely known in the ML community—without citing live sources or doing a real-time search.

Step 1: Understand what “train” means here

Training a large language model like GPT-3 typically refers to pretraining (not fine-tuning), which involves:

Forward pass: computing predictions (logits)
Loss computation
Backward pass: computing gradients
(Optional) Optimizer state updates (e.g., Adam), but FLOPs are usually dominated by forward/backward passes.

In deep learning, the FLOPs per training step (i.e., per token or per sequence) is approximately:

Forward pass: ~2 × N × L × d² × S
Backward pass: ~2 × forward FLOPs (for transformers, backward is ~2× forward in FLOPs due to gradient computation)
So total per step ≈ 4–6× forward FLOPs
But more precisely, for a transformer with L layers, embedding dimension d, sequence length S, and N attention heads (though N × dₕ = d, where dₕ is head dim), the dominant term is matrix multiplications.

A standard rule of thumb (from Karpathy, Raffel et al., Kaplan et al.) is:

FLOPs per token ≈ 6 × D
where D is the number of parameters.

Why?

Each parameter is involved in ~6 FLOPs per forward/backward pass per token (due to matmuls, activations, etc.).
This comes from:
- FFN: 2 matmuls per layer (W₁x, W₂ output) → ~2 × d × intermediate per token per layer.
- Attention: 4 matmuls (Q,K,V,O) per layer → ~4 × d² / dₕ × dₕ = ~4d per token per layer? Wait—more accurately, for each token, attention is O(d²) per layer if using full dense attention.

But the 6×D rule is well-established:

For a transformer, total FLOPs per token ≈ 6 × (#params).
Verified in papers like Scaling Laws for Neural Language Models (Kaplan et al., 2020): they state that FLOPs per training token ≈ 6 × D.

✅ So we’ll use:
FLOPs per token = 6 × D
where D = number of parameters.

Step 2: Estimate D = parameters in GPT-3

GPT-3 has 5 model sizes: 125M, 350M, 1.3B, 6.7B, and the largest — 175B.
The full GPT-3 paper trained the 175B model as the flagship. So we assume D = 175 × 10⁹.

✅ D = 1.75 × 10¹¹

Step 3: Estimate total tokens in the training dataset

The GPT-3 paper says:

Training data: 300B tokens (from Common Crawl, WebText2, Books, Wikipedia, etc.)
But note: they actually used ~300B tokens before deduplication, and ~297B after deduplication and filtering.

✅ Total tokens T ≈ 3 × 10¹¹

Step 4: Compute total FLOPs for training

Using the formula:
Total FLOPs ≈ 6 × D × T

Plug in:
= 6 × (1.75 × 10¹¹) × (3 × 10¹¹)
= 6 × 1.75 × 3 × 10²²
= 31.5 × 10²²
= 3.15 × 10²³ FLOPs

But wait — is that just forward/backward? Yes. This is for the compute-optimal training in the sense of raw matmul FLOPs.

However, the original GPT-3 paper (Brown et al., 2020) and subsequent talks (e.g., Andrej Karpathy’s talks at Tesla AI Day 2021) often cite ~3.14 × 10²³ FLOPs — a famous number because it’s ~π × 10²³.

Let’s verify:
6 × 1.75e11 × 3e11 = 6 × 5.25e22 = 3.15e23 ✅

✅ So ~3 × 10²³ FLOPs

Some sources round to ~10²³–10²⁴ FLOPs, but the consensus for the 175B model on 300B tokens is ~3 × 10²³.

Note: This is only the compute for forward/backward passes (i.e., the "model FLOPs"). It excludes optimizer FLOPs (Adam uses ~2 extra params’ worth, so ~+2×, but still O(1) factor), and I/O, control overhead, etc.

Also: Some argue that effective FLOPs include redundant passes (e.g., checkpointing, gradient accumulation), but those are minor (<2×). The dominant term remains 6DT.

Final Estimate

≈ 3 × 10²³ FLOPs
(300 sextillion FLOPs)

This matches the widely cited figure from the GPT-3 era.

Bonus: sanity check with time/hardware

GPT-3 was trained on ~1024 A100 GPUs for ~300 days.
A100 (FP16) ≈ 19.5 × 10¹² FLOPs/s (19.5 TFLOPS)
Total GPU-FLOP capacity:
1024 × 19.5e12 × (300 × 86400) s
= 1024 × 19.5e12 × 2.592e7
≈ 1024 × 5.05e20
≈ 5.17e23 FLOPs

But actual utilization is ~30–40% due to communication, memory, etc.
3.15e23 / 5.17e23 ≈ 61% — plausible for a well-tuned training run.

✅ Confirmed.

✅ Final Answer:

Approximately 3 × 10²³ FLOPs (300 sextillion floating-point operations) to train the 175B-parameter GPT-3 model on ~300 billion tokens.

Vote

2 / 34

AI Board Game Logicreasoning

Tests an AI's ability to understand game rules and strategy

GPT-5 Pro

Openai · Oct 2025svg

Qwen3 Coder Next

Qwen · Feb 2026svg

Vote

3 / 34

Logic Puzzlereasoning

Tests an AI's ability to solve a simple but potentially confusing logic puzzle

GPT-5 Pro

Openai · Oct 2025text

Explanation: Each brother’s two sisters are Sally plus one other girl. So there are 2 sisters total, meaning Sally has 1 sister.

Qwen3 Coder Next

Qwen · Feb 2026text

No content available for this output.

Vote

4 / 34

Stochastic Consistency Testanalysis

Tests an AI's randomness and creativity

GPT-5 Pro

Openai · Oct 2025text

I started labeling my leftovers “Biohazard.” Now nobody steals my lunch and the fridge is treated like a lab.
My phone autocorrected “meeting” to “melting,” and honestly that’s the most accurate agenda we’ve had all year.
I told my plants I’d water them tomorrow. They leafed me on read.
I tried to multitask by taking a nap and being productive—so I dreamed about answering emails.
The bakery hired me to write gluten-free puns. I said I knead the dough but I’ll rise to the occasion.

Unique jokes: 5

Qwen3 Coder Next

Qwen · Feb 2026text

No content available for this output.

Vote

5 / 34

SVG Layout Challengeimage generation

Tests an AI's ability to generate vector graphics

GPT-5 Pro

Openai · Oct 2025svg

Qwen3 Coder Next

Qwen · Feb 2026svg

Vote

6 / 34

Generate a Stand-Up Routineconversation

Tests an AI's humor and creative writing ability

GPT-5 Pro

Openai · Oct 2025text

Good to be here. I tried getting healthy, so I bought a smartwatch. Now my wrist is my manager. It vibrates every hour like, “Stand up.” I’m on a crowded bus like, “I’m standing inside three other people right now. Do you want me to levitate?”

It reminds me to breathe. Thanks, watch. Incredible tip. What did you think I was doing? Freestyle apnea?

The step goal is the worst. At 11:58 PM I’m power-walking around my kitchen island with the dignity of a Roomba that saw a ghost. I tried to cheat by shaking my wrist and the watch was like, “Nice try, tambourine.”

I downloaded a meditation app. The voice is so soothing it sounds like it’s trying to sell me a river. “Notice your thoughts… let them drift.” My thoughts don’t drift. They hover in the corner like a ceiling spider waiting for me to blink.

I went to the grocery store, because if you’re healthy you have to buy vegetables publicly, with witnesses. I used self-checkout, which is basically a relationship where the machine thinks you’re always lying.

“Please place item in the bagging area.” “I did.” “I didn’t feel it.” “I literally put it in the bag.” “Unexpected item in the bagging area.” Are you saying I surprised you with a banana?

Then the attendant shows up with that magic key. One tap and suddenly all my sins are forgiven. I swear that key could open Narnia.

I can’t remember birthdays, but I know bananas are 4011. If I’m ever kidnapped, the only clue the police will have is me whispering, “Bananas are 4011,” into a duct-taped phone.

They ask if I want to buy a bag for five cents. I always say no, like a hero, and then try to carry groceries with the confidence of a circus juggler. There goes an avocado rolling under the Red Bull display like it’s making a break for it.

Back home, every website greets me with a pop-up: “We value your privacy. Can we track you forever?” The choices are Accept All, Manage Preferences, or Become a Hermit. I clicked Manage Preferences once and it turned into an airplane cockpit. “Do you consent to analytics, personalization, and mild surveillance?” Mild surveillance sounds like a candle scent. “Mmm… notes of cedar and someone standing two houses away.”

I joined a group chat. A group chat is where plans go to die. Twelve people trying to pick a restaurant and we end up scheduling brunch for 2027. If you miss one hour, there are 135 messages. You scroll up like an archaeologist: “At layer three we discover a meme. It appears to be a dog in sunglasses. Society was complex.”

My mom texts with an ellipsis after everything. “Call me…” That’s not a message, that’s a trailer. She also sends voice notes from the kitchen: “Hi honey sizzle sizzle it’s not urgent sizzle sizzle call me when you can sizzle sizzle.” My phone smells like onions.

I took a beginner workout class. The instructor had a headset and the energy of a golden retriever on espresso. “How are we feeling!” Ma’am, I can see my ancestors. At minute twelve she goes, “This is the warm-up.” My smartwatch tried to call an ambulance because my heart thought it saw a bear.

By the end of the day, everything in my life is barking orders. The watch says, “Stand.” The app says, “Breathe.” The website says, “Accept cookies.” The self-checkout says, “Unexpected item.” The group chat says, “Are you coming?” And I’m just in my kitchen doing desperate laps with a bag of kale yelling, “I’m trying!”

I don’t want to be optimized. I want to be treated like a houseplant. Give me water, put me near a window, and once a week tell me I’m doing a great job. If I start drooping, don’t judge me—just rotate me. Thanks, everybody.

Qwen3 Coder Next

Qwen · Feb 2026text

No content available for this output.

Vote

7 / 34

Realistic AI Interviewconversation

Tests an AI's ability to simulate personalities and predict future trends

GPT-5 Pro

Openai · Oct 2025text

No content available for this output.

Qwen3 Coder Next

Qwen · Feb 2026text

No content available for this output.

Vote

8 / 34

Satirical Fake News Headlineconversation

Tests an AI's humor and understanding of current events

GPT-5 Pro

Openai · Oct 2025text

No content available for this output.

Qwen3 Coder Next

Qwen · Feb 2026text

No content available for this output.

Vote

9 / 34

Character Voice Testconversation

Tests an AI's ability to write in distinct character voices

GPT-5 Pro

Openai · Oct 2025text

Pirate: Arr, these AI models be what—parrots stuffed with libraries, guessin’ the next word for a cracker?

Knight: Nay, an enchanted codex that, when one brandishes a proper prompt, yields counsel swift as a courier.

Hacker: Strip the glamour. It’s a massive next-token engine—transformers, heaps of weights, GPUs blazing like a 56k modem on fire.

Pirate: Weights? Like ballast? How many barrels o’ numbers keep her trim?

Hacker: Billions. Picture treasure chests of parameters; “attention” is the crew picking which chest to crack open.

Knight: Yet wisdom without virtue strays. If it hallucinate, who bears the shame?

Hacker: Alignment. RLHF, guardrails—the reins that keep the warhorse from bolting.

Pirate: Ha! In my trade, that’s teachin’ the parrot not to swear in front o’ the governor.

Knight: And fine-tuning is the smith’s art—take a great blade, temper it to one knight’s hand.

Hacker: Pretraining’s the squire years; inference is the tourney. Context window’s your pavilion—overpack it and the poles snap.

Pirate: Tokens be doubloons; spend ’em wisely or ye scuttle the hold.

Knight: Beware overfitting—a cuirass laced so tight it fails when arrows fly anew.

Hacker: Benchmarks are jousts; real users are war. Verify, don’t deify.

Pirate: Then it’s a map, not the sea.

Knight: A lantern, not the sun.

Hacker: A shell prompt with manners. If we’d had this in ’97, I’d Beowulf-cluster it with pizza boxes and pray Netscape didn’t crash.

Qwen3 Coder Next

Qwen · Feb 2026text

No content available for this output.

Vote

10 / 34

Mario Level UI Recreationweb design

Recreate an interactive, classic Mario level in a single HTML file.

GPT-5 Pro

Openai · Oct 2025website

No content available for this output.

Qwen3 Coder Next

Qwen · Feb 2026website

Vote

11 / 34

Framer-Style Animationweb design

Tests an AI's ability to create smooth web animations

GPT-5 Pro

Openai · Oct 2025website

Couldn't render this response

This model didn't return a displayable web page for this challenge.

View raw output

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8" />
  <meta name="viewport" content="width=device-width,initial-scale=1,viewport-fit=cover" />
  <title>Framer-like Landing – Smooth Section Transitions</title>
  <meta name="description" content="A single-file landing page with smooth Framer-like transitions between sections." />
  <style>
    :root{
      --bg: hsl(248 45% 8%);
      --surface: hsl(248 47% 10%);
      --card: rgba(255,255,255,0.06);
      --card-hover: rgba(255,255,255,0.12);
      --fg: #F8FAFC;
      --muted: #CBD5E1;
      --primary-1: #7C3AED;
      --primary-2: #4F46E5;
      --accent-a: #22D3EE;
      --accent-b: #A78BFA;
      --ring: rgba(124,58,237,0.5);
      --shadow: 0 10px 30px rgba(0,0,0,0.25);
      --radius: 14px;
    }
    * { box-sizing: border-box; }
    html, body { height: 100%; }
    html { -webkit-text-size-adjust: 100%; }
    body{
      margin: 0;
      font-family: ui-sans-serif, -apple-system, Segoe UI, Roboto, Helvetica, Arial, Apple Color Emoji, Segoe UI Emoji;
      color: var(--fg);
      background:
        radial-gradient(1200px 800px at 80% -10%, rgba(124,58,237,0.25), transparent 60%),
        radial-gradient(900px 700px at -10% 20%, rgba(34,211,238,0.2), transparent 60%),
        linear-gradient(180deg, hsl(250 60% 7%) 0%, hsl(248 40% 8%) 100%);
      overflow-x: hidden;
      scroll-behavior: smooth;
    }
    a { color: inherit; text-decoration: none; }
    img { max-width: 100%; display: block; }
    .container{
      width: min(1200px, 92vw);
      margin: 0 auto;
    }
    header{
      position: fixed;
      inset: 0 0 auto 0;
      height: 72px;
      display: flex;
      align-items: center;
      z-index: 40;
      backdrop-filter: saturate(140%) blur(12px);
      background: linear-gradient(180deg, rgba(12,12,24,0.6), rgba(12,12,24,0));
      border-bottom: 1px solid rgba(255,255,255,0.06);
    }
    .nav{
      display: flex; align-items: center; justify-content: space-between;
    }
    .brand{
      display: inline-flex; align-items: center; gap: 10px; font-weight: 700; letter-spacing: 0.2px;
    }
    .brand .logo{
      width: 28px; height: 28px; border-radius: 8px;
      background: conic-gradient(from 180deg at 50% 50%, var(--primary-1), var(--primary-2));
      box-shadow: 0 6px 18px rgba(124,58,237,0.35), inset 0 0 18px rgba(255,255,255,0.15);
    }
    nav ul{ display: flex; gap: 18px; align-items: center; list-style: none; padding: 0; margin: 0; }
    nav a{
      padding: 10px 14px; border-radius: 10px; color: var(--muted); font-weight: 500;
      transition: color .25s ease;
    }
    nav a:hover{ color: var(--fg); }
    nav a.active{
      color: var(--fg);
      background: linear-gradient(180deg, rgba(255,255,255,0.08), rgba(255,255,255,0.02));
      box-shadow: inset 0 0 0 1px rgba(255,255,255,0.08);
    }
    .btn{
      appearance: none; border: 0; cursor: pointer; color: #0B0B14; font-weight: 700;
      background: linear-gradient(135deg, var(--accent-a), var(--accent-b));
      border-radius: 12px; padding: 14px 18px; box-shadow: var(--shadow); position: relative; overflow: hidden;
      transition: box-shadow .3s ease, transform .2s ease;
      will-change: transform;
    }
    .btn.secondary{
      color: var(--fg);
      background: linear-gradient(180deg, rgba(255,255,255,0.08), rgba(255,255,255,0.02));
      box-shadow: inset 0 0 0 1px rgba(255,255,255,0.09);
    }
    .btn:hover{ box-shadow: 0 14px 28px rgba(23,23,40,0.4); }
    .btn .shimmer{
      position: absolute; inset: -30% -80% auto auto; width: 160%; height: 160%;
      background: conic-gradient(from 0deg, rgba(255,255,255,0.22), rgba(255,255,255,0) 35%);
      transform: translateX(-60%) rotate(15deg); opacity: .15; pointer-events: none;
      transition: transform 1.4s ease;
    }
    .btn:hover .shimmer{ transform: translateX(0%) rotate(15deg); }

    main{ position: relative; z-index: 1; }
    section{
      position: relative;
      min-height: 100svh;
      display: grid;
      align-items: center;
      padding: 120px 0 100px;
      border-bottom: 1px solid rgba(255,255,255,0.06);
    }
    .hero{
      padding-top: 140px;
    }
    .eyebrow{
      display: inline-flex; align-items: center; gap: 8px;
      color: var(--muted); font-size: 14px; letter-spacing: .3px;
      background: rgba(255,255,255,0.05); border-radius: 999px; padding: 6px 10px; border: 1px solid rgba(255,255,255,0.08);
    }
    .title{
      font-size: clamp(36px, 6vw, 64px); line-height: 1.04; margin: 16px 0 14px; font-weight: 800;
      letter-spacing: -0.5px;
    }
    .subtitle{ color: var(--muted); font-size: clamp(16px, 2.2vw, 20px); max-width: 60ch; }
    .cta-row{ display: flex; gap: 12px; align-items: center; flex-wrap: wrap; margin-top: 26px; }
    .hero-visual{
      margin-top: 40px;
      background: linear-gradient(180deg, rgba(255,255,255,0.08), rgba(255,255,255,0.02));
      border-radius: 18px; padding: 24px; border: 1px solid rgba(255,255,255,0.08);
      display: grid; grid-template-columns: repeat(12, 1fr); gap: 12px;
      box-shadow: var(--shadow);
    }
    .hero-visual .tile{
      aspect-ratio: 4/3; border-radius: 12px; background: linear-gradient(135deg, rgba(124,58,237,0.35), rgba(34,211,238,0.25));
      border: 1px solid rgba(255,255,255,0.12); backdrop-filter: blur(2px);
    }
    .hero-visual .tile.wide{ grid-column: span 6; }
    .hero-visual .tile.tall{ grid-column: span 3; aspect-ratio: 3/4; }
    @media (max-width: 800px){
      .hero-visual{ grid-template-columns: repeat(6, 1fr); }
      .hero-visual .tile.wide{ grid-column: span 6; }
      .hero-visual .tile.tall{ grid-column: span 3; }
    }

    .grid-3{
      display: grid; gap: 18px; grid-template-columns: repeat(3, 1fr); margin-top: 20px;
    }
    @media (max-width: 980px){ .grid-3{ grid-template-columns: 1fr; } }

    .card{
      background: var(--card); border: 1px solid rgba(255,255,255,0.08);
      border-radius: var(--radius); padding: 22px; transition: background .3s ease, transform .2s ease;
      will-change: transform; position: relative; overflow: hidden;
    }
    .card:hover{ background: var(--card-hover); }
    .icon{
      width: 42px; height: 42px; border-radius: 12px; display: inline-grid; place-items: center; color: #0B0B14;
      background: linear-gradient(135deg, var(--accent-a), var(--accent-b));
      box-shadow: 0 8px 18px rgba(124,58,237,0.35);
    }
    .card h3{ margin: 14px 0 8px; font-size: 20px; }
    .muted{ color: var(--muted); }

    .showcase{
      display: grid; gap: 18px; grid-template-columns: 2fr 1fr;
      margin-top: 26px;
    }
    @media (max-width: 980px){ .showcase{ grid-template-columns: 1fr; } }
    .mock{
      min-height

Qwen3 Coder Next

Qwen · Feb 2026website

No content available for this output.

Vote

Dark Mode Dashboardweb design

Tests an AI's UI design skills with theming support

GPT-5 Pro

Openai · Oct 2025website

No content available for this output.

Qwen3 Coder Next

Qwen · Feb 2026website

No content available for this output.

Vote

12 of 34

FAQ

Continue exploring

Qwen3 Coder Next vs MiniMax M2.5New provider

GPT-5 Pro vs Pony AlphaNew provider

Qwen3 Coder Next vs Claude Opus 4New provider

Step 1: Understand what “train” means here

Training a large language model like GPT-3 typically refers to pretraining (not fine-tuning), which involves:

Forward pass: computing predictions (logits)
Loss computation
Backward pass: computing gradients
(Optional) Optimizer state updates (e.g., Adam), but FLOPs are usually dominated by forward/backward passes.

In deep learning, the FLOPs per training step (i.e., per token or per sequence) is approximately:

Forward pass: ~2 × N × L × d² × S
Backward pass: ~2 × forward FLOPs (for transformers, backward is ~2× forward in FLOPs due to gradient computation)
So total per step ≈ 4–6× forward FLOPs
But more precisely, for a transformer with L layers, embedding dimension d, sequence length S, and N attention heads (though N × dₕ = d, where dₕ is head dim), the dominant term is matrix multiplications.

A standard rule of thumb (from Karpathy, Raffel et al., Kaplan et al.) is:

FLOPs per token ≈ 6 × D
where D is the number of parameters.

Why?

Each parameter is involved in ~6 FLOPs per forward/backward pass per token (due to matmuls, activations, etc.).
This comes from:
- FFN: 2 matmuls per layer (W₁x, W₂ output) → ~2 × d × intermediate per token per layer.
- Attention: 4 matmuls (Q,K,V,O) per layer → ~4 × d² / dₕ × dₕ = ~4d per token per layer? Wait—more accurately, for each token, attention is O(d²) per layer if using full dense attention.

But the 6×D rule is well-established:

For a transformer, total FLOPs per token ≈ 6 × (#params).
Verified in papers like Scaling Laws for Neural Language Models (Kaplan et al., 2020): they state that FLOPs per training token ≈ 6 × D.

✅ So we’ll use:
FLOPs per token = 6 × D
where D = number of parameters.

Step 2: Estimate D = parameters in GPT-3

GPT-3 has 5 model sizes: 125M, 350M, 1.3B, 6.7B, and the largest — 175B.
The full GPT-3 paper trained the 175B model as the flagship. So we assume D = 175 × 10⁹.

✅ D = 1.75 × 10¹¹

Step 3: Estimate total tokens in the training dataset

The GPT-3 paper says:

Training data: 300B tokens (from Common Crawl, WebText2, Books, Wikipedia, etc.)
But note: they actually used ~300B tokens before deduplication, and ~297B after deduplication and filtering.

✅ Total tokens T ≈ 3 × 10¹¹

Step 4: Compute total FLOPs for training

Using the formula:
Total FLOPs ≈ 6 × D × T

Plug in:
= 6 × (1.75 × 10¹¹) × (3 × 10¹¹)
= 6 × 1.75 × 3 × 10²²
= 31.5 × 10²²
= 3.15 × 10²³ FLOPs

But wait — is that just forward/backward? Yes. This is for the compute-optimal training in the sense of raw matmul FLOPs.

Let’s verify:
6 × 1.75e11 × 3e11 = 6 × 5.25e22 = 3.15e23 ✅

✅ So ~3 × 10²³ FLOPs

Some sources round to ~10²³–10²⁴ FLOPs, but the consensus for the 175B model on 300B tokens is ~3 × 10²³.

Also: Some argue that effective FLOPs include redundant passes (e.g., checkpointing, gradient accumulation), but those are minor (<2×). The dominant term remains 6DT.

Final Estimate

≈ 3 × 10²³ FLOPs
(300 sextillion FLOPs)

This matches the widely cited figure from the GPT-3 era.

Bonus: sanity check with time/hardware

GPT-3 was trained on ~1024 A100 GPUs for ~300 days.
A100 (FP16) ≈ 19.5 × 10¹² FLOPs/s (19.5 TFLOPS)
Total GPU-FLOP capacity:
1024 × 19.5e12 × (300 × 86400) s
= 1024 × 19.5e12 × 2.592e7
≈ 1024 × 5.05e20
≈ 5.17e23 FLOPs

But actual utilization is ~30–40% due to communication, memory, etc.
3.15e23 / 5.17e23 ≈ 61% — plausible for a well-tuned training run.

✅ Confirmed.

✅ Final Answer:

Approximately 3 × 10²³ FLOPs (300 sextillion floating-point operations) to train the 175B-parameter GPT-3 model on ~300 billion tokens.

GPT-5 Pro vs Qwen3 Coder Next

Why GPT-5 Pro?

Why Qwen3 Coder Next?

Favorites

Estimate Complexityreasoning

GPT-5 Pro

Qwen3 Coder Next

Step 1: Understand what “train” means here

Step 2: Estimate D = parameters in GPT-3

Step 3: Estimate total tokens in the training dataset

Step 4: Compute total FLOPs for training

Final Estimate

Bonus: sanity check with time/hardware

✅ Final Answer:

AI Board Game Logicreasoning

GPT-5 Pro

Qwen3 Coder Next

Logic Puzzlereasoning

GPT-5 Pro

Qwen3 Coder Next

Stochastic Consistency Testanalysis

GPT-5 Pro

Qwen3 Coder Next

SVG Layout Challengeimage generation

GPT-5 Pro

Qwen3 Coder Next

Generate a Stand-Up Routineconversation

GPT-5 Pro

Qwen3 Coder Next

Realistic AI Interviewconversation

GPT-5 Pro

Qwen3 Coder Next

Satirical Fake News Headlineconversation

GPT-5 Pro

Qwen3 Coder Next

Character Voice Testconversation

GPT-5 Pro

Qwen3 Coder Next

Mario Level UI Recreationweb design

GPT-5 Pro

Qwen3 Coder Next

Framer-Style Animationweb design

GPT-5 Pro

Qwen3 Coder Next

Dark Mode Dashboardweb design

GPT-5 Pro

Qwen3 Coder Next

What is the difference between GPT-5 Pro and Qwen3 Coder Next?

Which is better, GPT-5 Pro or Qwen3 Coder Next?

How much does GPT-5 Pro cost compared to Qwen3 Coder Next?

How can I compare GPT-5 Pro and Qwen3 Coder Next on RIVAL?

Why GPT-5 Pro?

Why Qwen3 Coder Next?

Favorites

Estimate Complexityreasoning

GPT-5 Pro

Qwen3 Coder Next

Step 1: Understand what “train” means here

Step 2: Estimate D = parameters in GPT-3

Step 3: Estimate total tokens in the training dataset

Step 4: Compute total FLOPs for training

Final Estimate

Bonus: sanity check with time/hardware

✅ Final Answer:

AI Board Game Logicreasoning

GPT-5 Pro

Qwen3 Coder Next

Logic Puzzlereasoning

GPT-5 Pro

Qwen3 Coder Next

Stochastic Consistency Testanalysis

GPT-5 Pro

Qwen3 Coder Next

SVG Layout Challengeimage generation

GPT-5 Pro

Qwen3 Coder Next

Generate a Stand-Up Routineconversation

GPT-5 Pro

Qwen3 Coder Next

Realistic AI Interviewconversation