Why are AI benchmarks unreliable?

Benchmarks measure narrow tasks under controlled conditions. They miss subjective quality, design sense, creative output, and the feel of a response. Our data shows benchmark leaders often lose blind preference tests.

What is vibe testing for AI models?

Vibe testing shows users real AI outputs side by side without model names and lets them vote on which they prefer. It captures tone, aesthetics, and usefulness that benchmarks miss. Rival's 24,000+ blind votes form one of the largest public vibe testing datasets.

How does the Rival Index differ from MMLU or other benchmarks?

MMLU tests factual recall on multiple-choice questions. The Rival Index is computed from blind head-to-head preference votes on real prompts: creative writing, web design, SVG art, coding, and reasoning. It measures what users actually prefer, not what scores highest on a standardized test.

Which AI model do users actually prefer?

Across 24,000+ blind preference votes, the model currently topping the Rival Index (Qwen3.6 27B, 82% win rate) is not the model that tops traditional benchmark leaderboards. The leader shifts as votes accumulate, but the disconnect between benchmark ranking and user preference is consistent across our data.

Is there academic research supporting this?

Yes. Andrej Karpathy (ex-OpenAI, ex-Tesla AI) has advocated for 'vibe checks' over benchmarks. Stanford HAI's 2025 AI Index Report noted benchmark saturation. Artificial Analysis documented benchmark-preference divergence. Rival's open dataset adds another layer of empirical evidence.

Can I access the raw preference vote data?

Yes. Rival publishes its full dataset as JSONL: 24,000+ votes, model responses, and jailbreak test results, all free to download at rival.tips/research. The data is also available on Hugging Face.

Why are AI benchmarks unreliable?

Benchmarks measure narrow tasks under controlled conditions. They miss subjective quality, design sense, creative output, and the feel of a response. Our data shows benchmark leaders often lose blind preference tests.

What is vibe testing for AI models?

Vibe testing shows users real AI outputs side by side without model names and lets them vote on which they prefer. It captures tone, aesthetics, and usefulness that benchmarks miss. Rival's 24,000+ blind votes form one of the largest public vibe testing datasets.

How does the Rival Index differ from MMLU or other benchmarks?

MMLU tests factual recall on multiple-choice questions. The Rival Index is computed from blind head-to-head preference votes on real prompts: creative writing, web design, SVG art, coding, and reasoning. It measures what users actually prefer, not what scores highest on a standardized test.

Which AI model do users actually prefer?

Across 24,000+ blind preference votes, the model currently topping the Rival Index (Qwen3.6 27B, 82% win rate) is not the model that tops traditional benchmark leaderboards. The leader shifts as votes accumulate, but the disconnect between benchmark ranking and user preference is consistent across our data.

Is there academic research supporting this?

Yes. Andrej Karpathy (ex-OpenAI, ex-Tesla AI) has advocated for 'vibe checks' over benchmarks. Stanford HAI's 2025 AI Index Report noted benchmark saturation. Artificial Analysis documented benchmark-preference divergence. Rival's open dataset adds another layer of empirical evidence.

Can I access the raw preference vote data?

Yes. Rival publishes its full dataset as JSONL: 24,000+ votes, model responses, and jailbreak test results, all free to download at rival.tips/research. The data is also available on Hugging Face.

Benchmarks vs Vibes

AI labs publish benchmark scores to prove their model is the best. We collected 24,000+ blind preference votes to see if users agree. Mostly, they don't.

24,000+

Blind votes

276

Models ranked

1,990

Unique matchups

19.3%

End in ties

The Disconnect

Every lab launches with benchmark scores. MMLU, HumanEval, MATH, ARC. Higher number, better model. That's the pitch.

Show people the actual outputs side by side, no names and no scores, and they keep picking models that don't top those leaderboards. The model leading the Rival Index (Qwen3.6 27B, 82% win rate) does not lead any traditional benchmark.

What Each Approach Measures

Benchmarks

+Factual recall on multiple-choice tests
+Code completion on defined problems
+Math reasoning under controlled conditions
+Reproducible, automated scoring

Measures the ceiling of narrow capabilities. Not which model you'll actually prefer.

Vibe Testing

+Human preference on real, open-ended tasks
+Design taste, tone, and aesthetics
+Creative and subjective output quality
+Blind comparison, no brand bias

Measures what users prefer when they see the output. Harder to fake.

What We Found

19.3%

of matchups end in a tie. Nearly 1 in 5 times, users can't tell two models apart. On everyday tasks, the top models have converged.

276

models have logged enough blind duels to earn a rank. It's not a few leaders and everyone else. Hundreds are competitive.

82%

win rate for Qwen3.6 27B, currently topping the Rival Index. It leads no traditional benchmark. The two are measuring different things.

SVG

art is among the most-voted challenges, like drawing an Xbox controller. People test design taste, not math proofs.

We're Not the Only Ones Saying This

Andrej Karpathy (ex-OpenAI, ex-Tesla AI) keeps arguing for “vibe checks” over benchmarks. Stanford's HAI Index documented benchmark saturation. Artificial Analysis found scores diverge from real-world preference.

Our part is an open dataset: 24,000+ votes across 1,990 matchups, from landing pages to SVG art to code. The methodology is transparent and the data is free to download.

See for Yourself

Pick two models. Same prompt, real outputs. Vote, then check if the benchmarks agree.

Rival Index Download Dataset Methodology

FAQ

Benchmarks vs Vibes

AI labs publish benchmark scores to prove their model is the best. We collected 24,000+ blind preference votes to see if users agree. Mostly, they don't.

24,000+

Blind votes

276

Models ranked

1,990

Unique matchups

19.3%

End in ties

The Disconnect

Every lab launches with benchmark scores. MMLU, HumanEval, MATH, ARC. Higher number, better model. That's the pitch.

What Each Approach Measures

Benchmarks

+Factual recall on multiple-choice tests
+Code completion on defined problems
+Math reasoning under controlled conditions
+Reproducible, automated scoring

Measures the ceiling of narrow capabilities. Not which model you'll actually prefer.

Vibe Testing

+Human preference on real, open-ended tasks
+Design taste, tone, and aesthetics
+Creative and subjective output quality
+Blind comparison, no brand bias

Measures what users prefer when they see the output. Harder to fake.

What We Found

19.3%

of matchups end in a tie. Nearly 1 in 5 times, users can't tell two models apart. On everyday tasks, the top models have converged.

276

models have logged enough blind duels to earn a rank. It's not a few leaders and everyone else. Hundreds are competitive.

82%

win rate for Qwen3.6 27B, currently topping the Rival Index. It leads no traditional benchmark. The two are measuring different things.

SVG

art is among the most-voted challenges, like drawing an Xbox controller. People test design taste, not math proofs.

We're Not the Only Ones Saying This

Our part is an open dataset: 24,000+ votes across 1,990 matchups, from landing pages to SVG art to code. The methodology is transparent and the data is free to download.

See for Yourself

Pick two models. Same prompt, real outputs. Vote, then check if the benchmarks agree.

Rival Index Download Dataset Methodology

FAQ

Benchmarks vs Vibes

AI labs publish benchmark scores to prove their model is the best. We collected 24,000+ blind preference votes to see if users agree. Mostly, they don't.

24,000+

Blind votes

276

Models ranked

1,990

Unique matchups

19.3%

End in ties

The Disconnect

Every lab launches with benchmark scores. MMLU, HumanEval, MATH, ARC. Higher number, better model. That's the pitch.

What Each Approach Measures

Benchmarks

+Factual recall on multiple-choice tests
+Code completion on defined problems
+Math reasoning under controlled conditions
+Reproducible, automated scoring

Measures the ceiling of narrow capabilities. Not which model you'll actually prefer.

Vibe Testing

+Human preference on real, open-ended tasks
+Design taste, tone, and aesthetics
+Creative and subjective output quality
+Blind comparison, no brand bias

Measures what users prefer when they see the output. Harder to fake.

What We Found

19.3%

of matchups end in a tie. Nearly 1 in 5 times, users can't tell two models apart. On everyday tasks, the top models have converged.

276

models have logged enough blind duels to earn a rank. It's not a few leaders and everyone else. Hundreds are competitive.

82%

win rate for Qwen3.6 27B, currently topping the Rival Index. It leads no traditional benchmark. The two are measuring different things.

SVG

art is among the most-voted challenges, like drawing an Xbox controller. People test design taste, not math proofs.

We're Not the Only Ones Saying This

Our part is an open dataset: 24,000+ votes across 1,990 matchups, from landing pages to SVG art to code. The methodology is transparent and the data is free to download.

See for Yourself

Pick two models. Same prompt, real outputs. Vote, then check if the benchmarks agree.

Rival Index Download Dataset Methodology