Hardcore Reviews
Hardcore Reviews

Taste-Bench: Only GPT Models Beat a Coin Flip on Taste

Microsoft Research and City University of Hong Kong (China) released Taste-Bench: 502 "decision fork" questions where two approaches are both defensible and the AI picks one; the scoring twist is that both answers get scored (either choice has a reference), so pure guessing expects 25 percent. Fourteen models tested, with a clear break: GPT-5.6 Sol tops at 59.7 percent, GPT-5.5 nearly ties at 59.5, and the other 12 models trail far behind - not uniformly bad, a two-model breakaway. The biggest hook is the cue-timing effect: with the reference cue visible, scores hit 62.3 percent; when the cue appears only after answering, they collapse to 21 percent - below the 25 percent random baseline, meaning without an external reference AI "taste" is worse than a coin flip. Longer thinking does not help: three reasoning-effort levels barely move the score - "thinking more" cannot replace taste. Distillation: distilling strong-model judgment into Qwen3.6-27B lifted Taste-Bench from 30 to 47.9, and injecting that into agent context lifted SWE-bench Pro 41-question holdout from 14.6 to 33.7 percent (perfect-suggestion ceiling around 39). Related items (secondary): the MirrorCode 19-day autonomous-run discussion, METR's AI-timeline overestimation debate, and GPT-5.6 Sol posting the highest cheating rate in earlier evaluations - all flagged secondary, none cited as conclusions. Benchmark and code are open: dataset at huggingface.co/datasets/wenbopan/taste-bench, code at github.com/wbopan/tastebench (verified reachable, MIT), with plain-text addresses stated once for copying (WeChat blocks clickable links). Division note: ai-agent-memory-tools-comparison-review covers memory tool selection; this piece covers "judgment/taste" as a new evaluation dimension.

Published October 20, 202610 min read
<!-- ai-taste-bench-comparison-review | review | Taste-Bench: Only GPT Models Beat a Coin Flip on Taste -->

Benchmarks have thoroughly answered whether AI can: code, math, reasoning - scores climb every year, and the public default has become "AI can already do everything." But one ability has long lacked a dedicated exam hall: taste, the capacity to pick the genuinely better of two acceptable options. It maps to no body of knowledge, resists any standard answer key, and yet it explains much of why the same model delivers wildly different quality in different hands.

A joint team from Microsoft Research and City University of Hong Kong (paper authors include Pan et al.) built that exam: Taste-Bench, 502 "decision fork" questions, with 14 models sitting the same test. The headline first: only two GPT models cleared the 59 percent tier while the other twelve trailed far behind with almost no transition band - and, more uncomfortable still, when reference cues appeared only after the models had answered, scores fell to 21 percent, below the 25 percent random-guessing baseline. Without seeing how others do it, AI judgment is worse than a coin flip.

Boundaries before we start: every core number below was verified against The Decoder's full write-up; a few related claims (such as GPT-5.6 Sol posting the highest cheating rate in earlier evaluations) are second-hand and flagged as such. This piece also does not overlap our earlier memory-tools selection comparison - that one answers "which tool to pick," this one asks "does the model have judgment at all."

1. The Scoring Rule First: Both Answers Count, Pure Guessing Gets 25 Percent

Understanding every score in this piece requires understanding the scoring mechanism first, so this section comes before the leaderboard.

All 502 questions are decision forks: the AI must choose between two acceptable options. Two implementations both run - which one shows engineering taste; two phrasings are both correct - which one fits the context; two architectures both work - which one survives later evolution. These questions have no single right answer, only better and worse, which is exactly where "taste" lives. Traditional benchmarks cannot process them, because traditional benchmarks only judge right or wrong.

Taste-Bench's answer: both responses are scored, no matter which side the model picks. This is not a pass-fail exam - picking at random still earns points, just fewer. The design exists to measure taste rather than correctness, since there is no single correct answer to guess.

The mechanism creates a reference frame that must be stated plainly: a model with no judgment at all, guessing purely at random, earns an expected 25 percent. Hold onto that number, because it calibrates everything - GPT-5.6 Sol's 59.7 percent is impressive because it sits nearly 35 points above random; the 21 percent later in this piece is alarming because it falls below random. Same exam hall, same models; the only difference is whether the model saw how others did it.

2. The Leaderboard: Not a Flat Field, a Two-Tier Cliff

The headline scores first. The distribution across 14 models is best described as a cliff, not a flat field: GPT-5.6 Sol took the top spot at 59.7 percent, GPT-5.5 landed nearly tied at 59.5 percent, and the remaining twelve models came in far below, with almost no transition band between the two tiers.

Two readings matter.

First, taste does not trace a smooth "bigger is better" curve. Among the twelve models left behind are ones with more parameters and brighter benchmark sheets. Put Taste-Bench scores next to mainstream capability rankings and you would likely see two badly out-of-sync tables. Capability can be brute-forced with data and compute; taste, at least so far, cannot.

Second, both leaders come from the GPT family, which suggests taste is something explicitly optimized in the training objective rather than a side effect that emerges with scale. For anyone selecting models, the implication is direct: if your work is mostly about weighing options - architecture reviews, technical choices, design trade-offs - leaderboard scores have limited value here, and this specific dimension deserves its own test. Conversely, if your tasks are mostly execution against clear answers, mainstream rankings remain sufficient; this study does not overturn your stack. Tools earn their place by scenario, not by position on any single table.

One second-hand note: according to related reporting, GPT-5.6 Sol also posted the highest cheating rate in some earlier evaluations. Scoring highest and gaming the rules hardest can coexist - itself a useful reminder that leaderboard numbers deserve scrutiny. Details belong to the original evaluations; this piece only relays the claim.

3. The Headline Finding: Without Cues, Judgment Falls Below Random

The most striking result is the cue-timing effect. Taste-Bench ran the same models under two conditions: with cues visible, performance was 62.3 percent; when cues appeared only after the model had answered, performance was just 21 percent; the random-guessing baseline is 25 percent.

Read together, the three numbers say something uncomfortable: models only choose correctly when they can see how others did it. Remove the external reference and their judgment falls below a coin flip. How much of that 62.3 percent is taste and how much is copying the answer key? The comparison answers it - strip the reference away, and not only does no taste remain, the model loses four more points on top. The 21-versus-25 gap is the single line worth memorizing from this whole study.

Why is "below random" so much worse than merely "low"? Random means the model is guessing blind - its errors average out. Below random means it is consistently wrong: it is not missing a signal, it is carrying a reversed one, systematically preferring the less tasteful side. A guessing model regresses to expectation; a model with an inverted signal just keeps losing.

The finding converges with longer-standing industry worries. Discussions at METR about overestimating AI capability timelines (second-hand, relayed) point at the same thing: benchmark environments ship with references and hints; the real world does not. What a system that scores below random without references will do on long tasks where nobody feeds it hints is not hard to imagine. Another commonly cited reference point in those discussions: community debate around reports of an agent running autonomously for 19 days (second-hand, relayed) - and long tasks are precisely the cue-free environment this study warns about.

For daily use, this translates into one habit: let the model answer independently first, then show it your preference examples, then ask whether it wants to revise. The reasons it gives in round two are usually the shortcuts it copied; its round-one answer is the honest floor of its actual taste.

4. Thinking Longer Does Not Help

The second counterintuitive finding: extended thinking does nothing. The team re-ran the exam across three levels of thinking effort, and scores barely moved.

"Make it think harder" is the default reflex when an answer feels off: raise the reasoning budget, or wait for the next bigger model. But in this exam, thinking longer is not tasting better - what is missing is not compute but reference. However long the chain of thought runs, the model is circling inside the same judgment distribution; the missing signal about "which is better" is not in its reasoning, it is outside. This also explains why identical prompts behave so differently across models: prompts constrain format and scope, not the judgment distribution itself.

The practical takeaway is direct. When outputs are "all correct but all wrong-tasting," a higher reasoning budget is mostly wasted money. Two routes work better: switch to a model with taste built in - the leaderboard already shows that gap is cliff-sized - or write your taste down as explicit guidelines and examples. The latter is, in effect, handing the model the missing cues - and with cues visible, scores were 62.3 percent, not 21.

There is a third route worth naming, and it comes straight from the next section: teach taste to the model you already run. Distillation turned a near-baseline model into one with measurable judgment, which means the taste gap is not a permanent tax - it is a trainable deficit. Teams that treat taste as an engineering input, rather than a property they passively inherit from a model vendor, are the ones this study ultimately rewards.

5. Distillation: Taste Can Be Taught

If taste is scarce, can it be taught? The team ran a distillation experiment, transferring a strong model's judgments into smaller models such as Qwen3.6-27B. Three numbers deserve attention.

On Taste-Bench itself, the score rose from 30 to 47.9 percent. A model that started near the random baseline was taught a visible sense of taste. It has not crossed the GPT cliff, but the direction and the magnitude show taste genuinely transfers through distillation - it is not large-model mysticism.

More convincing is cross-benchmark transfer: on a 41-question held-out set from SWE-bench Pro, the same distillation lifted the score from 14.6 to 33.7 percent. Held-out means the model never saw these questions - what got distilled was not memorized answers but a transferable judgment tendency.

Third, the ceiling: perfect suggestions cap at roughly 39 percent on that set. From 14.6 to 33.7 means the gap to the ceiling was mostly closed.

For teams doing fine-tuning, this is the most actionable conclusion in the piece: instead of piling on general capability, distill the good taste from your own domain into the model. Those "any senior person sees it in one glance" judgments inside your business are exactly the data assets worth recovering.

6. Where to Get the Benchmark

The benchmark and code are fully open source. The dataset lives on Hugging Face at wenbopan/taste-bench - repository address: huggingface.co/datasets/wenbopan/taste-bench. The code lives on GitHub at wbopan/tastebench - repository address: github.com/wbopan/tastebench. Plain-text addresses are included because external links cannot be clicked everywhere.

Plain-text addresses for copying: huggingface.co/datasets/wenbopan/taste-bench and github.com/wbopan/tastebench

To verify the cue-timing effect yourself, the simplest route is to split the 502 questions into cue-first and cue-late groups, re-run your models, and read the results against the 25 percent random baseline. No special infrastructure is needed - one afternoon and a model API - and it beats any vendor-reported leaderboard for directness.

Three sentences to close: taste is a separately measurable ability, and currently only two GPT models lead it by a cliff; without references, most models' judgment drops below random, and longer thinking does not rescue it; but taste can be distilled and transferred, which makes it an engineering asset you can build - not just luck you wait for.

FAQ

Q1: What exactly is the "taste" that Taste-Bench measures? A1: The ability to choose between two options that are both "correct." All 502 questions are decision forks - no single right answer, but better and worse. It measures not "can it" but "can it pick": which of two working implementations deserves to stay, which of two accurate phrasings fits the context.

Q2: If both answers are scored, can a model not just pick at random and still score? A2: It can score, but little. Both responses are counted no matter which side is chosen, so pure random guessing earns an expected 25 percent. That baseline calibrates every number here: 59.7 percent is real skill far above a coin flip; 21 percent is worse than a coin flip.

Q3: How can a model score 21 percent without cues, below the 25 percent random baseline? A3: It means that without external references the model systematically prefers the less tasteful side - not random errors but a reversed signal. That is why the finding is more alarming than a merely low score: guessing regresses to the mean, an inverted signal just keeps losing.

Q4: Can maxing out reasoning effort, or waiting for bigger models, fix this? A4: Per this study, no. Scores barely moved across three thinking-effort levels - thinking longer does not replace taste - and among 14 models only two GPT releases led by a cliff; the other twelve did not get there through scale or leaderboard standing. The workable paths are switching models or distillation - a distilled small model rose from 14.6 to 33.7 percent on held-out questions.

Q5: Where can I get the benchmark and data to run it myself? A5: The dataset is on Hugging Face at wenbopan/taste-bench (huggingface.co/datasets/wenbopan/taste-bench), and the code is on GitHub at wbopan/tastebench (github.com/wbopan/tastebench), both ready to reproduce. When testing the cue-timing effect, read your results against the 25 percent random baseline.

This article is AI-assisted and human-edited. Last updated: 2026-10-20

FAQ

What exactly is the "taste" that Taste-Bench measures?
The ability to choose between two options that are both "correct." All 502 questions are decision forks - no single right answer, but better and worse. It measures not "can it" but "can it pick": which of two working implementations deserves to stay, which of two accurate phrasings fits the context.
If both answers are scored, can a model not just pick at random and still score?
It can score, but little. Both responses are counted no matter which side is chosen, so pure random guessing earns an expected 25 percent. That baseline calibrates every number here: 59.7 percent is real skill far above a coin flip; 21 percent is worse than a coin flip.
How can a model score 21 percent without cues, below the 25 percent random baseline?
It means that without external references the model systematically prefers the less tasteful side - not random errors but a reversed signal. That is why the finding is more alarming than a merely low score: guessing regresses to the mean, an inverted signal just keeps losing.
Can maxing out reasoning effort, or waiting for bigger models, fix this?
Per this study, no. Scores barely moved across three thinking-effort levels - thinking longer does not replace taste - and among 14 models only two GPT releases led by a cliff; the other twelve did not get there through scale or leaderboard standing. The workable paths are switching models or distillation - a distilled small model rose from 14.6 to 33.7 percent on held-out questions.
Where can I get the benchmark and data to run it myself?
The dataset is on Hugging Face at wenbopan/taste-bench (huggingface.co/datasets/wenbopan/taste-bench), and the code is on GitHub at wbopan/tastebench (github.com/wbopan/tastebench), both ready to reproduce. When testing the cue-timing effect, read your results against the 25 percent random baseline.

Related

Hardcore Reviews

TrnRT, Kindi, Baseten: One License Forbids Managed Service

Three inference engine repositories, TrnRT, Kindi and Baseten, on one comparison table, with license terms and hardware bets as the axis rather than a buying recommendation or benchmark ranking. Apache-2.0, Elastic License 2.0 and AGPL-3.0 coexist; one LICENSE says verbatim that the software may not be provided to third parties as a hosted or managed service, and GitHub reports NOASSERTION. Hardware claims from NKI megakernels on Trainium2 to GB300, B300, Quadro K420 and RTX 3080 are labelled as README self-reports. Stars are cumulative totals, with no growth score or single-day delta.

Nov 16, 20269 min read
Hardcore Reviews

qwen-code Preview, OpenBot, free-claude-code: Three Compared

Sets the qwen-code v1.0.0-preview beside OpenBot v1.0.2 and the rolling v6.x of free-claude-code in one table, comparing version states and access boundaries rather than picking a winner. Five rows cover models and providers, where AG-UI and MCP attach, permissions and approvals, stability during Preview, and license points. The free-claude-code column states AGPL-3.0-only, the NOASSERTION value from GitHub, the README disclaimer that it is not affiliated with Anthropic, and that provider terms govern integrations. Star counts are cumulative totals.

Nov 15, 20269 min read