Hardcore Reviews
Hardcore Reviews

24 of 30 Firsts on One Self-Test Table, Still Not a Win

The official Baidu model card publishes a 30-row evaluation table comparing ERNIE-5.5-Flash-Base (280B/14B) with DeepSeek-V4-Flash-Base (284B/13B), Hy3-preview-Base (295B/21B) and MiMo-V2.5-Base (310B/15B) under one framework, identical prompts and few-shot examples and a single judge model. In this vendor-run evaluation ERNIE tops 24 of 30 rows with MMLU 93.3, MMLU-Pro 78.4, HumanEval+ 82.3, LiveCodeBench 40.1 and RepoQA 91.7, while losing six rows including ChineseSimpleQA, BBH with chain of thought, INCLUDE-lite, FullStackBench, HELMET-Rerank and InfiniteBench. This is not a license comparison. It explains what such a self-reported table can and cannot show: no third-party replication, all four are pre-post-training Base checkpoints, scores are not chat or agent experience, and cross-leaderboard comparisons are invalid. It then covers activated-parameter ratio, KV cache, 128K training versus 1M inference extrapolation and post-training cost as selection criteria.

Published November 6, 20269 min read
<!-- base-model-benchmark-comparison-review | review | 24 of 30 Firsts on One Self-Test Table, Still Not a Win -->

Let me draw the boundaries first, so this piece does not overlap with older site posts or the three sibling articles in the same batch. The October 23, 2026 review Open Weight, But How Open? Five Model Licenses Compared is a license-tiering exercise, grading terms, territory, and revenue thresholds. This article is not a license comparison; licensing gets one sentence and a link near the end. The August 29, 2026 review Tencent's 770B Flagship Activates Only 49B: Five Open-Weight Flagships Compared covers 700B-to-2.8T scale and VRAM thresholds; this article moves to the 280B-to-310B band, to Base checkpoints at the end of pre-training, and asks how to read one self-test table. In the same batch, the hotspot piece covers the November 5, 2026 release and license boundary, and the setup SOP covers installation. This piece does methodology and selection only: no release recap, no command lines, no license breakdown. Those three answer what it is, whether you may use it, and how to install it; this one answers why the score table is worth trusting, and why it still is not the whole story.

1. Put the four base models and the table on the desk

On November 5, 2026, Baidu released ERNIE 5.5 Flash Base as open weights, and the Hugging Face model card shipped with a 30-item evaluation table, placing four Base checkpoints under one process: ERNIE-5.5-Flash-Base, DeepSeek-V4-Flash-Base, Hy3-preview-Base, and MiMo-V2.5-Base. Up front: every one of the 30 scores is from Baidu's unified evaluation pipeline, self-reported, with no third-party reproduction published. They are vendor self-reported numbers, not an objective ranking, and support no absolute claim about who is strongest. Every figure below carries that qualifier.

ModelTotal paramsActivated per tokenActivation ratio
ERNIE-5.5-Flash-Base280B14B5.0%
DeepSeek-V4-Flash-Base284B13B4.6%
Hy3-preview-Base295B21B7.1%
MiMo-V2.5-Base310B15B4.8%

Setup, per the official blog: all four use their own publicly released checkpoints, one shared evaluation framework, identical prompts and few-shot examples, and the same judge model for free-form answers. The 30 items span seven groups: Exam, Knowledge, STEM, Reasoning, Multilingual, Code, and Long Context. Inside this table, ERNIE takes the top score on 24 items and is not top on 6. "Inside this table" means a relative position from Baidu's unified evaluation, self-reported, without third-party reproduction; swap the framework or prompts and the order could reshuffle.

2. What the shared ruler actually standardizes

The value is not the scores; it pins down the variables that inflate benchmark numbers. The same model scores differently with a different harness, prompt, few-shot set, or judge model, and a heavier scaffold inflates the score, so vendor leaderboards often measure packaging rather than capability. Baidu folds four models into one framework, one prompt and few-shot set, and one judge, so the remaining gap can be attributed to the models, not the scaffold. But "unified" and "trustworthy" differ; the next section takes the second apart.

3. Five things the ruler still does not fix

First, it is still vendor self-reported: Baidu's unified evaluation, self-reported, with no independent lab re-running all four checkpoints. Self-reported is not fabricated, but it stays self-reported data, never an objective ranking.

Second, all four are Base checkpoints at the end of pre-training: no mid-training and no post-training, a starting point for domain adaptation, instruction tuning, and RL research, not a chat model. The 30 items measure a foundation for further training, not chat-box experience or agent task completion. Endorsing an assistant with them is a category error.

Third, scores are comparable only within this table; cross-leaderboard comparison is forbidden. Terminal-Bench, Agent Arena, and the AA Intelligence Index have their own task sets, scaffolds, compute budgets, and scoring rules; their setups cannot be horizontally aligned. Being ahead on 24 items here says nothing about placing first on some agent leaderboard, nor the reverse. This is the reddest line in this batch.

Fourth, the table has scores and no costs: no throughput, latency, VRAM footprint, parallelism config, and no architecture, context length, license, training data, or deployment cost for the other three. The first two rows give only total and activated parameters; this article copies exactly that and speculates about nothing outside it.

Fifth, 24 first places is not comprehensive dominance. The 6 misses cluster on sharply defined capabilities, making "leading" a structured claim. The honest reading: a unified, but still self-reported, and Base-only reference starting point.

4. Read the six misses one by one

ItemERNIE-5.5-Flash-BaseTop scorer in this tableIts score
ChineseSimpleQA73.0DeepSeek-V4-Flash-Base75.3
BBH (CoT)89.2Hy3-preview-Base90.0
INCLUDE-lite79.5DeepSeek-V4-Flash-Base79.8
FullStackBench54.6Hy3-preview-Base56.7
HELMET-Rerank56.2DeepSeek-V4-Flash-Base60.0
InfiniteBench46.5DeepSeek-V4-Flash-Base48.0

Qualifier as before: all six are from Baidu's unified evaluation, self-reported, with no third-party reproduction; the opponent score is the highest value for that item within this table, not any cross-leaderboard comparison.

ChineseSimpleQA tests Chinese factual QA. Top is DeepSeek-V4-Flash-Base at 75.3, ERNIE-5.5-Flash-Base at 73.0. These hinge on precise recall of Chinese world knowledge, so ERNIE is not uniformly ahead here; "best at Chinese" does not follow from 24 first places.

BBH (CoT) tests long-chain reasoning. Top is Hy3-preview-Base at 90.0, ERNIE at 89.2, the smallest gap. In a CoT setting, changing the prompt or dropping chain of thought can reshuffle the order, so one item cannot become a verdict on reasoning.

INCLUDE-lite tests multilingual coverage. Top is DeepSeek-V4-Flash-Base at 79.8, ERNIE at 79.5, again a hair apart. For a multilingual scenario, total wins do not help; this cell does.

FullStackBench tests full-stack code. Top is Hy3-preview-Base at 56.7, ERNIE at 54.6. Inside the Code group ERNIE is top here on HumanEval+ 82.3, MBPP+ 81.5, LiveCodeBench 40.1, and RepoQA 91.7, yet trails on full-stack. "Strong at code" must be broken into sub-capabilities.

HELMET-Rerank tests reranking in long context. Top is DeepSeek-V4-Flash-Base at 60.0, ERNIE at 56.2, the largest gap. In the Long Context group ERNIE leads on HELMET-ICL 85.9, RepoQA 91.7, and LongBench-v2 48.7, but trails on reranking, so long-context ability cannot be one number.

InfiniteBench tests very long documents. Top is DeepSeek-V4-Flash-Base at 48.0, ERNIE at 46.5. With HELMET-Rerank it points to one thing: ERNIE's 128K training context with 1M for fine-tuning and inference does not push every long-text benchmark to the top of this table.

The misses cluster on Chinese factual QA, CoT reasoning, multilingual coverage, full-stack engineering, reranking, and very long documents, while the 24 first places sit mostly in Exam, STEM, basic code, and most of Long Context. Leading is not leading everywhere; every digit remains self-reported with no third-party reproduction.

5. Read the wins: ten checkable samples

ItemERNIE-5.5-Flash-BaseDeepSeek-V4-Flash-BaseHy3-preview-BaseMiMo-V2.5-Base
MMLU93.389.487.587.6
MMLU-Pro78.469.266.965.7
HumanEval+82.369.578.074.4
MBPP+81.567.278.374.3
LiveCodeBench40.131.933.532.4
GPQA60.255.950.251.4
RepoQA91.781.091.356.5
LongBench-v248.744.539.844.6
HELMET-ICL85.983.883.080.2
PIQA96.095.095.293.6

Whole table: Baidu's unified evaluation, self-reported, no third-party reproduction; values as on the official model card, showing relative position under one shared ruler, not an objective ranking.

Even one capability is uneven internally. The gaps between MMLU 93.3 and MMLU-Pro 78.4, and between HumanEval+ 82.3 and LiveCodeBench 40.1, come from item difficulty and evaluation format, not a model shifting by mood. On RepoQA, ERNIE 91.7 and Hy3-preview-Base 91.3 are nearly tied while MiMo-V2.5-Base is 56.5, so the same group has different strengths on different sub-items. Those highs and lows came from one framework, one prompt set, and one judge model, so relative positions are clean. Still self-reported; re-run comparable items from your own tasks rather than copy a total.

6. Selection: six accounts beyond the scores

Account one is activation ratio and per-token compute. The four total/activated pairs are 280B/14B (about 5.0%), 284B/13B (about 4.6%), 295B/21B (about 7.1%), and 310B/15B (about 4.8%). Activated parameters set per-token compute, total parameters set weight size and VRAM footprint; the two must be computed separately.

Account two is KV cache and the long-context implementation, applying to ERNIE only. Per the official account, 37 of 43 layers use HCA heavily compressed attention, compressing every 128 tokens into one cache entry, while 6 layers of MLA-DSA keep token-level full-context access; training context is 128K, and fine-tuning plus inference support 1M. That sets VRAM and throughput on long sequences, but applies to ERNIE alone; the official table offers no comparable detail for the other three, and this article speculates about none. To check the implementation or wire up inference, start from PaddlePaddle/FlashMask and baidu/vLLM-ERNIE.

Account three is the 128K-to-1M extrapolation boundary. Training at 128K with 1M supported for fine-tuning and inference is ERNIE's official account, so 1M is extrapolated, not a natively pre-trained length; at that scale verify with your own task runs rather than trust a point such as LongBench-v2 48.7 or InfiniteBench 46.5.

Account four is post-training cost. All four stop at the end of pre-training; turning one into a usable assistant means mid-training, instruction tuning, and alignment on your side, covered in the Base fine-tuning SOP. The table measures a starting point for further training, not an out-of-the-box product; waiting for the official post-trained version is steadier.

Account five is the serving engine and parallelism config. The VRAM budget and sharding of 280B-class weights need their own calculation; see the VRAM planning guide. The table carries no throughput, latency, or hardware config, so a high score does not mean it runs cheaply.

Account six is license territory, in one sentence: ERNIE 5.5 Flash Base is open weights, not unconditionally open source, and its custom license carries territory and revenue thresholds; for clause-level checks see the license comparison. The official table gives no license information for the other three either, and nothing is inferred.

7. Conclusion

For continued training or domain adaptation, choose a Base whose provenance is clear and whose evaluation methodology is written down, and treat this table as a reference starting point under one shared condition. For ready-to-use behavior, wait for the official post-trained version; do not let pre-training-endpoint scores stand in for product experience. End on your own tasks: run a few real sample types blind a few times and take the mean, which beats staring at 24 first places. Once more: every number here is from Baidu's unified evaluation, self-reported, with no third-party reproduction; it is not an objective ranking, it does not align across leaderboards, and it cannot be packaged into an absolute cross-leaderboard claim.

FAQ

Q1: 24 first places, so can we say ERNIE won? A1: No. These are from Baidu's unified evaluation, self-reported, with no third-party reproduction, so self-reported data, not an objective ranking; 6 of the 30 items are not the highest here; all four are un-post-trained Base models, so scores do not equal chat or agent experience; and cross-leaderboard comparison is forbidden.

Q2: Can this table be compared with Terminal-Bench, Agent Arena, or the AA Intelligence Index? A2: No, cross-leaderboard comparison is forbidden. Those leaderboards have different task sets, scaffolds, compute budgets, and scoring rules; their setups cannot be horizontally aligned, so a position here neither implies a position there nor can be inferred back.

Q3: All four are Base models, so do the scores represent real conversational experience? A3: No. All four are checkpoints at the end of pre-training with no mid-training and no post-training, a starting point for further training rather than a chat model. For ready-to-use behavior, wait for the official post-trained version or do post-training yourself.

Q4: Does this table give the other three models' context, license, or deployment cost? A4: No. The first two rows give only total and activated parameters; this article copies that and speculates about nothing. 128K training, 1M for fine-tuning and inference, 37 HCA layers with one cache entry per 128 tokens, and 6 MLA-DSA layers apply to ERNIE only and cannot be carried over.

Q5: So what should I actually do with this table? A5: Treat it as a reference starting point under one shared condition. It unifies framework, prompts, few-shot examples, and judge model, good for shortlisting candidates and seeing relative strengths. But it remains self-reported with no third-party reproduction, so re-run on your own tasks and fold in activation ratio, KV cache, the 128K-to-1M extrapolation, post-training cost, and deployment thresholds.

Related reading

This article is AI-assisted and human-edited. Last updated: 2026-11-06

FAQ

24 first places, so can we say ERNIE won?
No. These are from Baidu's unified evaluation, self-reported, with no third-party reproduction, so self-reported data, not an objective ranking; 6 of the 30 items are not the highest here; all four are un-post-trained Base models, so scores do not equal chat or agent experience; and cross-leaderboard comparison is forbidden.
Can this table be compared with Terminal-Bench, Agent Arena, or the AA Intelligence Index?
No, cross-leaderboard comparison is forbidden. Those leaderboards have different task sets, scaffolds, compute budgets, and scoring rules; their setups cannot be horizontally aligned, so a position here neither implies a position there nor can be inferred back.
All four are Base models, so do the scores represent real conversational experience?
No. All four are checkpoints at the end of pre-training with no mid-training and no post-training, a starting point for further training rather than a chat model. For ready-to-use behavior, wait for the official post-trained version or do post-training yourself.
Does this table give the other three models' context, license, or deployment cost?
No. The first two rows give only total and activated parameters; this article copies that and speculates about nothing. 128K training, 1M for fine-tuning and inference, 37 HCA layers with one cache entry per 128 tokens, and 6 MLA-DSA layers apply to ERNIE only and cannot be carried over.
So what should I actually do with this table?
Treat it as a reference starting point under one shared condition. It unifies framework, prompts, few-shot examples, and judge model, good for shortlisting candidates and seeing relative strengths. But it remains self-reported with no third-party reproduction, so re-run on your own tasks and fold in activation ratio, KV cache, the 128K-to-1M extrapolation, post-training cost, and deployment thresholds.

Related

Hardcore Reviews

Eight Flagship LLMs, Only Half Accept Your Image

A November 4, 2026 snapshot of the OpenRouter API lists 453 models, 308 of them accepting image input. This comparison puts eight flagships side by side on the same API fields: input modality, context length, and per-million-token input/output/cache-read prices. tencent/hy4 is text+image with a 1,048,576 context at 0.417/1.2505/0.021 dollars, while tencent/hy4-preview, GLM-4.6, Kimi K2 Thinking and DeepSeek V3.2 remain text-only on that channel. Includes a full table, scenario picks, and three caveats: no cross-leaderboard scoring, aggregator prices are not vendor list prices, and modality tags are not capability measurements.

Nov 4, 20269 min read
Hardcore Reviews

Comparing 11 Models by Real Token Cost After the August 31 Repricing: Peak Hours, Cache Hits, and Tokenizer Effects

A model's list price wears at least three more layers. Time of day: DeepSeek moved to peak and off-peak pricing on August 17, charging peak rates on weekdays from 09:00-12:00 and 14:00-18:00, halving them off-peak, and applying off-peak rates all weekend, so the same model costs twice as much at 3pm as at 10pm. Caching: prefix cache hits are billed far below standard input, and the variable sits with your prompt structure rather than the vendor. Tokenization: Sonnet 5 changed tokenizers, so the same input now maps to 1.0x to 1.35x more tokens, and the multiplier floats with content type. This comparison fixes one unit throughout, blended rate equals input plus output divided by two, assuming equal token volumes, as a neutral starting point, then recalculates under three realistic load profiles across 11 models, covering list price, cached input, peak and off-peak, and post-tokenizer position. The finding is not which model is cheapest, it is that no model is cheapest, only cheapest for your particular load: any comparison that ignores input-output ratio, cache hit rate, and content type is comparing list prices, not costs. Chinese model prices come from a page-by-page check of official pricing pages on 2026-08-24, re-confirmed on 08-28; overseas prices from a 2026-08-31 roundup. Conflicts are flagged per line. No live benchmarking was performed.

Aug 31, 20269 min read
Hardcore Reviews

What Does a 30-Second 1080P Video Actually Cost: A Six-Way AI Video Generation API Cost Comparison

Wan3.0's launch turns "what does one 30-second 1080P clip actually cost" into a question you can compute precisely. This comparison runs the money ledger across six video generation APIs: Wan3.0 at an official 1.2 RMB/s, 36 RMB for a single-segment 30-second clip (25.2 RMB discounted through 09-23); Kling 3.0 around 30 RMB but requiring 3 stitched segments; Sora 2 pro breaking 100 RMB for 30 seconds; Hailuo's 768P at just 12 RMB across 3 segments, the cheapest in the table. Our exclusive ledger exposes the single-segment duration cap as an overlooked hidden cost - segment count x gacha multiplier (15-20% below-bar rate) x stitching labor is the real price - plus a two-tier playbook (480P gacha, 1080P final render) that saves 65%. All prices tagged with official vs aggregator provenance; a representative comparison, not a stress test.

Aug 25, 20269 min read