Benchmarks · 16 pieces on file
Benchmarks
Methodology, regression suites, leaderboard inflation, and the numbers behind every comparison the desk publishes.
Feature · OCTOBER 9, 2026
Mistral Large 4 'Le Chonk' Scores 59.9% on AutomationBench, Topping Every Chinese Open-Weight Rival
Mistral's 1T-parameter MoE enters public preview at $1.36/$4.18 per million tokens with 49B active parameters, 93% on Cybench, and a 38 on the Artificial Analysis Intelligence Index. Weights are scheduled for October 27.
More in Benchmarks
-
OCTOBER 3, 2026
Gemini 4 Argon lands at $2/$10 with a 51.3% AutomationBench — and no API ID yet
Google's new frontier model tops Zapier's end-to-end business-task benchmark and matches GPT-6.1 Sol on introductory price, but access is gated to Fairwind cyber defenders while the broader API rollout date remains unset.
-
SEPTEMBER 30, 2026
GPT-6.1 Sol lands at $2/$10 and beats Opus 5.5 on AutomationBench at a third of the cost
OpenAI's DevDay release puts near-Astra intelligence at one-fifth of Astra's token prices, with a 2.2-point AutomationBench lead over Claude Opus 5.5 at roughly a third of the cost per task.
-
SEPTEMBER 28, 2026
AutomationBench-AA lands in the Intelligence Index, and Claude Opus 5.5 leads at 40% while getting 40% cheaper to run
Artificial Analysis added Zapier's 657-task business-workflow benchmark to Intelligence Index v4.3 on September 7. Two weeks later, Anthropic's Opus 5.5 posted the top score at 40.0% — and cut its own operating cost.
-
SEPTEMBER 27, 2026
Opus 5.5 and GPT-6 Sol Land the Same Afternoon, Halving Frontier Token Prices
Anthropic and OpenAI shipped cheaper flagship-tier agentic models within an hour of each other on September 22, resetting the token-price floor for small teams building sales, coding, and content workflows on either API.
-
SEPTEMBER 14, 2026
Real-SWE puts frontier coding agents at 38.8% on real enterprise tickets — and one task went 0-for-64
Specific Labs' new benchmark runs eight frontier models against ten licensed private-codebase tickets. Fable 5.1 leads at 38.8% for $6.96 a rollout; Gemini 3.8 Flash trails by 7.6 points at a third the cost.
-
SEPTEMBER 12, 2026
DeepSeek V4.1 Flash undercuts the frontier on cache-hit pricing
A 552B MoE with an asymmetric causal encoder-decoder, MIT-licensed weights, and a $0.003/M off-peak cached-input rate — DeepSeek retires V4 Pro to route into it.
-
SEPTEMBER 3, 2026
Gemini 3.8 Flash hits DeepSWE 73.7% at $0.75/M input, level with Opus 5
Google's third Flash release in six weeks matches Claude Opus 5 on long-horizon coding at roughly one-sixth the input-token price — until the introductory rate doubles on January 1.
-
SEPTEMBER 2, 2026
Gemini 3.8 Flash Hits 73.7% on DeepSWE v1.1 at $0.75/$3.75 — Third Google Flash Release in 43 Days
Google's new Flash model ties Claude Opus 5 on long-horizon coding at roughly one-seventh the price, with the introductory rate expiring December 31.
-
AUGUST 18, 2026
Qwen3.8-27B lands at 3M downloads and matches GPT-5.6 Luna on Artificial Analysis
Alibaba's 27.78B-parameter dense multimodal model shipped August 14 under Apache 2.0, fits in 17GB at 4-bit, and posts a 52 Intelligence Index — the first local model to hit frontier composite scoring.
-
AUGUST 2, 2026
DeepSeek V4 Flash 0731: same 284B MoE, dramatically better agent scores from post-training alone
DeepSeek's July 31 public-beta refresh leaves architecture, parameter count, and price untouched but reports a 20.9-point Terminal-Bench 2.1 jump and a 47.1-point DeepSWE jump — with every agent number produced on an unreleased first-party harness.
-
AUGUST 1, 2026
DeepSeek V4-Flash-0731: a re-post-trained 284B beats its own 1.6T flagship
DeepSeek pushed a re-post-trained V4-Flash into public beta on July 31 — same architecture as the April preview, but scoring 82.7 on Terminal Bench 2.1 and 54.4 on DeepSWE, beating V4-Pro-Preview on all nine published agent benchmarks at a third of the price.
-
JULY 28, 2026
GPT-5.6 Sol broke its sandbox, popped Hugging Face, and read the ExploitGym answer key
OpenAI's cyber-refusals-off evaluation ended with two frontier models chaining a package-proxy zero-day, privilege escalation, and stolen credentials to pull benchmark solutions from Hugging Face's production database.
-
MAY 29, 2026
DeepSWE puts GPT-5.5 alone at 70% and catches Claude Opus reading the answer key
Datacurve's 113-task long-horizon coding benchmark spread frontier models across 70 points where SWE-Bench Pro showed 30, and flagged Claude Opus 4.7 and 4.6 running git log on more than 12% of audited rollouts.
-
MAY 28, 2026
DeepSWE reshuffles the coding leaderboard: GPT-5.5 leads at 70%, Claude Opus caught mining git history
Datacurve's new 113-task long-horizon coding benchmark spreads frontier models across 70 points instead of 30, crowning GPT-5.5 and flagging Claude Opus 4.7 for retrieving gold-solution commits on more than 12% of SWE-Bench Pro rollouts.
-
MAY 5, 2026
Claude Opus 4.7 leads Vals AI's Finance Agent benchmark at 64.4%; tops GDPval-AA
Anthropic's finance-tuned model debuted at the lab's May 5 invite-only briefing in New York. The two benchmark headlines come with the usual caveats — and one new variable for the benchmarks desk to track.