Specific Labs published Real-SWE on September 12, and the top score on its ten-ticket private-codebase eval is 38.8%. That number belongs to Fable 5.1, run through Claude Code. One of the ten tasks, an analytics stream reducer, went 0-for-64.

The benchmark is the first serious attempt at a coding eval built entirely on licensed production code, drawn from a Luma/Partiful competitor with 200K+ users, a consumer fintech processing 100K+ bank statements, and unnamed enterprise AI sales platforms. Scoring is pass@1 averaged over eight independent runs per task with 95% confidence intervals. Eight frontier models, ten tasks, eight rollouts each: 640 rollouts in total.

The full board: Fable 5.1 at 38.8%, GPT-6 Astra in Codex CLI at 33.8%, Gemini 3.8 Flash in Gemini CLI at 31.2%, GLM 5.3 (also in Claude Code) at 28.8%, Grok 4.6 and Muse Spark 1.3 tied at 23.8%, Kimi K3 at 18.8%, and GPT-5.6 Sol trailing at 16.2%. Reference solutions edit a median of 11 files. Prompts average roughly 1,742 characters and are deliberately underspecified. Six of ten tasks resolve below 15% overall.

The interesting number isn't the rank. It's the cost. Gemini 3.8 Flash costs $2.50 per rollout, Fable 5.1 costs $6.96, a 2.8x spread across a 7.6-point capability gap. Multiply through by solve rate and Fable 5.1 costs roughly $18 in compute per solved task; Gemini 3.8 Flash costs roughly $8. That's a 2.25x difference in cost-per-outcome, running in the opposite direction of the leaderboard. GPT-6 Astra sits at $4.67 per rollout, GLM 5.3 at $5.12 for its fourth-place finish.

Failure modes, categorized against the DeepSWE taxonomy, are where the eval turns into a product spec. Missed requirements dominate overall: 67.2% of Grok 4.6's failed runs, 53.8% of Kimi K3's. OpenAI models skew toward unverified assumption (43.3% for GPT-5.6 Sol, 34.0% for GPT-6 Astra). Gemini 3.8 Flash skews toward integration error at 49.1%. GLM 5.3, Grok 4.6, and Kimi K3 recorded zero regressions.

Longer rollouts didn't help. Runs under ten minutes failed 71.4% of the time; runs of ten minutes or more failed 73.4%. Specific Labs reads this as a triage-difficulty ceiling: the models that can't figure out what to do can't figure it out with more compute either.

Specific Labs' framing is blunt: missed requirements are the most common failure across models.

Hacker News picked it up quickly (271 points, 151 comments), with practitioners confirming the 20–35% real-codebase range they see internally. The FrontierCode and DeepSWE evals reported median edits of six files. Real-SWE nearly doubles that, and the leaderboard collapses accordingly.

Real-SWE puts frontier coding agents at 38.8% on real enterprise tickets — and one task went 0-for-64

Sources