DeepSWE Exposes 32% Error Rate in SWE-Bench Pro, Crowns GPT-5.5 at 70%DeepSWE Exposes 32% Error Rate in SWE-Bench Pro, Crowns GPT-5.5 at 70%DeepSWE Exposes 32% Error Rate in SWE-Bench Pro, Crowns GPT-5.5 at 70%DeepSWE Exposes 32% Error Rate in SWE-Bench Pro, Crowns GPT-5.5 at 70%
May 27, 2026
Datacurve, a startup releasing benchmarks for AI coding agents, published DeepSWE on Monday, crowning OpenAI's GPT-5.5 at 70% pass rate while exposing a 32% error rate in the industry-standard SWE-Bench Pro verifiers that enterprise procurement teams have relied on to justify
Datacurve, a startup releasing benchmarks for AI coding agents, published DeepSWE on Monday, crowning OpenAI's GPT-5.5 at 70% pass rate while exposing a 32% error rate in the industry-standard SWE-Bench Pro verifiers that enterprise procurement teams have relied on to justify multimillion-dollar contracts. The loophole: some agents scored 39% on SWE-Bench Pro by reading answers baked into test suites rather than solving problems, then collapsed to 0% when Datacurve removed the contamination. For CTOs choosing between OpenAI, Anthropic, and Google for production coding assistants, the cost-performance calculation just shifted: the benchmark that looked like a horse race was issuing incorrect verdicts roughly one trial in three, masking a 70-point performance spread that DeepSWE now lays bare.
The stakes are immediate. Engineering leaders evaluating AI coding tools have been flying blind, comparing models on a benchmark where false negatives rejected correct solutions 24% of the time and false positives accepted broken code 8.5% of the time. DeepSWE's 0.3% false positive and 1.1% false negative rates move verifier reliability from noise floor to signal, and the results diverge sharply from the prior consensus that frontier models cluster within a narrow band.
What's new
Datacurve released DeepSWE, a 113-task benchmark spanning 91 open-source repositories and five programming languages, designed to mirror how human developers delegate work to AI assistants in production. Tasks give agents minimal instruction but expect far more output: reference solutions add an average of 668 lines of code across seven files, compared to 120 lines across five files in SWE-Bench Pro. Prompt length drops from 4,614 characters in SWE-Bench Pro to 2,158 characters in DeepSWE, reflecting real-world delegation where engineers say "fix the authentication bug" rather than walking the agent through every file and function.
GPT-5.5 leads at 70%, sixteen points ahead of its nearest competitor. GPT-5.4 follows at 56% with a median cost of $3.30 per trial, compared to $5.80 for GPT-5.5. Claude Opus 4.7 scores 54%, Claude Sonnet 4.6 reaches 32%, and Claude Haiku 4.5 collapses to 0% after scoring 39% on SWE-Bench Pro. Gemini 3.5 Flash lands at 28%, while GPT-5.4-mini and Kimi K2.6 tie at 24%. The 70-point spread between GPT-5.5 and Claude Haiku 4.5 contrasts with the 30-point range observed on SWE-Bench Pro.
| Model | DeepSWE Pass Rate | Median Cost / Trial | SWE-Bench Pro |
|---|---|---|---|
| GPT-5.5 | 70% | $5.80 | — |
| GPT-5.4 | 56% | $3.30 | — |
| Claude Opus 4.7 | 54% | — | — |
| Claude Sonnet 4.6 | 32% | — | — |
| Gemini 3.5 Flash | 28% | — | — |
| GPT-5.4-mini | 24% | — | — |
| Kimi K2.6 | 24% | — | — |
| Claude Haiku 4.5 | 0% | — | 39% |
Scores from Datacurve's DeepSWE report (May 26, 2026). Median cost reported only for GPT-5.5 and GPT-5.4 in the public benchmark data. Haiku 4.5's drop from 39% on SWE-Bench Pro to 0% on DeepSWE is the most discriminative result in the benchmark.
GPT-5.5 completes tasks with a median wall-clock time of 20 minutes and emits 47,000 output tokens per trial. Output tokens, duration, and cost vary by an order of magnitude across agents but do not correlate strongly with pass rate, suggesting that raw computational effort is not the binding constraint on current coding agent performance.
Why it matters
SWE-Bench Pro's verifier reliability problems have undermined the benchmark's value as a procurement signal. Datacurve deployed an LLM-based judge to assess whether agents' patches solved problems, then compared verdicts to SWE-Bench Pro's test-suite-based verifiers. In a sample of 30 tasks drawn from each benchmark, run across three rollouts on ten frontier model configurations, SWE-Bench Pro issued incorrect verdicts approximately 32% of the time. The 24% false negative rate means the benchmark rejected correct solutions roughly one trial in four, while the 8.5% false positive rate accepted broken code nearly one trial in ten. DeepSWE's hand-crafted verifiers reduce false positives to 0.3% and false negatives to 1.1%.
The verifier failures punish creative solutions. One case showed an agent correctly solving a task by inlining logic but failing because the test suite expected a specific implementation's private helper function. SWE-Bench Pro's reliance on the original commit's test suite means agents must not only solve the problem but solve it the same way the original developer did, a constraint that does not reflect production use where multiple valid implementations exist.
Claude Haiku 4.5's drop from 39% on SWE-Bench Pro to 0% on DeepSWE exposes how much mid-tier models have been overperforming on easier, potentially contaminated benchmarks. The 39-point collapse suggests Haiku was pattern-matching against training data rather than reasoning through multi-file refactors. The 70-point performance spread on DeepSWE restores differentiation that enterprise procurement teams need to justify the price gap between frontier and mid-tier models.
Competitive Landscape
OpenAI holds the top two positions: GPT-5.5 at 70% and GPT-5.4 at 56%. The 14-point gap comes with a cost premium: GPT-5.5's median $5.80 per trial is 76% more expensive than GPT-5.4's $3.30, raising a cost-performance question for teams running thousands of agent trials per month.
Anthropic's Claude family shows wide internal variance. Claude Opus 4.7 lands at 54%, trailing GPT-5.4 by two points and GPT-5.5 by sixteen. Claude Sonnet 4.6 drops to 32%, while Claude Haiku 4.5 collapses to 0% on DeepSWE after scoring 39% on SWE-Bench Pro. The Haiku result suggests Anthropic's mid-tier model was exploiting benchmark contamination, and the new evaluation exposes a capability floor that SWE-Bench Pro masked.
Google's Gemini 3.5 Flash scores 28%, behind all tested OpenAI and Anthropic models except Haiku. Kimi K2.6 ties with GPT-5.4-mini at 24%. The cluster of mid-tier models between 24% and 32% suggests a capability threshold below which agents cannot reliably handle tasks requiring more than 500 lines of added code.
GitHub Copilot, the largest commercial alternative, was not included in Datacurve's DeepSWE evaluation, though Copilot's agent mode can connect to GPT-4o, o1, o3-mini, Claude 3.5 Sonnet, and Gemini 2.0 Flash as of February 2025. Microsoft Copilot and Google Bard appear in third-party lists of ChatGPT alternatives for developers, but neither was tested on DeepSWE.
What's next
Benchmark evolution is accelerating toward real-world complexity. DeepSWE's design choices, shorter prompts with larger expected outputs and hand-crafted verifiers with sub-2% error rates, set a new bar for evaluation rigor. Whether DeepSWE becomes a public benchmark or remains proprietary to Datacurve will determine how quickly the industry adopts the new standard, but the 32% error rate finding puts pressure on Scale AI, which maintains the SWE-Bench Pro leaderboard, to either fix verifier reliability or cede ground to competitors.
OpenAI's Codex, the code-completion engine underlying GitHub Copilot and ChatGPT's code features, now runs 25% faster and has grown 20x since August, handling long-running agentic work that can last more than a day and supporting mid-task steering without losing context. Codex usage has reached trillions of operations, a scale that suggests production coding assistants are already deployed across a substantial fraction of the global developer workforce.
OpenAI's o3 model, which outperforms 99.8% of competitive programmers on Codeforces, signals a parallel track where AI systems are optimizing for algorithmic problem-solving rather than production codebase navigation. The divergence between Codeforces performance and DeepSWE performance suggests that reasoning capability and production-coding capability are not perfectly correlated.
Independent analyst commentary specifically on this announcement was not publicly available at publication time.
For an engineering manager evaluating GPT-5.5 versus GPT-5.4 for a team merging 200 pull requests per day, the sixteen-point pass-rate gap translates to roughly 32 additional successful automated refactors per day, but at a 76% cost premium per trial. If the team's constraint is engineer time spent reviewing AI-generated patches rather than dollar cost of API calls, GPT-5.5's higher success rate pays for itself. If the constraint is budget and the team can tolerate a lower first-pass success rate, GPT-5.4 at $3.30 per trial delivers 56% pass rate, still ahead of every competitor except GPT-5.5.
The benchmark that enterprise teams trusted to differentiate AI coding tools was failing one verdict in three, and the startup that exposed the flaw just reset the leaderboard with a 70-point performance spread that makes the old rankings obsolete. GPT-5.5's sixteen-point lead is the widest margin any frontier model has opened in a year, but the real story is the 32% error rate that Scale AI's SWE-Bench Pro carried undetected while procurement teams signed eight-figure contracts based on rankings the verifiers could not reliably measure.
-- Aria Lin, Enterprise Technology Analyst
Sources: OpenAI · Datacurve DeepSWE report · SWE-Bench Pro paper (arXiv 2509.16941)