In March 2024, Cognition Labs introduced Devin as "the first AI software engineer" — a framing that implied something close to a drop-in replacement for a human developer. The claim traveled fast. The scrutiny came almost as fast behind it, and it's worth separating what was real from what was marketing, because both parts are true at once.
The number that made headlines
Devin's headline result was 13.86% on SWE-bench, a benchmark that hands an AI real, unresolved GitHub issues and checks whether it produces a genuinely working fix — no hints about which files to edit, no scaffolding. The previous best score on this benchmark was roughly 2%, so a jump to 13.86% was, credit where due, a real leap in what was achievable at the time.
Two details are easy to miss in the headline version, though, and both come from Cognition's own reporting: it was self-reported, and it was run on a random 25% subset of the benchmark, not the full set. Flip the number around, and the same result reads differently — the most capable autonomous coding system demonstrated at that point still failed to resolve more than 86% of the tasks it was given.
What happened when someone else tested it
The more concrete evidence came in January 2025, when an independent technical reviewer ran a month-long, hands-on test: twenty real development tasks, no cherry-picking. Three were completed satisfactorily. Three were inconclusive. Fourteen failed outright. Twenty tasks is a small sample, not a census — but the failure pattern was the more informative part than the raw score: tasks that should have taken hours stretched into days, with Devin burrowing into dead ends and continuing to dig rather than recognizing it was stuck, producing elaborate but unusable output instead of surfacing the blocker to a human. That specific failure mode — not knowing when to stop and ask — is exactly the kind of judgment the "AI software engineer" framing implied it had.
What's actually true as of 2026
Devin has continued to develop since — pricing dropped substantially, later versions report improved task completion, and it's been piloted inside real engineering organizations. Independent reviews in 2026 broadly converge on a more modest, more useful framing than the original launch pitch: capable on narrow, well-specified tasks with a human reviewing the output, not a standalone replacement for engineering judgment on ambiguous, real-world work.
Why this belongs on a site about AI coding dependency
Devin is one company's product, but the pattern generalizes: the gap between how an AI coding tool is marketed and what it reliably does is usually wider than the headline implies, in either direction — dismissing all of it as useless is as inaccurate as the original "first AI software engineer" framing was. The skill this site is actually about — being able to independently judge what a tool got right and wrong — is exactly what you need to navigate that gap sensibly, for Devin or for whatever claims the next tool makes.
Check how well you catch what AI gets wrong →
Sources: Cognition Labs' original SWE-bench reporting, March 2024; independent testing reported by The Register, January 2025; 2026 reviews of Devin's subsequent development.
Related reading: Is Vibe Coding Actually Dangerous? · Is AI Making Developers Worse?