Overview
SWE-CoWork asks whether a coding agent can inherit an ongoing engineering assignment and finish it as a dependable teammate. Instead of receiving a complete issue, the agent starts with a short handoff, a real repository, and access to an authored workplace. It must establish what remains to be built, implement the changes, verify them, and report the outcome.
The benchmark contains 101 assignments across 101 repositories, 11 languages, and 12 domains. Its central finding is a gap between partial implementation and complete delivery: the strongest evaluated agents pass roughly 63% of new-behavior tests, but fully complete fewer than 8% of assignments.
A workplace, not just an issue
Each organization has 26 simulated colleagues with distinct responsibilities, bounded local knowledge, and controlled disclosure rules. Requirements are spread across accountable owners, project boards, versioned documents, chat, reviews, and CI history. The agent must reconcile outdated decisions, revision-specific evidence, and information that appears during implementation.
These are substantial changes: a median assignment involves 2,955 production lines across 60 files and is checked by 32 executable test nodes. Every scored requirement is linked to an owner or discovery route and a fixed test contract, allowing the evaluation to trace it from clarification through final handoff.
How delivery is evaluated
- Fail-to-Pass (F2P) measures newly required behavior. Scores pool verified passes over the release’s 2,699 F2P test nodes; tasks with more nodes carry more weight.
- Pass-to-Pass (P2P) measures preservation of existing behavior over 562 regression test nodes. Complete task pass rate requires every F2P and P2P node to pass within the same assignment.
- The paired Oracle-Spec condition supplies the full requirements up front while retaining the same repository, tests, scaffold, and execution budget. It is evaluated for Seed Evolving and GPT-6 Astra, not all eleven models.
Partial progress, limited completion
The manuscript evaluates eleven models on all 101 tasks. The two leading F2P results are shown below; the remaining nine models stay below 40% F2P and at or below 0.99% complete task success. These are manuscript-reported results, not a live leaderboard.
| Model | F2P (%) | P2P (%) | Task pass (%) |
|---|---|---|---|
| Claude-Opus-5.5 | 63.84 | 93.24 | 6.93 |
| GPT-6 Astra | 62.91 | 89.15 | 7.92 |
Giving agents the complete specification improves F2P: Seed Evolving rises from 39.09% to 55.76%, and GPT-6 Astra from 62.91% to 78.84%. Yet their task pass rates remain 0.99% and 7.92%. Better access to requirements helps, but does not resolve the remaining implementation and verification failures.
Where requirements get lost
Analysis of Claude-Opus-5.5’s 101 recorded trajectories shows that finding the right person is not the main obstacle: 93% of first questions target the accountable owner, and the agent obtains 64% of colleague-held requirements. The larger challenge is keeping individual requirements open until they are resolved.
The agent leaves 153 of 450 deferred threads unresolved. Requirements already disclosed to it account for 56% of failed linked F2P nodes, while 31% of requirements reported as done fail their tests. These findings describe this model’s recorded cohort, rather than every evaluated agent.
What dependable digital employees need
The next step is sustained requirement tracking: connect each requirement to its source, follow up on deferred answers, verify the implemented behavior, and keep the final handoff consistent with test evidence. Gathering context or producing a large patch is not the same as completing an assignment.
Scope matters. The workplaces and colleague behavior are simulated around real repositories. Oracle-Spec measures both direct access to requirements and the search effort saved under a fixed budget, so its gains do not isolate information quality alone. This page summarizes the supplied manuscript dated October 23, 2026; model labels and results are retained as reported, without independent reproduction.