CloudOpsBench Finds Top AI Agents Fail Many Real Cloud Operations Tasks

CloudOpsBench was built by a team led by Ang Chen at the University of Michigan. It sets agents typical DevOps jobs on Amazon Web Services, including repairing services, removing resources, tightening access rights, rolling out updates and decommissioning projects. Version 1.0 has 45 tasks, all written by experienced DevOps engineers, according to the report.

Anthropic's Claude Fable 5.1 had the best result, solving 60 per cent of the tasks. OpenAI's GPT-6 Astra solved 53 per cent.

The researchers say the models mostly fail because they reason poorly, not because they lack knowledge. About 80 per cent of failures happened when agents misjudged how far a task extended, did not check every relevant service, or stopped too soon. The other roughly 20 per cent were wrong actions, and some of those would have caused irreversible damage to the infrastructure.

Every task runs identically against the live AWS cloud and against an emulator, so teams can test agents without touching production systems. The team plans to extend the benchmark to Google Cloud, Microsoft Azure and Oracle Cloud Infrastructure. The code is published on GitHub under the Apache 2.0 licence, along with a detailed analysis of the failures.

Why It Matters

Infrastructure management is one of the jobs AI vendors most often say agents can take over, and cloud providers and coding-tool makers are building agents that can change production environments directly. The CloudOpsBench results suggest that even frontier models need human review for operations work, especially for actions that cannot be undone, such as deleting resources or changing permissions. The finding that agents often stop early or miss related services also points to failure modes that standard knowledge tests may not catch.

CloudOpsBench adds to a growing set of agent benchmarks aimed at operations work. Earlier this year AWS published its own open-source suite, aws-bench, built from analysis of real AWS usage. CloudOpsBench differs in that it comes from academic researchers and focuses on multi-step administrative tasks with real consequences.

The results come from a single benchmark of 45 tasks on one cloud platform. How well they carry over to other environments will become clearer once the planned extensions to other providers are released.