AI Fails on More Than Half of Real Enterprise IT Tasks

You were sold AI agents that could replace a technical team. The first serious benchmark to put them in front of real enterprise IT tasks has just landed, and the result is sober: the best model succeeds on fewer than one task in two. Before signing the next "autonomous agent" contract, that figure deserves three minutes of your attention.

What the benchmark actually says

ITBench-AA is the first independent test bed that evaluates AI agents on real enterprise IT tasks, not on quiz questions. In practice, the model faces an incident on a Kubernetes infrastructure: alerts, logs, traces, a service topology. Its job is to trace the root cause, the way an on-call engineer would at 3 in the morning.

The verdict is public and measured. Across 59 tasks, the best model tops out at 47 percent success, the next at 46 percent. Every leading model stays under the 50 percent line. These are not fringe tools: they are the most advanced models available in May 2026.

One detail matters more than the ranking. Researchers found that giving the model more thinking time does not improve the result. A model that explores the incident over 83 steps scores lower than a more direct model over 31 steps. The agent is not held back by a lack of effort. It is held back by its ability to land on the right call in an ambiguous environment.

Why this figure, and not another

You may have seen benchmarks where AI clears 90 percent. They exist, and they are not lying. The difference comes down to the nature of the task.

When the question is closed (translate a text, classify an email, summarize a document), AI is excellent. When the task is open-ended, multi-step and grounded in a living system, the score collapses. A production incident has no single answer written down somewhere. You have to form hypotheses, test them, rule out false leads, and stop at the right moment.

A companion analysis from IBM and UC Berkeley dissected the failures. The causes are not exotic. The agent loses the thread of its reasoning. It concludes too early. Or it verifies its own answer badly. On one model's failed traces, poor verification of its own conclusion stands out as the number one fatal flaw. AI often finds the right lead, then buries it under false leads it cannot eliminate.

Augmenting is not replacing

This is where the figure becomes useful for deciding. 47 percent success in full autonomy is not zero, and it is not a failure. It is a boundary.

Put the same result another way: on a complex diagnostic task, AI produces a correct analysis in minutes roughly one time in two, and a usable starting point far more often. For an engineer who supervises, that is a serious accelerator. For a promise to "replace the team", it is a wall.

The right reading for a leader fits in one sentence: AI is a strong copilot, not an autopilot. The project that puts a competent human in supervision captures the value. The project that takes the human out of the loop inherits the 53 percent of errors, with no one to catch them.

Where AI holds, where it breaks

So the real question is not "should we do AI", it is "on which task". Here is the grid I use to decide.

The criterion that separates the two columns is not technical difficulty. It is the presence of a checkpoint. A task where you can review the result before it produces an effect is good ground for AI. A task where the decision goes straight to production with no review is a minefield, whatever the model.

The framing mistake that costs you

The classic trap is not choosing AI. It is framing it as a replacement where it can only be an augmentation. The project starts on the promise "the agent handles it alone", reality comes back at 47 percent, and the verdict drops: "AI does not work". Wrong. It was the framing that did not work.

The same tool, repositioned as a copilot with a human validating the sensitive cases, becomes profitable. The technology did not change. The place you give it in the process did. Defining that place before launching the project saves you from paying for a disappointment you could see coming.

How SolidScale handles this limit in the S3 method

The Scan, Solve, Scale method starts from this boundary, not from the marketing promise.

Scan: the free 30-minute audit does not look for where to "add AI". It looks for which tasks sit on the right side of the boundary: closed or supervisable, with a clear checkpoint. An open-ended, high-stakes task with no possible control is ruled out from the start. Better to say so before than after.

Solve: the tool delivered in 4 to 8 weeks keeps the human in the loop where the benchmark shows AI breaks. AI prepares, proposes, accelerates. The sensitive final decision stays validated. You capture the speed gain without inheriting the autonomous error rate.

Scale: scope expands only on tasks where the measured results hold. We never widen an agent's autonomy on the strength of a demo. We widen it on the strength of figures observed in real conditions.

Key takeaways

The first serious benchmark of enterprise IT tasks puts a clear figure on the table, and that figure is good news for anyone who knows how to read it.

• No leading model exceeds 47 percent success on autonomous IT diagnostic tasks.
• AI augments work, it does not replace it: excellent as a supervised copilot, fragile in open autonomy.
• The boundary is not technical difficulty, it is the presence of a checkpoint.
• The most common failure in business is not the technology, it is the framing that promises a replacement where only augmentation holds.

The concrete action: before any project, classify the target task. On the right side of the boundary, go. On the other side, keep the human in control.

Frequently asked questions

Does the benchmark say AI is useless in business?

No, the opposite. It says AI in full autonomy fails on complex tasks, which is different. On framed or supervised tasks, it remains very strong. The 47 percent figure measures an extreme case (the agent alone, with no human), not the real recommended use.

Should you wait for models to improve before starting?

No. Tasks on the right side of the boundary (classification, extraction, supervised copiloting) are already profitable today. Waiting means postponing accessible gains for tasks that are not ready yet. The right instinct is to sort, not to wait.

How do you know if a task is "supervisable"?

Ask one simple question: can you review the result before it produces a real effect. If yes, a human can validate and AI becomes a safe accelerator. If the decision goes straight to production with no possible review, the task is risky, whatever the model.

Will these figures age fast?

The precise score will move, models keep improving. But the underlying lesson is stable: AI stays stronger on closed tasks than on open-ended decisions in complex environments. Good framing stays valid even when the percentages climb.

Sources

• Artificial Analysis and IBM Research, ITBench-AA: Frontier Models Score Below 50% on the First Benchmark for Agentic Enterprise IT Tasks, May 27, 2026, huggingface.co/blog/ibm-research/itbench-aa
• IBM Research, ITBench-AA Benchmark Leaderboard, artificialanalysis.ai/evaluations/itbench-aa
• IBM Research and UC Berkeley, Diagnosing Why Enterprise Agents Fail Using ITBench and MAST, huggingface.co/blog/ibm-research/itbenchandmast
• Jha et al., ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks, ICML 2025, arxiv.org/abs/2502.05352