Why AI Agents Fail in Production and How to Build Reliable Ones: Q&A With Suman Debnath

Reliable artificial intelligence (AI) agents are attracting growing attention as organizations move beyond demonstrations (demos) and into real-world deployment. Early agent demos often appear impressive, yet many fail when exposed to unpredictable production environments. Nearly 95% of AI pilots fail to reach production, highlighting the persistent gap between experimentation and operational reliability.

Understanding that gap requires looking beyond model performance. In many cases, the challenge is not model intelligence but engineering maturity, specifically the infrastructure, governance, and reliability practices required to support autonomous systems.

Suman Debnath, director of developer relations at Crusoe and an IEEE Senior Member, has spent nearly two decades in distributed systems engineering. Previously head of developer relations at Anyscale and a principal developer advocate for generative AI at AWS, he has worked on large-scale AI infrastructure and developer frameworks. In this interview, Debnath discusses why AI agent demos collapse in production and how organizations can build reliable agentic systems.

Q: Why do impressive agent demos fail when exposed to real users and production environments?

Debnath: Most demos operate under ideal conditions. Inputs are predictable, constraints are clearly defined, and workflows are carefully orchestrated. Production environments are fundamentally different. In a demo, a user might say “schedule a meeting at 3 p.m.” A real request might be “schedule a meeting next week, but not on Tuesday,” requiring time zone resolution, cross-calendar availability, and constraint handling.

Agents function as chains of multiple actions, and small errors accumulate as workflows grow more complex. For example, a 10% error rate at each step reduces a seven-step workflow to just 48% end-to-end reliability.

Many organizations misinterpret these failures as model weaknesses. In practice, the issue is often a system-level gap involving integration, orchestration, or infrastructure. The challenge is not model intelligence, but system surrounding it. We saw this repeatedly at AWS with enterprise teams building on Amazon Bedrock. The model performed well, but the surrounding system lacked the error handling, state management, and integration resilience that production demands.

Q: What reliability primitives are required for production-ready agent systems?

Debnath: Moving agents from prototypes to production requires engineering controls similar to those used in distributed systems. In practice, this requires persistent state management, which preserves context, intermediate outputs, and execution progress. When failures occur, agents can resume from a checkpoint rather than restarting entire workflows. It also includes  structured retry-and-recovery mechanisms that handle intermittent failures gracefully and prevent brittle execution patterns.

Behavioral guardrails validate inputs and enforce output schemas to ensure that agent responses align with application expectations. Audit logging records every decision an agent makes, providing visibility into system behavior and supporting debugging, compliance reviews, and root-cause analysis.

Agents can take hundreds of actions during a workflow. Without audit logging, diagnosing failures becomes significantly more difficult. According to IBM’s 2024 Cost of a Data Breach Report, organizations that implement AI governance controls reduce breach costs by an average of $2.22 million, underscoring the value of these operational safeguards.

Q: How can organizations secure tool access and prevent unintended agent behavior?

Debnath: As agents interact directly with application programming interfaces (APIs), internal services, and enterprise data, security architecture becomes critical. According to Obsidian Security's analysis of enterprise SaaS environments, 90% of AI agents are over-permissioned, often holding 10 times more privileges than necessary.

Organizations shouldn’t treat agents as extensions of the users who launched them. Instead, they should implement authentication and authorization policies like a microservice and use a three-layer security architecture.

This begins with tool-level access control, which defines the tools an agent can use and under what conditions. It also includes runtime sandboxing, where execution occurs in isolated environments with limits on time, memory, and system access, helping constrain failures.

Finally, semantic guardrails evaluate actions for potentially destructive behavior, such as deleting large volumes of data or modifying sensitive systems.

Security architecture is most effective when integrated into orchestration frameworks from the beginning rather than introduced after deployment.

Q: What governance models preserve autonomy while maintaining safety?

Debnath: Autonomous agents introduce a balance challenge: excessive oversight slows automation, while insufficient oversight increases risk. I recommend a governance framework based on three operational modes. These modes are human-in-the-loop for high-risk actions, human-on-the-loop for monitoring autonomous processes, and human-out-of-the-loop for low-risk, fully automated tasks.

One widely applicable approach is exception-based governance, where agents operate autonomously until predefined triggers activate human review. These triggers may include cost thresholds, low-confidence model outputs, interactions with sensitive resources, or actions with a large operational impact.

This approach enables agent autonomy while facilitating human intervention when defined boundaries are exceeded. It also aligns with emerging regulatory expectations, including risk-based frameworks in the European Union’s AI governance policies.

Q: What design principles determine whether agents operate autonomously or assist humans?

Debnath: Three factors determine suitability for autonomous execution. The first is reversibility. Tasks, such as drafting emails or generating reports, that can be easily undone lend themselves to autonomous workflows. Irreversible actions, such as financial transactions or production system changes, typically benefit from human oversight.

The second factor is context completeness. Agents require sufficient context to make reliable decisions. When information is incomplete or subjective judgment is required, assistive automation is more appropriate. The third factor is repeatability. Tasks with predictable inputs and outputs are easier to automate safely than ambiguous or one-off scenarios.

Organizations often implement these principles through a staged maturity model that progresses from shadow mode (where agents observe human workflows), supervised assistive mode (where agents help with tasks), supervised autonomous mode (with limited independence), and fully autonomous execution. This progression allows teams to build confidence in agent reliability over time.

Q: What innovations are shaping the future of reliable agentic systems? 

Debnath: Several developments could define the next generation of AI infrastructure. One emerging concept is the agentic operating system, a standardized runtime environment designed specifically for autonomous workflows. Think of it like Kubernetes for agents. It provides scheduling, state management, observability, and policy enforcement in a unified platform.

Another trend involves agent-to-agent communication protocols, such as emerging standards for multi-agent collaboration. These protocols enable autonomous systems developed by different vendors to interact seamlessly. Finally, distributed computing frameworks are enabling parallel agent execution, enabling organizations to run thousands of agent workflows simultaneously.

Several industries are implementing agents in operational environments. In semiconductor manufacturing, companies such as AMD and Synopsys have used reinforcement-learning agents to assist with chip design verification—a complex process that requires near-perfect accuracy.

Healthcare organizations are also exploring agentic automation.  In one case, Fujitsu-supported hospital systems in Japan reported saving more than 400 staff hours and generating approximately $1.5 million in additional revenue through improved operational efficiency. These deployments demonstrate measurable value and underscore the importance of reliable underpinning services and governance.

From my own work at AWS, the zero-setup RAG approach I helped develop for Amazon Bedrock Knowledge Bases is now used by over 1,000 enterprises. Adoption is driven by reliability guarantees rather than model capability.

The Goal of System Reliability

As more AI agents are used in mission-critical systems, the transition from prototype to a production is a harsh wake-up call. Model intelligence alone is not the determining factor. Rather, adherence to proper systems engineering principles and practices is key to achieving system reliability.

The evolution of modern enterprise systems, along with the widespread adoption of multi-agent systems, enterprise service buses, and cloud computing, has led to a significant shift in software architecture. These trends include agentic operating systems, standard communication protocols, and large-scale platforms.

Intelligent agents will act as glue points to enable communication and interaction among systems and allow the composition of new activities and workflows across an enterprise environment, also known as the fourth generation of software architecture. In that future, the reliability of autonomy will be more a function of the systems in which the algorithms are embedded than of the sophistication of the code itself.

About the Author

Paul Chaney is a seasoned writer, editor, and content strategist who helps businesses craft compelling, ethical marketing narratives through his consultancy, Prescriptive Writing. With a focus on clarity, authenticity, and responsible communication, Paul empowers organizations to tell their stories with purpose and precision. Connect with him on LinkedIn.

Disclaimer: The authors are completely responsible for the content of this article. The opinions expressed are their own and do not represent IEEE’s position nor that of the Computer Society nor its Leadership.