When AI Fails: Engineering Fault-Tolerant Mission-Critical Systems

Engineers who have supported production systems know the feeling. A dashboard turns red at two in the morning, and the first question is not just what broke but also how far will it spread. Over two decades of building financial systems, I have learned that the answer to that question is almost never decided during the incident. It was decided months earlier, in architecture reviews, when someone chose whether a dependency would be allowed to take the whole system down with it.

AI is now one of those dependencies. It is increasingly used for fraud screening, customer service, dispute handling, operational tooling, and workflows that influence financial transactions. Like any dependency, AI can fail or degrade.

What makes AI different is how it fails. Traditional services tend to fail loudly: they time out, return errors, or stop responding. AI services fail loudly too, but they also fail quietly. A model can return a fast, well-formed, confident answer that is simply wrong. From the network's point of view, nothing happened. From the business's point of view, something very important did.

Fault tolerance for AI means handling both kinds of failure. The good news is that we do not have to invent the discipline from scratch. The patterns that made distributed systems dependable (timeouts, circuit breakers, bulkheads, retries, fallbacks, graceful degradation) still apply. They just need to be extended to cover failures that look like success.

Start with the Right Assumption

The most important design decision is a mental one. In too many architectures I review, the AI component is drawn as a solid box in the middle of the diagram, with arrows going in and out and no failure path at all. It is treated as if it were a database lookup or a pure function.

An AI component should be treated as a dependency that can fail, much like a third-party API over the public internet, with explicit limits and a tested fallback. That framing immediately changes the questions people ask:

• What is the latency at the 99th percentile, not the average?
• What happens when the provider rate-limits us during a traffic spike?
• What happens when a model version changes underneath us?
• How do we detect an answer that is well-formed but wrong?
• What does the business process do if the AI is gone for an hour?

The classic dependability literature gives us useful language here. Avižienis, Laprie, Randell, and Landwehr, in their widely cited IEEE taxonomy of dependable computing [1], separate faults, errors, and failures: a fault is the cause, an error is the incorrect internal state, and a failure occurs when the delivered service deviates from correct service. With AI, the fault is often invisible (a subtle data shift, a gap in training data, an ambiguous prompt), and the error looks like a normal response. It becomes important to stop that error before it becomes a failure.

That leads to the principle behind the rest of this article: reliability must come from the system around the model, not from the model itself.

The Classic Patterns, Revisited for AI

Michael Nygard popularized circuit breakers and bulkheads in Release It! [2] nearly twenty years ago. These patterns remain useful around AI. But each needs some adjustment, because AI calls behave differently from the service calls these patterns were first designed for.

Timeouts: Budget for the Long Tail

Large model inference has a wide latency spread. Response times can vary substantially with load, input size, output length, and shared computing resources. Setting a timeout based only on average latency can leave the system exposed to slow responses.

The better approach is to start from the business deadline and work backward. If a card authorization must complete in well under a second end to end, the AI step gets a fixed slice of that budget, and when the slice is spent, the system moves on without it. For generated text, I also recommend limiting output length explicitly. Limiting output length helps control latency, but the call still needs an end-to-end deadline.

Circuit Breakers: Trip on Bad Answers, Not Just Errors

A traditional circuit breaker opens when the error rate crosses a threshold. For AI, that is necessary but not enough. If a model starts returning well-formed responses that fail validation, a conventional breaker will happily stay closed, because every call "succeeded."

I think of the fix as a semantic circuit breaker. It counts validation failures, low-confidence results, and policy rejections alongside timeouts and HTTP errors. When the combined rate crosses a threshold, the breaker opens and traffic goes to the fallback path. These signals should be tracked by use case and model version so that one problematic request does not disable unrelated workflows. Recovery should use controlled probes that pass both availability and validation checks.

Bulkheads: Keep AI Off the Critical Thread

AI calls are slow and variable. If they share a thread pool, connection pool, or rate limit with core transaction processing, a slowdown in the model becomes a slowdown in everything. Bulkheads isolate those resources. In practice, that means separate pools for AI calls, separate quotas per use case, and a firm rule that a batch job summarizing documents can never consume the capacity reserved for real-time use cases on the critical path.

Retries: Fewer, Smarter, and Budgeted

Retries need careful limits. Retrying an AI call is expensive in both latency and cost, and when a provider is overloaded, aggressive retries make the overload worse. The guidance from Google's SRE practice [3] and Amazon's Builders' Library [4] applies directly: cap attempts, use exponential backoff with jitter, and enforce a retry budget across the whole service so that retries cannot exceed a small fraction of total traffic.

Two additional rules are specific to AI. First, do not retry a response simply because you did not like it; a response that failed validation should normally go to the fallback path rather than be repeatedly sent to the same model with the same prompt. Any bounded repair attempt should fit within the same deadline and validation controls. Second, if the AI can take actions, such as calling tools or updating records, retried actions need idempotency keys or another explicit mechanism to prevent duplicate effects. A retried agent step should not update the same record or execute the same transaction twice.

Model Fallback and Response Validation

The classic patterns protect us from loud failures. Quiet failures need two more mechanisms working together: validation, which reduces the risk of using an unacceptable answer, and fallback, which decides what to do when it cannot.

Validate Every Answer Before It Is Used

In financial systems, we never trust a message just because it arrived. We check its format, its signature, its amounts, and its business rules before it touches a ledger. AI output deserves the same discipline. I usually describe validation in four layers:

1. Structure. Does the response match the expected schema? Are required fields present, types correct, values in range?
2. Business rules. Does the answer respect known invariants? A refund recommendation should never exceed the original transaction amount; a risk score should never be applied to an account type it was not trained on.
3. Grounding. For retrieval-based systems, is the answer supported by the source documents it cites, or has the model added facts that are not there?
4. Confidence. Where a calibrated confidence or uncertainty estimate is available, does it meet the threshold for this decision? A model's own statement that it is confident should not be treated as evidence of correctness.

The first two layers are cheap and deterministic, and every production system should have them. Grounding and uncertainty checks can require additional processing, so teams should choose them based on the use case and the harm a wrong answer could cause. Passing validation does not guarantee that an answer is correct.

Design the Fallback Chain Deliberately

When the primary model times out, the breaker is open, or validation fails, the request needs somewhere to go. That destination should be designed with the same care as the primary path, not added as an afterthought.

A typical chain moves from more capable to more predictable: a smaller model with reduced scope or a cached, previously validated response that remains applicable, then a deterministic rules engine, then a human review queue or a safe hold. Each step trades capability for more constrained behavior. The chain should also be honest with downstream systems. A decision made by the rules engine should be labeled as such, so that monitoring, audit, and customer communication all know which path produced it.

A fallback model offers limited protection if it shares the primary model’s infrastructure, quotas, or other failure modes. If the primary and fallback share a failure mode, they could fail together. Independent failure modes improve resilience. Deterministic logic can provide a useful fallback when its rules remain current and appropriate for the use case.

Safe Degradation When AI Is Unavailable

Fallbacks can also become unavailable or reach their limits, and the AI capability may need to be switched off when its behavior is no longer acceptable. The question is what the business does then.

The answer should never be improvised. For each AI-assisted process, the best practice is to define its degradation modes in advance and write them down:

The most important decision in this table is one that architects cannot make alone: for each use case, should the system fail open or fail closed? A fraud model that is unavailable during a holiday peak could continue processing under approved static rules (a constrained fallback) or hold or decline transactions when the required risk assessment is unavailable. Strictly failing open would bypass the unavailable check; failing closed would block the action. A rules-based fallback sits between those choices. Neither answer is universally right. What matters is that the business chose it deliberately, before the incident, and that the choice is encoded in the system rather than left to whoever is on call.

Two practices make degradation real rather than theoretical.

• Keep the deterministic path alive. Many organizations retire their rules engines after the model goes live. Where it remains an approved fallback, the deterministic path should stay deployed and be monitored like any production service. Exercise it through controlled traffic or shadow testing, as appropriate for the risk of the workflow.
• Break it on purpose. Chaos engineering has taught our industry that a fallback needs to be tested under realistic failure conditions. Game days should include switching off the AI dependency entirely, injecting slow responses, and feeding the validator deliberately bad outputs.

Where to Start

For teams that already run AI in production, a short review will usually surface the biggest gaps:

• Find the missing failure paths. For every AI call, trace what happens on timeout, on error, and on an invalid answer.
• Upgrade the circuit breakers. Make validation failures and low-confidence results count toward tripping them.
• Separate the resources. Give AI calls their own pools and quotas, away from core transaction processing.
• Write the degradation table. Agree on fallback rules and fail-open or fail-closed behavior with the business owner, and encode them.
• Schedule the game day. Turn the AI off in a controlled test and watch what happens.

Conclusion

Fault tolerance starts with acknowledging that dependencies can fail even in a resilient system. We build redundancy because we know hardware fails, networks partition, and people make mistakes. AI makes another failure mode especially important: a component can be available, fast, and confident, and still be wrong.

The patterns we need are mostly the ones we already know. Timeouts, circuit breakers, bulkheads, retries, fallbacks, and graceful degradation remain the backbone of dependable systems. What changes is the definition of failure. Once we accept that a wrong answer is a failure even when the response code says success, those familiar patterns extend naturally to cover AI.

In my earlier article [5], I wrote that designing systems that cannot fail does not mean eliminating failure. Fault-tolerant AI is where that idea becomes concrete. The model will sometimes fail. The system should contain those failures and preserve the agreed business safeguards.

References

• [1] A. Avižienis, J.-C. Laprie, B. Randell, and C. Landwehr, “Basic concepts and taxonomy of dependable and secure computing,” IEEE Transactions on Dependable and Secure Computing, vol. 1, no. 1, pp. 11–33, Jan.–Mar. 2004, doi: 10.1109/TDSC.2004.2.
• [2] M. T. Nygard, Release It! Design and Deploy Production-Ready Software, 2nd ed. The Pragmatic Bookshelf, 2018.
• [3] B. Beyer, C. Jones, J. Petoff, and N. R. Murphy, Eds., Site Reliability Engineering: How Google Runs Production Systems. O’Reilly Media, 2016.
• [4] M. Brooker, “Timeouts, retries, and backoff with jitter,” Amazon Builders’ Library. [Online]. Accessed: Oct. 6, 2026.
• [5] A. Sardana, “Mission-Critical AI: Engineering Systems That Cannot Fail,” IEEE Computer Society Tech News, Jul. 23, 2026.

About the Author

Aman Sardana is a technology leader and recognized industry expert in financial technology, payments infrastructure, and enterprise software architecture. He is recognized for his contributions to the design and modernization of large-scale financial systems, with a focus on cloud architecture, resilient platform design, and the modernization of mission-critical digital infrastructure. Aman shares insights on emerging technology trends in cloud architecture and system reliability at international technology and leadership forums. Aman holds a Master of Science in Information Technology from Northwestern University, is a Fellow of BCS, and a Senior Member of IEEE.

Disclaimer: The authors are completely responsible for the content of this article. The opinions expressed are their own and do not represent IEEE’s position nor that of the Computer Society nor its Leadership.