Detecting silent agent failures with Amazon Bedrock AgentCore optimization

If you’re operating AI agents at scale, Amazon Bedrock AgentCore surfaces a category of insights you’ve probably experienced but struggle to detect: your dashboards show green across the board. 99% completion rate, healthy latency, zero error spikes. And yet customer complaints trickle in about incorrect outcomes.

An order modification that was never actually executed. A product reported as “in stock” when the inventory API timed out. An approval step that was skipped. These are behavioral failures, and they look different from infrastructure issues. They complete successfully from the system’s perspective. They pass health checks and only surface through customer escalations, often weeks after they began affecting users at scale.

Even for failures that do produce error signals, a separate challenge exists: when an agent serving thousands of daily sessions accumulates hundreds of errors, which ones deserve attention first? Reviewing individual traces reveals what happened in a single session. But it tells you little about whether you’re looking at a pattern affecting 30% of traffic or an edge case affecting three sessions.

Amazon Bedrock AgentCore optimization provides insights that help you discover, explain, and prioritize behavioral failures in your deployed AI agents, including the silent ones that never generate an error signal. These insights shift the observability model from reactive trace inspection to proactive pattern detection. It helps you find the failures, understand their scope, and fix them in priority order. In this post, we walk through how insights in AgentCore optimization works, how to set it up, and what it reveals when pointed at a real agent.

What insights in Amazon Bedrock AgentCore optimization delivers

Insights operate one layer above your existing observability stack. It consumes the trace data your tools already collect and transforms it into actionable behavioral intelligence. This provides an investigation capability that helps you understand broader patterns affecting agent performance at scale, beyond individual session failures.

The following figure shows the end-to-end analysis performed by AgentCore to deliver insights. Each session analysis method extracts one or more attributes from each session. The attributes from each session analysis method are clustered independently. Each cluster is summarized to enhance interpretability.

The insights report surfaces the following:

Let’s now see each capability in depth.

AgentCore analyzes each session trace against a structured taxonomy of behavioral failure types. It detects 11 categories including hallucination, incorrect actions, task instruction violations, orchestration errors, context handling issues, and others. The analysis reasons about policy compliance and behavioral correctness rather than relying on error signals alone. This approach detects silent failures. Each detected failure produces a location in the trace, a category, and a natural language description of what went wrong. Insights then group the descriptions into clusters. This surfaces a single named pattern across hundreds of sessions, rather than hundreds of individual trace entries.

The following figure shows the failure clustering and root cause analysis (RCA pipeline). We identify failures for each session, cluster them into sub-clusters. For each session in a failure sub-cluster, we produce root cause explanations.

For each failure cluster, AgentCore traces backward through the execution graph to identify what caused the failure, not just where it appeared. The session is represented as a graph of spans (inference calls, tool executions, sub-agent invocations), and the system prunes branches unrelated to the failure before reasoning about causality. This pruning makes RCA tractable on long sessions by narrowing a 50-step workflow down to the specific execution path that led to the failure. The output surfaces a root cause location (a span ID in your logs), a causality classification, and a fix recommendation (system prompt change, tool description update, or infrastructure work). Insights then group root causes within a cluster. You might see that 90 of 100 failures share the same underlying cause and the same fix type.

Clustering alone tells you what kinds of failures exist. Scope ranking tells you which ones matter most. Each failure cluster carries a count of affected sessions, and clusters are ordered by that count relative to total sessions analyzed. The two-level hierarchy separates broad failure categories from specific sub-patterns.

A top-level cluster might show “Agent Bypasses Information Gathering” affecting 116 sessions. Drilling in reveals that 114 of those are a single sub-pattern (“Skipped Prerequisite Information Retrieval”). The remaining 2 are unrelated edge cases. This structure means you can immediately identify the single fix that addresses the largest share of failures.

Beyond failures, AgentCore extracts two additional views of agent behavior. User request clustering embeds and groups the actual prompts users send to your agent, revealing the distribution of use cases in production. You may discover that 40% of traffic is a use case you designed for. Another 30% is partially supported. The remaining 30% is something you never anticipated.

Execution summary extracts the approach the agent took and the outcome it reached for each session. It uses a hierarchical summarization strategy that handles sessions of varying lengths. AgentCore then clusters these summaries to reveal execution patterns. This includes common strategies the agent employs, how those strategies correlate with success or failure, and where the agent’s actual behavior diverges from your intended design.

Together, these give you a behavioral map of your agent in production that goes beyond error detection into understanding how the system is being used and how it responds.

How to set up insights in Amazon Bedrock AgentCore optimization

When you have an agent with its telemetry recorded in Amazon Bedrock AgentCore observability, you can use insights to understand failure modes, user intents and execution summaries of this agent.

You can create an insights configuration from the AgentCore console under Optimizations > Insights > Create Insights.

Under “Insights to generate,” select the analysis types you want:

You can run them independently or together.

You have two options for pointing insights at your session data:

Insights can be set up in two modes:

Recurring generates reports automatically on a cadence you choose. You can enable a combination of daily, weekly, and monthly analysis frequencies. Once created, insights processes sessions in the background with no further action required.

One-time generates a single report for a specific time range. Use this for post-deployment reviews, complaint-driven investigations, or metric anomalies you want to explain.

The “Enable this configuration once created” checkbox (checked by default) starts processing immediately after you save.

Expand “Filters and sampling” to scope analysis to specific session criteria or sample a subset of traffic. Expand “Permissions” to configure the IAM role insights uses to access your logs.

Once configured, choose Create Insights and your first report will be available based on the schedule you chose.

Consider a market trends agent that provides investment analysis, stock recommendations, and sector performance data. After running 10 sessions, here is what each insight analysis view surfaces.

Insights identified one failure category: “Hallucinated Financial Data Without Tool Invocation,” affecting 1 out of 10 sessions. The agent generated specific financial data points, market rates, and numerical claims and presented them as factual information without invoking available data retrieval tools. This violates the system prompt directive to retrieve actual data before making claims.

The root cause (“Fabricated Data Claims Without Tool Invocation”) traces this to a gap between stated requirements and execution enforcement. The system prompt tells the agent to ground responses in actual data but lacks sufficient enforcement mechanisms to mandate tool usage before presenting factual assertions. The agent bypasses data validation steps and presents unverified information as authoritative.

This failure produced no error. The session completed successfully. With insights, you see the root cause, the specific session affected, and a clear path to fix: strengthen the system prompt to enforce tool invocation before numerical claims.

Understanding what users ask your agent to do normally requires manually reading session transcripts. Intent analysis gives you this distribution automatically from the traces you already collect. User requests here are clustered into 3 categories, revealing the actual distribution of what users ask the agent to do:

Half of all traffic is profile and portfolio retrieval. This tells you where to invest in reliability first. If profile lookups fail or return stale data, you are degrading the experience for 50% of your users.

You know what your agent was designed to do but have limited visibility into what it does across sessions at scale. Execution summaries close that gap. Here the market agent behavior is grouped into three execution patterns:

The dominant execution pattern shows the agent completing full advisory workflows end-to-end. The two smaller patterns reveal alternative behavioral paths.

Amazon Bedrock AgentCore optimization provides insights when your agent is silently wrong. Use it for post-deployment validation when complaints spike but dashboards look normal. Perform proactive weekly monitoring that catches invisible failures before customers do. Use it for scope ranking assessment when a handful of complaints turns out to be less than 10% of actual impact. Whether you are validating a system prompt change against a time window, or tracing complaints back to affected sessions, the workflow is the same: submit sessions, receive ranked patterns, act on specific root causes. Enable insights in Amazon Bedrock AgentCore optimization for your agent and see what your dashboards are missing. Visit the documentation to learn more.

Acknowledgements: Thank you to the science and engineering team that delivered this capability: Sanjana Agarwal, Abhimanyu Siwach, Vincent Chen, Swarnim, Frankie Kim, Stephanie Yuan, Aron Thakur, Jaejin Cho, Muhyun Kim, Arron Bailiss, Shoaib Javed, Gitika Jha and Michael Fan.