Configure rate limits for AI traffic on AgentCore gateway

Amazon Bedrock AgentCore gateway is a fully managed, serverless AI gateway that provides a single, secure entry point for AI traffic. AgentCore gateway routes traffic to tools such as managed web search, managed knowledge bases, MCP servers, inference models (LLMs), agents (A2A, agents as tools, etc.), or HTTP endpoint. Today, we are announcing support for rate limiting on AgentCore gateway, giving you fine-grained control over how much traffic individual users can consume through your gateway.

Rate limiting in AgentCore gateway gives you per-user control over how users consume your tools, inference models, and agents. Define OAuth or IAM-based rules for requests per minute, concurrent connections, and token throughput, making sure downstream services remain available under heavy traffic spikes.

Centralized rate limiting for AI traffic with AgentCore gateway

AgentCore gateway provides three target types: MCP targets, inference targets, and HTTP passthrough targets. The following rate limiting metrics are supported on the targets.

For this use case, assume three user groups: Basic, Advanced, and Beta. AgentCore Identity handles inbound authentication using JSON Web Tokens (JWT) with Microsoft Entra ID as the identity provider and also serves as the token vending service for outbound targets. Policy in Amazon Bedrock AgentCore enforces role-based access control (RBAC), scoping each group’s access to specific targets and models. The following diagram illustrates this configuration.

Figure 1: AgentCore gateway rate limiting architecture with user groups, identity, and policy enforcement

Basic users operate under more restrictive rate limits than Advanced users, while Beta users receive elevated limits on restricted models, enabling the organization to benchmark performance and suitability before rolling these models out to the broader organization. Before setting up rate limits for each user-group, review the rate limit structure.

A rate limit configuration consists of two parts: dimension keys and entries. Dimension keys define how the gateway groups incoming traffic into rate buckets. Entries define the allowed throughput for each bucket.

In this post, we use the AWS Command Line Interface (AWS CLI) to create the rate limit configuration. The following example demonstrates the relationship between dimension keys and entries. This rate limit uses targetName as the dimension key and defines two entries: a specific entry for the Booking target (MCP server), a high-traffic target, at 100 requests per second, and a wildcard entry that applies 10 requests per second individually to each remaining target, meaning every other target receives its own 10 RPS bucket.

Figure 2: Rate limit structure with dimension keys and entries

Dimension keys define how the gateway groups traffic into rate buckets. When a request arrives, the gateway resolves each dimension key to its value from the request context and uses the resulting combination to assign the request to the correct rate bucket. AgentCore gateway supports the following dimension keys: targetName, toolName, qualifiedModelId, $.context.jwt., $.context.iam.principal, and $.context.iam.sourceIdentity. We will explore each of these through examples in the sections that follow.

Entries are the rules within a rate limit. Each entry specifies a set of dimension keys to match, and the allowed throughput for that match. Entries support the special catch-all default value * that gives each distinct value its own independent bucket at the configured rate. When the gateway evaluates a request, it checks whether an entry matches by name before falling back to the wildcard. A named entry takes precedence because it refers to the value explicitly rather than relying on the catch-all.

Taking the preceding rate limit as an example, when a request arrives for the Booking target (MCP server), the gateway matches the first entry and allows up to 100 RPS. This entry takes precedence because the most specific value match wins over default value * as it refers to the Booking target by name. For any other target, no named entry exists, so the gateway falls back to the wildcard entry and allows up to 10 requests per second. Each target that matches the wildcard (Docs, BedrockMantle, CustomPlatform, and awsdocsagent) gets its own independent bucket.

You can combine multiple dimension keys for more granular control. For example, dimensionKeys: [“targetName”, “$.context.jwt.role”] groups traffic by both target and caller identity role claim, giving each user-group (Basic, Advanced, or Beta in the preceding example) their own independent rate bucket per target.

Types of rate limits and example configurations

AgentCore gateway enforces two layers of rate limiting: customer-defined rate limits and Service Quotas. Customer-defined rate limits are evaluated first. If the request passes, service quotas are evaluated. The following sections explain service quotas and the different types of customer-defined rate limits.

These are the limits enforced on AgentCore gateway per AWS account by the service. Service managed quotas define the ceiling that customer-defined rate limits cannot exceed. The effective rate for requests is the minimum of the customer-defined limit and the service-managed limit. You can request increases for some quotas using the Service Quotas console.

User-level limits use $.context.jwt., $.context.iam.principal, and $.context.iam.sourceIdentity as the dimension keys to control how much traffic individual users or entire user-group can consume. These limits enforce fair usage across your caller base and prevent any single caller from monopolizing gateway capacity. The following example assigns different request rates per user group. The JWT role claim is an array, so each unique combination requires its own entry.

In this configuration, Basic users receive two buckets, 100 RPM and 50 CPS, meaning every request from any Basic user counts toward the same 100 RPM total, and every connection counts toward the same 50 CPS total. If one Basic user sends 80 requests in a minute, only 20 remain for all other Basic users in that window. Advanced users receive their own two buckets at 300 RPM and 150 CPS, governed by the same collective behavior. Users with [“Advanced”, “Beta”] group membership receive two buckets at 300 RPM and 200 CPS. The higher connection allowance accommodates their streaming-heavy benchmarking workloads.

However, within a group, a single user can still consume the entire group rate bucket, throttling everyone else in that group. For example, one Basic user sending 100 requests in a minute would leave zero capacity for all other Basic users. To prevent this, we create the following rate limit configuration as well.

With this configuration, each individual user is capped at their own rate regardless of how many users exist in their group. The $.context.jwt.sub claim from the JWT uniquely identifies each user, enabling the gateway to track and enforce limits at the individual level. Even if the group-level limit allows 100 RPM total for Basic, no single user can consume more than 20 RPM and 10 CPS of that shared pool. The same logic applies to Advanced and Beta users at their respective individual caps. Together, the per-group limit and the per-user limit create a two-layer enforcement model: the group ceiling helps prevent one group from starving another, and the per-user ceiling helps prevent one individual from starving their peers within the same group.

Both rate limits are evaluated independently using AND semantics. A request must pass both the group-level limit and the per-user limit to proceed. If either check denies the request, the gateway returns a throttling response. For example, if Arnav (Basic) has consumed 20 RPM individually, his next request is denied by the per-user limit even though the Basic group still has 80 RPM of remaining capacity. Conversely, if the Basic group has collectively consumed 100 RPM, all Basic users are throttled regardless of their individual consumption.

Target-level limits use targetName, qualifiedModelId, or toolName as the dimension key to control throughput to specific downstream targets, models, or tools. These limits protect backend capacity and distribute load across your target resources. The following example limits traffic on a per-target basis.

You can also use qualifiedModelId to set connection rate limits (CPS) per model, or toolName to set request rate limits (RPS) per individual tool such as Booking___bookTool or Docs___searchDocsTool.

Note: We exclude customer-defined target-level rate limits from our use-case configuration. Beta users run heavy benchmarking workloads against restricted models, consuming a disproportionate share of a shared target-level limit. Because this limit dimensions only on targetName, all users share a single ceiling, meaning high traffic from one group or individual can starve everyone else on that target. When a subset of users is expected to dominate token or connection consumption on a specific target, scope the limit by identity instead (for example, [“targetName”, “$.context.jwt.role”]). See the following example.

Hybrid limits combine target and user dimensions in a single rate limit configuration, giving you the most granular control. Using multi-dimension keys, you can scope rate limits to a specific user or user group on a specific target, model, or tool.

The following example enforces token limits at the model level, scoped to each user within their group. The qualifiedModelId dimension is the fully qualified model identifier for inference targets. It uniquely identifies the model being invoked (see documentation).

In this configuration, anthropic.claude-fable-5 is a restricted model. Only users with the [“Advanced”, “Beta”] role can invoke it, receiving 80,000 TPM per user for benchmarking and evaluation workloads. Both [“Basic”] and [“Advanced”] users are blocked* from invoking this model with a rate of zero. The same pattern applies to other restricted models (openai.gpt-5.6-luna and openai.gpt-5.6-terra), make sure to add entries following the same structure for each. For generally available models, Basic users receive 20,000 TPM per user while Advanced users receive 40,000 TPM per user through wildcard entries.

Specific entries take precedence over wildcards following the most-specific-match-wins rule. When María (sub: “María”, role: [“Advanced”, “Beta”]) invokes anthropic.claude-fable-5, the gateway matches the explicit entry and applies 80,000 TPM scoped to María individually. If María exhausts 80,000 TPM limit, other Beta users remain unaffected because * on $.context.jwt.sub gives each user their own isolated bucket. When John (sub: “John”, role: [“Advanced”]) attempts anthropic.claude-fable-5, the gateway matches the explicit [“Advanced”] entry for that model, which sets requests to zero blocking the call. When John invokes a generally available model like anthropic.claude-sonnet-5, no explicit entry exists for that model-role combination, so the gateway falls through to the wildcard entry for [“Advanced”] and applies 40,000 TPM. When Arnav (sub: “Arnav”, role: [“Basic”]) invokes the same generally available model, he receives 40,000 TPM through the Basic wildcard entry.

You can also combine dimensions such as [“$.context.jwt.role”, “targetName”] for per-role per-target request limits, [“$.context.jwt.sub”, “targetName”] for per-user per-target combined request and token limits, or [“$.context.jwt.role”, “toolName”] for per-role per-tool request limits. For more rate limiting configurations, see Rate limit API examples.

Consider the following AgentCore gateway configuration where a user invokes the AWS Documentation Agent. The agent uses two downstream resources through the gateway: the Docs MCP target for document search and retrieval, and the BedrockMantle inference target for reasoning through the anthropic.claude-sonnet-5 model. The following architecture shows this:

Figure 3: Agentic workload rate limiting with downstream resource consumption

There are two types of rate limits to consider for agentic workloads:

The first type protects how frequently users or other services can invoke the agent. These are request (RPM) and connection (CPS) limits scoped to the agent target itself. At the simplest level, you can dimension on targetName alone, for example, {“targetName”: “awsdocsagent”}, so that all users share a single invocation ceiling. For more granular control, pair targetName with user dimensions such as [“targetName”, “$.context.jwt.role”] or [“targetName”, “$.context.jwt.role”, “$.context.jwt.sub”] to cap how often each user group or individual user can invoke the agent.

The second type protects the downstream resources the agent consumes on each invocation. AWS Documentation Agent triggers multiple downstream requests. The agent calls the Docs MCP target for document retrieval and the BedrockMantle target for inference. How you rate limit these downstream calls depends on how the agent authenticates with the gateway when invoking those resources.

If the agent performs an on-behalf-of (OBO) token exchange based on the user’s incoming JWT token, then the downstream requests carry the original user’s identity. All existing user-based rate limits apply. The per-role and per-user limits you configured previously will enforce on the agent’s downstream calls as if the user made them directly.

However, if the agent performs a machine-to-machine grant to obtain a new token (for example, a client credentials flow), the downstream requests carry the agent’s own identity rather than the caller’s. In this case, the user-based rate limits will not match the user’s claims. You should add rate limits that identify the agent itself, based on your identity provider, use a claim that uniquely identifies the agent, such as $.context.jwt.azp (authorized party), and limit accordingly to prevent a single agent from exhausting shared resources.

Follow these best practices when creating rate limits with AgentCore gateway:

In this configuration, only Arnav has an explicit entry. If John ($.context.jwt.sub: “John”) invokes the gateway, the request does not match any entry, the rate limit is effectively skipped, and John falls through to service-managed quotas with no customer-defined enforcement. Adding a wildcard catch-all entry makes sure that all callers without an explicit entry receive their own per-user rate limit bucket:

Now John, María, and any other caller each receive their own isolated 50 RPM bucket through the wildcard, while Arnav retains his explicit 100 RPM allocation. Without the catch-all entry, unmatched callers bypass the rate limit entirely.

For more best practices, see the AgentCore gateway documentation.

In this post, we walked through how to configure rate limits on Amazon Bedrock AgentCore gateway to govern AI traffic, enforcing fair usage across roles and users while helping to prevent any single caller or workload from exhausting shared capacity. We demonstrated how to layer rate limits: user-level limits for per-role and per-user fairness, target-level limits for protecting downstream service capacity, and multi-dimensional limits that combine user and target to enforce fine-grained rate limits. We also covered rate limiting considerations for agentic workloads, where downstream resource consumption varies depending on how the agent authenticates with the gateway.

Rate limits are one layer of a comprehensive traffic management strategy. Combined with AgentCore Identity for authentication, Policy in AgentCore for role-based access control, and application logging for observability, they give you the tools to operate a production-grade AI gateway with confidence, facilitating fair usage, protecting backend services, and maintaining availability as your workloads scale.

To get started with rate limits on AgentCore gateway, explore the following resources:

We continue to invest in AgentCore gateway based on customer feedback. We look forward to how you use AgentCore gateway as a central enforcement system for your AI traffic.