Natural language security queries at Orca Security using Amazon Bedrock

Security analysts need fast answers about their cloud infrastructure without deep familiarity with domain-specific query languages. Orca Security’s tool covers over 900 cloud models (the resource types Sonar tracks, such as an Amazon Elastic Compute Cloud (Amazon EC2) instance or an Amazon Simple Storage Service (Amazon S3) bucket). Querying that data required precise model names, nested relationship navigation, and selection from 18 operators (comparison functions like equals or contains used to filter results).

To solve this, Orca Security partnered with AWS to build Discovery AI, a natural language query agent on Amazon Bedrock. Orca provides context-aware security for cloud, code, and AI, and Sonar is its domain-specific query system where users search, filter, and explore across the entire data graph.

In this post, we share the architecture and results of Discovery AI, which converts a natural language question into a Sonar query. As a result, Orca reduced query costs by 90 percent, input tokens by 93 percent, and latency by 75 percent compared to the initial baseline.

Bridging natural language and structured queries

Orca Security’s Sonar query system supports 900+ cloud infrastructure and application models with complex relationships between them. To query this data, users construct JSON objects that specify exact model names, select from 18 operators, and navigate parent-child hierarchies across cloud providers. Doing this well requires deep familiarity with the Sonar schema.

To build this capability, the Orca team set out to build a natural language interface that can generate valid Sonar queries from plain text.

Their initial proof of concept loaded the full Sonar schema into a single prompt on the largest available model. This included over 900 model definitions, every relationship type, and a generous set of example queries. The output quality was strong (design partners and internal users responded positively) but three significant drawbacks emerged at production scale.

Token consumption was inefficient, at roughly 2 million input tokens per workload, which drove up cost. Latency was high, with end-to-end response times of around 60 seconds per workload. Accuracy on complex, multi-model queries was limited because nested relationships and multi-condition queries often required additional refinement.

The Orca team validated that the concept worked. The next challenge was making it cost-effective, fast, and accurate enough for production. To address this, the Orca team designed and deployed a solution that handles thousands of queries per day at 15-second response times using Amazon Bedrock.

Orca designed Discovery AI as a hybrid solution on Amazon Bedrock that combines Retrieval Augmented Generation (RAG) with schema-aware query construction. The system converts natural language questions into valid Discovery filters through a multi-stage pipeline, with each stage matched to the most appropriate Amazon Bedrock model.

Orca chose Amazon Bedrock because it provides multi-model access through a single API, letting the team allocate different foundation models to each pipeline stage without managing separate integrations. Amazon Bedrock also inherits AWS governance controls through AWS Identity and Access Management (IAM), virtual private cloud (VPC) endpoints, and AWS CloudTrail, meeting Orca’s security requirements. The model-agnostic architecture meant the team could swap in newer model versions as they became available without rearchitecting the pipeline.

The following diagram illustrates the nine-stage pipeline architecture.

Figure 1: Multi-stage Discovery AI pipeline architecture showing the flow from natural language input to validated Discovery filter queries using foundation models on Amazon Bedrock

The Discovery AI pipeline converts a natural language request into a validated Discovery filter through nine steps.

Maximizing relevance and diversity in example retrieval

Example retrieval emerged as the single biggest accuracy lever in the pipeline. Strong examples teach the Generator how to translate questions into Sonar queries more efficiently than schema alone, so the Orca team benchmarked several retrieval strategies on Amazon Bedrock. Each strategy traded off relevance, diversity, or speed in a different way, and comparing them directly showed which combination worked best.

Pure top-K semantic search on Amazon Titan embeddings performed quickly, but it tended to return near-duplicate examples that starved the Generator of diversity. Maximal Marginal Relevance (MMR) improved diversity, but it traded off relevance when the top candidate was actually the right answer. Retrieving a larger candidate set and then reranking it pulled in a broader pool and used a reranker to select the most relevant subset, which pointed toward the winning approach.

The winner was top-20 nearest-neighbor retrieval using Amazon Titan embeddings. Orca then reranked these results with Cohere Rerank 3.5 down to the top five. This approach gives the Generator a small, diverse, highly relevant set of examples without forcing a trade-off between coverage and precision at retrieval time.

Loading over 900 models into context

A core constraint shaped the architecture. It wasn’t feasible to include 900+ Sonar model schemas into every prompt. Even with long-context models, doing so degrades both cost and accuracy.

The Model Selector resolves this. Running on Anthropic Claude Haiku 4.5, it works against compact, semantically rich representations of each model (name, purpose, key fields, and key relationships). Given the user’s question, it returns only the models the Generator actually needs. The Generator then receives full schema detail only for those selected models plus the relationships that connect them.

To iterate confidently, the Orca team built a continuous evaluation harness early in the project. The harness combines a curated dataset, an automated scoring layer, and a slice-based measurement approach so the team can trust every change before it ships. Together these elements turn evaluation into a routine step rather than an afterthought.

The curated evaluation dataset spans the question types, model families, and complexity tiers seen in production, so it reflects real usage. An LLM-as-judge scoring layer grades generated queries on correctness, completeness, and semantic equivalence to a reference query, which gives the team a consistent quality signal. The measurement methodology then breaks deltas down by slice, allowing the team to identify exactly where regressions occur and what to correct.

Every change to a prompt, a model, a retrieval parameter, or an example flows through the harness before it ships to production.

Domain-expert-in-the-loop labeling system for continuous improvement

Sustaining accuracy as customer environments evolve and new question patterns emerge required more than a strong initial deployment. The Orca team built an internal labeling tool for product managers, support engineers, and security researchers. With the tool, they can flag queries, correct outputs, and push corrections into the example bank in one action. Furthermore, the system automatically collects production queries with low-confidence matching examples and routes them to the appropriate domain expert for daily labeling. The next pipeline run retrieves the new example through Cohere Rerank 3.5, and accuracy on that class of question improves the same moment, with no deployment.

“The labeling UI is the single most important investment we made for long-term quality. It turns Discovery AI from a static deployment into a living system, with our domain experts continuously improving it inside their normal workflow.”

— Or Gabay, Director of AI, Orca Security

Combined with end-to-end pipeline observability and shared dashboards spanning product, engineering, and customer-facing teams, this feedback loop is what keeps Discovery AI improving after initial launch.

After deploying Discovery AI to production, the new solution outperformed the initial baseline across cost, latency, and accuracy. Input token usage fell by 93 percent, from approximately 2 million tokens down to 140,000 per comparable workload. Overall cost fell by 90 percent, driven by that lower token consumption together with per-stage model allocation across Amazon Bedrock.

Latency also improved, with end-to-end response times dropping by 75 percent, from approximately 60 seconds to 15 seconds. Accuracy on complex, multi-model queries also improved, measured continuously through the team’s evaluation harness.

Discovery AI now handles thousands of queries per day in production since March 2026, with internal subject matter experts and customer feedback validating quality.

In this post, we shared how Orca Security built Discovery AI on Amazon Bedrock. The solution reduced query costs by 90 percent, input token usage by 93 percent, and latency by 75 percent compared to the initial baseline. The solution combines several key technologies. These include RAG-based example retrieval with Cohere Rerank 3.5, per-stage model allocation across Anthropic Claude Haiku 4.5 and Amazon Nova 2 Lite, and semantic search powered by Amazon Titan. Additionally, a continuous human-in-the-loop labeling system provides ongoing improvement. This combination allowed Orca to turn a working prototype into a production-grade solution that is both cheaper to run and more accurate.

To get started building similar solutions, explore Amazon Bedrock for foundation model (FM) access, Cohere Rerank on Amazon Bedrock for retrieval optimization, and Amazon Titan for embeddings. To learn more about Orca Security’s cloud security tool, visit orca.security.

To learn more about how Orca Security applies AI across its framework, see how Orca drives security outcomes with AI and how Orca secures AI adoption, including Amazon Bedrock.

For more on the techniques used in this post, read related posts on the AWS Machine Learning Blog.