Agentic vision: Building visual intelligence with Amazon Bedrock and MCP servers

The integration of AI into real-world applications has long been hindered by a fundamental challenge: the disconnect between systems that can see, systems that can think, and systems that can act. Developers have struggled with complex integrations, managing multiple APIs, and creating custom solutions to bridge these gaps, resulting in inefficient, costly, and often fragile implementations.

We are converging the three key technologies: Computer Vision, Strands Agents, and the Model Context Protocol (MCP). Together, they create a pipeline where visual information can be captured, understood, and acted upon within a unified framework. This integration reduces the traditional barriers between perception, decision-making, and action, letting AI systems operate more like human intelligence by seeing, understanding, and responding in a coordinated way.

In this post, we walk you through the Computer Vision MCP Server, which illustrates this approach, representing how AI systems can process visual information and make intelligent decisions through a single, standardized interface. This convergence transforms what was once a complex integration challenge into a streamlined process, making AI capabilities accessible to a broader range of applications and developers.

In our architecture, the client interacts with multiple Amazon Web Services (AWS) through a centralized AWS Identity and Access Management (IAM) role, which serves as the security gateway for managing permissions. Amazon Simple Storage Service (Amazon S3) handles object storage to retrieve and manage data. Amazon OpenSearch provides search capabilities for querying the indexed data. Amazon Bedrock offers generative AI models, granting the client access to AI tools for tasks like text generation for the agent. Finally, Amazon Rekognition specializes in image analysis, performing functions such as object detection. The architecture emphasizes a unified security model, where the IAM role centralizes permission management, which removes the need for embedded credentials in the client and streamlines controlled access across multiple AWS services.

Computer vision, Strands Agents, and MCP servers

The solution uses three main technologies. Computer vision focuses on processing visual information such as photos and videos. Strands Agents is a framework for building AI agents that supports multiple model providers and deployment targets, offering a customizable agent loop with production capabilities including observability, tracing, and scalable deployment. Lastly, the Model Context Protocol (MCP) is a standard designed to simplify how AI systems integrate with tools and data sources, replacing the process of building separate connections for each AI model and data source pair.

The interface features a Streamlit chat UI. On the left side, there’s a menu panel where users can select their preferred foundation model for analysis, defaulted to Claude 4 Sonnet with reasoning capabilities that include both Claude 4 Sonnet and Claude 3.7 Sonnet options. Users also can reset their conversation history through a dedicated button in this sidebar.

To use this application, users can upload their visual content through the prominent Media Upload section in the center of the interface. The system accepts both images and videos, supporting a wide range of formats including PNG, JPG, JPEG, GIF, WEBP for images, and MP4, AVI, MOV, MKV, WEBM, MPEG4 for videos, with a maximum file size limit of 200 MB. Users can either drag and drop their files directly into the designated upload area or select Browse files to manually select files from their device. After the media is uploaded, the AI system can perform various analysis tasks such as object cropping, label detection, and detailed content analysis. Users can then interact with the system through the message input field at the bottom of the interface, asking specific questions about their uploaded media or requesting types of analysis.

The following is the system prompt used by the agent:

Our implementation is composed of two servers namely the CV server and the OpenSearch server. Each handles different types of requests as described in the following section.

The AWS Computer Vision MCP server provides a unified interface for image and video analysis by consolidating three Amazon AI services into a standardized protocol. This integration lets AI agents access and process visual content through a consistent API, which streamlines the development of computer vision applications. The following tools are available in the MCP server:

Agent Response: I’ll help you describe the uploaded image. Let me analyze it first using the computer vision agent. Now I’ll display the image with the description:

The uploaded image shows a rural outdoor scene featuring:

The image captures a peaceful rural scene with the sheep positioned in the center of the frame against the backdrop of green meadows. The dominant colors are various shades of green in the background (representing the grassland) and gray/white tones of the sheep in the foreground.

This is a nature photograph capturing livestock in its natural pastoral environment, showing the sheep grazing in an open field.

The following code snippet from the describe_image tool represents the overall structure of a tool:

User: Analyze the uploaded video

Agent Response: I’ll help you analyze the uploaded video. Let me delegate this task to the computer vision specialist.

l’ve analyzed your uploaded video (15dc6b31_19642406-uhd_3840_2160_30fps.mp4) and here’s what I found:

Agent Response: I’ll help you detect labels in your uploaded image. Let me analyze it using the computer vision specialist.Now that I’ve analyzed the image, let me display it with the detected labels:## Image Label Detection Results

The computer vision analysis detected the following labels in your image:

Main Labels (with confidence scores):

The image primarily shows multiple kittens or cats relaxing on a grassy lawn area.

The OpenSearch MCP server provides a unified interface for the ingestion and retrieval of images. The following tools are available in the MCP server:

Agent Response: I’ll help you find similar images to the one you provided in the “images” index. Let me analyze this image first and then search for similar ones. Based on the search results, I found one similar image in the “images index that matches your query. Let me show you both the original image and the similar one I found:l’ve found a similar image to the one you provided in the”images” index.

The similarity score between these images is 0.65 (on a scale where higher is more similar), indicating a good match based on the visual elements and context of both images. Both photos capture owls in their natural forest habitat with similar composition and subject matter.

Would you like me to search for more similar images using different keywords, or would you like more information about either of these owl images?

Now that we’ve gone through the different features of the application, we will now walk through the deployment process. To get started, make sure you have the following installed:

To deploy the solution please follow the README found in the GitHub repository. Click here to open the AWS console and follow along.

The Computer Vision MCP servers have many applications including but not limited to the ones mentioned below:

The Infrastructure-less Computer Vision Pipeline offers an approach to implementing visual analysis capabilities without the traditional complexities of setting up and maintaining extensive infrastructure. This setup lets developers perform tasks such as generating bounding boxes for detected objects, creating detailed image descriptions, and analyzing video content, all without the need for dedicated servers or complex management systems. The Inline Agent serves as the orchestrator, efficiently coordinating with the MCP Server to access a suite of computer vision tools. This streamlined approach removes the need for server management and infrastructure maintenance. It also offers the advantages of a pay-per-use model and rapid deployment. As a result, developers can quickly implement robust computer vision pipelines with minimal overhead, focusing their efforts on using the technology rather than managing its underlying infrastructure.

The Intelligent Image Cataloging system improves traditional image search by implementing embedding-based similarity algorithms that support semantic understanding of visual content. By generating and storing image embeddings in a vector database, this solution transcends the limitations of conventional keyword-based searches, allowing for context-aware image retrieval. The system uses AI models to create rich embeddings that capture the semantic essence of images, letting users find visually similar content even when exact keyword matches aren’t available. This approach creates a scalable cataloging system that understands visual relationships and context, making it particularly valuable for large image collections where traditional search methods fall short. The resulting application offers intuitive similarity searches that can identify related images based on visual characteristics, compositional elements, and semantic meaning, providing a more natural and effective way to organize and retrieve visual content.

The Visual Memory Database for Contextual Reasoning represents a fusion of computer vision and semantic understanding, combining the strengths of infrastructure-less CV pipelines with embedding-based similarity search. This system processes scenes to extract objects and their bounding boxes, generates embeddings for each element, and stores them with rich metadata including temporal and spatial information. This approach supports contextual reasoning across multiple cameras and time periods, allowing for complex queries and insights. The system excels in applications such as security monitoring, people tracking, and comprehensive scene understanding, offering temporal reasoning capabilities that can identify patterns and anomalies. It can answer nuanced questions about individuals, their associations, and object presence in specific areas, while also integrating with video monitoring agents to build a contextual visual memory. This integration facilitates features like suspicious activity detection and historical pattern analysis, creating a dynamic and intelligent visual understanding system that goes beyond simple object recognition to provide deep, context-aware insights.

To avoid ongoing charges, complete the following cleanup steps:

By integrating the powerful AI capabilities of Amazon Bedrock with standardized MCP protocols, we’ve demonstrated how modern computer vision applications can be both sophisticated and accessible. The three use cases demonstrate the practical applications of this unified approach. This integration addresses challenges in the field by reducing infrastructure complexity through serverless architectures, semantic understanding, contextual reasoning capabilities, and offering standardized interfaces that simplify development and deployment.

The combination of standardized protocols, powerful AI models, and flexible deployment options positions this solution as a foundation for the next generation of visual intelligence applications. As AI continues to evolve, the principles demonstrated here, namely standardization, accessibility, and integration, will become increasingly important in building practical, scalable solutions that bridge the gap between visual perception and intelligent action. This marks not just a technical achievement, but a fundamental shift in how we approach the development of AI-powered visual systems.

We’d love to hear how you’re using the Computer Vision MCP servers. Share your use cases and questions in the comments.