Deployment Overview
Agent Kernel is a multi-cloud AI agent runtime that supports multiple deployment flavors across AWS, Azure, and GCP, from a single REST container to queue-backed, auto-scaling, WebSocket-streaming topologies.
Deployment Flavors
Execution Modes
Independently of where you deploy, execution.mode selects how requests are processed and replies delivered. Chat execution always runs on the queue pipeline: in-process with the default in_memory transport on every server flavor, over durable SQS queues on AWS Lambda/ECS, or over Kafka/NATS on-prem or Kubernetes via the Helm chart. WebSocket delivery runs on API Gateway on AWS and on the pipeline's own WebSocket gateway tier everywhere else.
| Mode | Transport | How the client gets the reply | Queues | Response store | Supported on |
|---|---|---|---|---|---|
rest_sync (also when mode is unset) | HTTP | JSON on the same connection (server awaits the store internally) | in_memory / SQS FIFO / Kafka / NATS | In-memory / DynamoDB / Redis / Valkey | All server flavors (in-process); AWS Lambda, AWS ECS (SQS); Kubernetes (Kafka/NATS/SQS) |
rest_async | HTTP | 202 ACCEPTED + request_id, client polls | in_memory / SQS FIFO / Kafka / NATS | In-memory / DynamoDB / Redis / Valkey | All server flavors (in-process); AWS Lambda, AWS ECS (SQS); Kubernetes (Kafka/NATS/SQS) |
async | WebSocket | Single CHAT_RESPONSE push when the agent finishes | Any transport | Not used | AWS Lambda, AWS ECS (API Gateway); Kubernetes and self-hosted (the pipeline's gateway tier) |
stream | SSE or WebSocket | One StreamChunk per stream event as they are generated | in_memory (SSE path) / any broker (WebSocket path) | Not used | REST API surfaces (SSE); AWS Lambda, AWS ECS (API Gateway WebSocket); Kubernetes and self-hosted (the pipeline's gateway tier) |
Surfaces mounted with explicit handlers (the thread handler, messaging integrations, custom handlers) execute inline rather than through the pipeline.
AWS ECS supports execution_mode = "stream" for WebSocket mode in both direct and queue-backed topologies: in direct mode the chat route broadcasts each chunk inline via ChatService.process_stream_chat_async; in queue mode ECSAgentRunner.run() dispatches to ECSStreamAgentRunner, which fans out one Output Queue message per chunk instead of one for the full reply. See AWS Containerized for details.
Protocol support by flavor:
| Flavor | JSON REST | SSE streaming | WebSocket (async + streaming) | Queue mode |
|---|---|---|---|---|
| Local REST API / self-hosted | ✅ | ✅ | - | ✅ in-process (in_memory) |
| AWS Lambda | ✅ | - (use WebSocket) | ✅ | ✅ SQS |
| AWS ECS Fargate | ✅ | - | ✅ (async and stream) | ✅ SQS |
| Azure Functions | ✅ | - | - | - (per-invocation, inline) |
| Azure Container Apps | ✅ | ✅ | - | ✅ in-process (in_memory) |
| GCP Cloud Run (both flavors) | ✅ | ✅ | - | ✅ in-process (in_memory) |
| Kubernetes (Helm chart) | ✅ | - | ✅ (async and stream, ws-gateway tier) | ✅ NATS / Kafka / SQS |
SSE streaming is served by the built-in FastAPI RESTAPI server, so it is available anywhere that server runs (local, ECS single-container REST, Azure Container Apps, GCP Cloud Run). AWS Lambda delivers streaming over WebSocket instead, since API Gateway REST endpoints don't support SSE responses from standard Lambda integrations. CrewAI and Smolagents don't support token streaming; use rest_sync with those frameworks.
Quick Comparison
| Flavor | Best For | Scalability | Cold Start | Cost | Fault Tolerance |
|---|---|---|---|---|---|
| Local/CLI | Development, testing | N/A | Instant | Free | Manual restart |
| REST API | Web apps, APIs | Manual scaling | Instant | Server costs | Manual |
| AWS Lambda | Variable load (AWS) | Auto-scaling | 1-3s | Pay per use | High - Auto-retry, multi-AZ, SQS retry/DLQ in queue mode |
| AWS ECS | Consistent/high load (AWS) | Auto-scaling (backlog-based in queue mode) | Instant | Running containers | Very High - Multi-AZ, auto-recovery |
| Azure Functions | Variable load (Azure) | Auto-scaling | 1-3s | Pay per use | High - Auto-retry, multi-region |
| Azure Container Apps | Consistent load (Azure) | Auto-scaling (KEDA) | Instant | Running containers | Very High - Multi-zone, auto-recovery |
| GCP Cloud Run Serverless | Variable load (GCP) | Auto-scaling (scale-to-zero) | 1-3s | Pay per use | High - Auto-retry, multi-zone |
| GCP Cloud Run Containerized | Consistent load (GCP) | Auto-scaling (min≥1) | Instant | Running containers | Very High - Always-on, auto-recovery |
| Kubernetes (Helm) | On-prem, any cluster, EKS | Auto-scaling (KEDA queue depth) | Instant | Cluster capacity | Very High - Broker-backed retry/DLQ, rolling deploys, graceful drain |
| MCP Server | AI integrations | Manual | Instant | Server costs | Manual |
| A2A Server | Agent networks | Manual | Instant | Server costs | Manual |
GCP "serverless" and "containerized" are both Cloud Run: the difference is min_instance_count = 0 (scale-to-zero) vs ≥ 1 (always-on), not a different compute product.
Scalable Queue Topologies
Queue mode decouples request ingestion from agent execution. The same five-component pipeline runs everywhere: in-process on any server flavor (the in_memory default), split across compute over SQS FIFO queues on AWS, or over Kafka/NATS JetStream on-prem or Kubernetes via the Helm chart (baremetal + EKS flavors).
| Role | Local / any server flavor (in-process) | AWS Lambda (serverless) | AWS ECS (containerized) | Kubernetes (Helm chart) |
|---|---|---|---|---|
| Request handler | RequestHandler on the rest-api thread | Request Handler Lambda | ECSQueueRequestHandler thread in the IO container | RequestHandler thread in the io-handler Deployment |
| Agent runner | AgentRunner worker threads (same process) | Agent Runner Lambda (SQS event source mapping) | ECSAgentRunner service, a pool of long-poll consumer threads | AgentRunner consumer threads in the agent-runner Deployment |
| Response handler | ResponseHandler worker thread (same process) | Response Handler Lambda | ECSOutputConsumer thread pool in the IO container | ResponseHandler thread pool in the io-handler Deployment |
| Reply delivery | Response store (in-memory), or SSE bridging (stream) | Response store, or WebSocket push (async/stream) | Response store, or WebSocket push (async/stream) | Response store (Valkey/Redis), or ws-gateway push (async/stream) |
| Scaling | no_of_consumers threads | Automatic per SQS batch | Backlog-per-task target tracking | KEDA on queue depth (Kafka lag / NATS pending / SQS length) |
See AWS Serverless, AWS Containerized, On-Prem / Kubernetes, and the Queue Mode Guide for full component walkthroughs.
Getting Started per Flavor
Local Development
Uses the agentkernel.cli module.
python my_agent.py
- Interactive CLI, instant feedback, no deployment needed
REST API Server
Uses the agentkernel.api.RESTAPI module.
python my_agent.py
- HTTP + SSE endpoints, easy integration, self-hosted
AWS Serverless
Uses Agent Kernel Terraform modules.
terraform init && terraform apply
- Lambda functions, API Gateway (REST + WebSocket)
- Optional SQS queue mode with response store
- Token streaming over WebSocket
- Auto-scaling, pay per request
AWS Containerized
Uses Agent Kernel Terraform modules.
terraform init && terraform apply
- ECS Fargate + Application Load Balancer
- Optional two-container scalable queue mode with backlog-based auto-scaling
- Optional WebSocket mode (
async/stream) for real-time, connection-based interactions - Consistent performance, lower latency
Azure Serverless
Uses Agent Kernel Terraform modules.
terraform init && terraform apply
- Azure Functions (Flex Consumption) + API Management
- Auto-scaling, pay per request
Azure Containerized
Uses Agent Kernel Terraform modules.
terraform init && terraform apply
- Azure Container Apps + API Management
- SSE streaming supported (runs the built-in REST server)
GCP Serverless
Uses Agent Kernel Terraform modules.
terraform init && terraform apply
- Cloud Run (scale-to-zero) + API Gateway
- SSE streaming supported, pay per request
GCP Containerized
Uses Agent Kernel Terraform modules.
terraform init && terraform apply
- Cloud Run (always-on,
min_instance_count ≥ 1) + API Gateway - SSE streaming supported, no cold starts
On-Prem / Kubernetes
Uses the Agent Kernel Helm chart.
helm install ak ak-deployment/ak-k8s/chart -f values-<flavor>.yaml
- io-handler + agent-runner Deployments over Kafka/NATS JetStream (or SQS on EKS)
- Optional WebSocket gateway tier (
async/stream), KEDA queue-depth autoscaling - Baremetal, EKS, and dev/micro-cluster flavors as values files
Choosing a Deployment Mode
- Development → Local/CLI: fast iteration, no setup
- Small web app → REST API: simple, self-hosted
- Variable traffic on AWS → AWS Lambda: auto-scales, pay per use; add queue mode for backpressure and retries
- High traffic / long-running agents on AWS → AWS ECS in queue mode: consistent performance, backlog-based auto-scaling
- Real-time UX on AWS → WebSocket mode:
asyncfor push delivery,streamfor token streaming: both on Lambda or ECS - Variable traffic on Azure → Azure Functions; high traffic → Azure Container Apps (KEDA scaling, SSE streaming)
- Variable traffic on GCP → Cloud Run scale-to-zero; high traffic → Cloud Run always-on
- On-prem, existing Kubernetes, or cloud-portable → the Helm chart: NATS-backed queue mode with KEDA autoscaling; Kafka where the organization standardizes on it
- AI integration → MCP/A2A: protocol-based integration
Multi-Cloud Strategy
Agent Kernel's multi-cloud support enables you to:
- Deploy the same agent code to AWS, Azure, or GCP without modification
- Avoid vendor lock-in: switch clouds or run on multiple clouds
- Optimize costs: choose the best pricing model for each workload
- Geographic redundancy: distribute across cloud providers
- Leverage cloud-specific services: use the best of each platform
Fault Tolerance Considerations
Agent Kernel provides different levels of fault tolerance depending on your deployment mode:
Production-Grade Fault Tolerance
AWS ECS/Fargate offers the highest level of fault tolerance on AWS:
- Multi-AZ task distribution for zone-level failures
- Automatic task replacement on failures; graceful in-container thread shutdown (
ThreadRunner) so a crashed consumer restarts the whole task cleanly - In queue mode: SQS visibility-timeout retries, optional dead-letter queues, and error responses written to the response store so clients never hang
- Backlog-based auto-scaling of the agent-runner service
- Rolling deployments with zero downtime behind an ALB
Learn more about AWS ECS fault tolerance →
AWS Lambda provides built-in fault tolerance:
- Serverless architecture with automatic scaling, multi-AZ execution by default
- In queue mode: partial-batch failure reporting (
batchItemFailures), visibility-timeout retries, optional DLQs - Automatic retry on failures, no infrastructure management
Learn more about AWS serverless fault tolerance →
Azure Container Apps offers the highest level of fault tolerance on Azure:
- Multi-zone replica distribution, automatic replica replacement
- Health check-based routing, KEDA-based auto-scaling
- Rolling deployments with zero downtime
Learn more about Azure Container Apps fault tolerance →
Azure Functions provides built-in serverless fault tolerance with automatic retry and scaling.
Learn more about Azure serverless fault tolerance →
GCP Cloud Run (both flavors) provides automatic scaling, multi-zone execution, automatic retries, and no infrastructure management; the containerized flavor adds always-on instances for consistent performance.
GCP serverless → · GCP containerized →
State Persistence
All production deployment modes support resilient state management:
AWS Options:
- DynamoDB: Multi-AZ replication, automatic backups, 99.999% SLA
- ElastiCache Redis / Valkey: Cluster mode with automatic failover, replication
Azure Options:
- Cosmos DB: Multi-region replication, automatic backups, 99.999% SLA
- Azure Cache for Redis: Cluster mode with automatic failover, replication
GCP Options:
- Firestore: Multi-region replication, automatic backups, 99.999% SLA
- Memorystore Redis: High availability with automatic failover
Learn more about fault tolerance →
Next Steps
- Local Deployment
- AWS Deployments:
- Azure Deployments:
- GCP Deployments:
- On-Prem / Kubernetes
- Queue Mode Guide
- Fault Tolerance
- Configuration
