Architecture Overview

Understanding Agent Kernel's architecture helps you build robust, scalable AI agent systems.

High-Level Architecture

Agent Kernel sits between your agent logic (written in any supported framework) and the surfaces that expose it to the world: CLI, REST/SSE, WebSocket, MCP, A2A, messaging platforms, and cloud deployment targets.

Layers at a glance:

Layer	Components	Responsibility
Application	Your agents and tools	Domain logic, written with any supported framework
Core	Module, Agent, Runner, Session, Runtime, hooks, AgentService (agent/session lifecycle for stateful clients), ChatService (execution core execute/execute_stream plus HTTP presentation wrappers process_*)	Framework-agnostic orchestration, state, and the run/stream pipeline
Framework adapters	OpenAI Agents SDK, CrewAI, LangGraph, Google ADK, Smolagents	Wrap native agents behind the core abstractions
System plugins	Guardrails, multimodal, conversation threads, knowledge bases, tracing	Cross-cutting features implemented as hooks, tools, and services
State stores	In-memory, Redis, Valkey, DynamoDB, Cosmos DB, Firestore	Pluggable persistence for sessions, threads, attachments, and responses
Execution surfaces	CLI, REST (JSON + SSE), WebSocket, MCP, A2A, messaging integrations, cloud deployments	How requests reach the runtime and how replies get back out

ChatService vs AgentService

Two services sit between the execution surfaces and the Runtime, and they solve different problems. AgentService is a stateful conversation object: it holds one selected agent and one session, and the caller drives its lifecycle. ChatService is a stateless chat-request processor: every call carries the full request envelope, and agent/session resolution happens fresh per request.

	AgentService	ChatService
Statefulness	Stateful: holds the selected agent and session across calls (select(), new(), clear(), load())	Stateless: a fresh agent/session resolution on every call
Input	A prompt string (run) or an AgentRequest list (run_multi/stream_multi)	A BaseChatRequest envelope (prompt, agent, session_id, user_id, ...), optionally with a prebuilt AgentRequest list
Validation and building	None: the caller prepares everything	Validates the envelope; builds the request list from the payload, or accepts a prebuilt one
Output	Typed AgentReply / raw StreamChunks	Execution core: typed reply plus session id. Presentation wrappers: JSON dicts, SSE frames, HTTPException
Error handling	Exceptions propagate	Core: exceptions propagate. Wrappers: ValueError maps to 400, anything else to 500
Callers today	CLI, A2A, MCP	REST handler and deployment adapters (wrappers); messaging integrations and the thread handler (core)

Use ChatService when handling chat traffic where each request arrives self-contained with its session_id: the presentation wrappers (process_*) if you want the standard HTTP shapes, or the execution core (execute/execute_stream) if your surface owns its own transport, reply formatting, and error UX.

Use AgentService when building an interactive or stateful client that owns a running conversation: selecting agents, reusing one session across turns, clearing or recreating it. The CLI's !select / !new / !clear commands are the canonical example.

They are layers, not alternatives: the ChatService core drives AgentService internally for agent selection and session loading, so going through ChatService never bypasses AgentService semantics. And neither layer should be skipped: entry surfaces never call Runtime directly, and behavior that must apply to every run regardless of surface belongs in a Runtime pre/post hook. See the execution flow for the per-surface layering diagram and call rubric.

Key Design Principles

1. Framework Agnostic

All core abstractions (Session, Agent, Runner, Module, Runtime) are framework-independent. Framework-specific logic lives exclusively in adapter modules; the same hooks, tools, session stores, and deployment targets work with every supported framework, and agents from different frameworks can run side by side in one runtime.

2. Minimal Overhead

The kernel adds minimal latency and complexity; it's primarily orchestration and state management around your framework's native execution.

3. Config-Driven Behavior

All runtime behavior is governed by AKConfig (Pydantic-based), loaded from YAML/JSON files and environment variables (AK_ prefix, __ for nesting). The same application code switches between synchronous REST, SSE streaming, queue-backed async, and WebSocket delivery purely through configuration.

4. Production Ready

Built-in support for:

• Multi-cloud session persistence (AWS, Azure, GCP)
• Event streaming (SSE over REST, WebSocket on AWS serverless)
• Queue-pipeline execution everywhere: in-process by default, SQS-backed on Lambda and ECS, Kafka and NATS JetStream for on-prem / Kubernetes (deployed by the Helm chart)
• Input/output guardrails and PII redaction
• Multi-agent coordination and multimodal attachments
• Observability and tracing (Langfuse, OpenLLMetry, Logfire)

5. Extensible

Pluggable via well-defined interfaces:

• New framework adapters (Agent/Runner/Module/ToolBuilder subclasses)
• Custom session, thread, and attachment storage backends
• Custom guardrail and tracing providers
• Pre/post execution hooks
• Knowledge base backends

The Run Pipeline

Runtime.run() is the heart of every execution surface. It wraps the framework call with session locking, hooks, and persistence:

Key properties:

• Session locking: async with session serializes concurrent requests for the same session and sets the session as the current context (Session.current() works anywhere inside the run).
• Pre-hooks can rewrite the request list or halt execution by returning an AgentReply (this is how input guardrails block a request before the LLM sees it).
• Post-hooks can transform the reply (output guardrails, disclaimers, redaction).
• Persistence and cleanup always run: the session is stored and the volatile cache cleared even if the run raises.

The Streaming Pipeline

Runtime.stream() is the streaming counterpart, sharing the same pre-hook pipeline but yielding StreamChunk objects carrying typed StreamEvents as they arrive:

• Each StreamChunk carries delta, event, done, error, and session_id fields. delta is populated only when event is a TextDelta; every other event type (message/step boundaries, tool calls, reasoning) is carried in event alone.
• Post-hooks see every event via on_stream_event() — tool arguments and results included — and can rewrite one in place, return None to drop it, or return a list to emit several in its place. Raising StreamHalt ends the run.
• Delivery depends on the surface: the REST API serves chunks as Server-Sent Events (text/event-stream); AWS serverless and AWS ECS containerized WebSocket modes push each chunk as a separate STREAM_CHUNK WebSocket message (optionally through SQS queues).
• OpenAI Agents SDK, LangGraph, and Google ADK stream natively; CrewAI and Smolagents declare supports_streaming = False and do not support token streaming (their runners raise NotImplementedError in stream mode).

See Execution Flow for the full request lifecycle including the queue-based and WebSocket paths.

Scalability: The Queue Execution Pipeline

Two facts shape how Agent Kernel is built to grow with demand:

1. Accepting a request and carrying it out are not the same kind of work. Receiving a message from a user is nearly instant. Actually producing a response can involve several rounds of reasoning, tool use, or lookups in outside systems, and can take anywhere from under a second to several minutes depending on what's asked. Treating these two things as one inseparable unit means one is always sized for the other's load: either paying for idle capacity, or getting stuck behind the workload that takes longer.
2. Real demand doesn't arrive at a steady pace. Usage comes in bursts, quiet periods, and peaks around business hours, campaigns, or events. A system built only for the average load either falls over during a peak or stays over-provisioned the rest of the time.

Agent Kernel's answer is to keep "receiving a request" and "producing the response" as two separate jobs, with a safe holding area in between. Incoming work lands in that holding area first and is then picked up and completed as capacity allows, rather than being handled the instant it arrives. This one design choice provides several guarantees at once:

• The two jobs scale independently. The part that talks to users and the part that does the underlying thinking can each be given more or less capacity on their own, based on what's actually under pressure, instead of scaling both together.
• Traffic spikes are absorbed, not dropped. A sudden surge in requests lengthens the queue of pending work rather than overwhelming the system or the outside services it depends on (such as the AI models themselves).
• Conversations stay in the right order. Messages belonging to the same conversation are always completed in the order they were sent, while unrelated conversations are free to proceed fully in parallel.
• Nothing is lost, and nothing is done twice. If a piece of work is interrupted partway through (for example, by a temporary outage), it is safely retried until it succeeds, up to a sensible limit, and it is never accidentally completed more than once.
• Capacity can shrink as well as grow. When demand drops, processing capacity can scale back down — in some deployments all the way to zero — so cost tracks actual usage rather than peak provisioning.
• This behavior is consistent everywhere Agent Kernel runs. The same guarantees apply whether Agent Kernel is deployed as a single small service, across a large-scale cloud environment, or on an organization's own private infrastructure. Moving between these is a deployment and configuration decision — it never requires changing how an agent is built or behaves.

How it works under the hood

Chat execution is built on one logical pipeline (#495): every chat request travels five components, with the queue transport and the process topology selected purely by configuration.

The queue transport is a pluggable backend (execution.queues.type):

Transport	Status	Durability	Typical use
in_memory	✅ the default	In-process only	Local development, single-container deployments: full queue semantics (per-session FIFO, bounded retry, deduplication) with zero backing services
SQS	✅ on AWS Lambda and ECS (via the deployment adapters)	Durable, FIFO	Production on AWS
kafka, nats	✅ (kafka/nats extras)	Durable	Production on-prem / Kubernetes

One pipeline, three topologies. The logical components map onto processes per deployment:

• Single-process: a bare RESTAPI.run() boots all five components as threads in one process: this is what local REST and single-container cloud deployments run. Sessions are processed in order and in parallel across worker threads, failed messages retry up to max_receive_count, and duplicates are dropped: the same semantics as the broker transports, minus durability.
• Two-process: the IO process (request handler + response handler) and the agent-runner process scale independently over a durable broker. Today this is AWS ECS Fargate with SQS; see AWS Containerized.
• Three-way: on AWS Lambda the three roles are three functions wired by SQS event source mappings; see AWS Serverless.

The activation rule: a bare RESTAPI.run() (no explicit handlers) runs the pipeline; surfaces constructed with explicit handlers (the thread handler, messaging integrations, custom handlers) and the AgentService clients (CLI, A2A, MCP) keep their direct execution paths. The client receives the reply either by polling the response store (rest_sync waits server-side, rest_async polls with a request_id), by SSE (stream on the REST surface), or by WebSocket push (async/stream on AWS today).

This decouples request ingestion from agent execution so each scales, fails, and recovers on its own terms.

Next Steps

• Execution Flow: request lifecycle across all execution modes
• Session Management: detailed session configuration
• Memory Management: advanced memory features
• Knowledge Bases: knowledge backends and KB routing
• Deployment Overview: choosing a deployment mode