Multimodal Attachments

Agent Kernel supports multimodal input processing: users can send images and files alongside text, and the framework automatically handles description generation, storage, and context injection.

Overview

Key Design Decisions

• No raw binary in session history: Images/files are stored externally; only text descriptions enter the conversation. This prevents session bloat.
• Pluggable storage: Choose between in-memory, Redis, or DynamoDB depending on your deployment.
• Automatic description: A vision-capable LLM generates brief descriptions of each attachment.
• System tool for recall: The agent can call analyze_attachments to retrieve previously stored images/files.

Supported Attachment Source Forms

AgentRequestImage.image_data and AgentRequestFile.file_data accept a source string in any of these forms:

Source form	Example	Handling
Bare base64	iVBORw0KGgo...	Described by the vision LLM and saved to storage
data:<mime>;base64,<payload>	data:image/png;base64,iVBORw0KGgo...	Described and saved to storage; the mime type comes from the URI itself, not the request's mime_type
data:<mime>,<payload> (no ;base64 marker)	data:text/plain,hello%20world	Left on the request undescribed and unsaved — its bytes are not base64, so decoding them would store the wrong thing
http:// / https:// URL	https://example.com/cat.png	Left on the request undescribed and unsaved — the pre-hook never fetches remote content, to avoid network I/O and SSRF exposure in a system hook that runs on every request
s3:// reference	s3://bucket/key.png	Left on the request undescribed and unsaved, same as other remote references

A data: URI with no payload after the comma (data:image/png;base64,) carries no bytes and is dropped, the same as an empty image_data/file_data. Scheme and data: header matching is case-insensitive.

Attachments that are left on the request list are not removed by MultimodalPreHook — they reach the agent/adapter as-is so it can resolve them itself (for example, framework/openai/openai.py already accepts all five forms natively).

With Conversation Thread Support enabled, ConversationThreadManager.store_attachments classifies through the same AttachmentSource, so every form above behaves identically. It adds one thing: each attachment gets a store record and an attachment_id so the thread's history is uniform. Base64 is saved as bytes; a remote reference is saved as a url with no bytes, and its request still travels on for the adapter to resolve.

Enabling Multimodal Support

Environment Variables

export AK_MULTIMODAL__ENABLED=true

Configuration File

multimodal:
  enabled: true
  max_attachments: 10                 # Max attachments per session
  description_max_length: 200         # Max chars for auto-generated descriptions
  storage_type: in_memory             # Default - no session bloat

Attachment Source Forms

AgentRequestImage.image_data and AgentRequestFile.file_data accept several source forms, and what MultimodalPreHook does with each differs. Only inline bytes can be described by a vision model or stored — a URL is a reference to something the hook never reads.

Source form	Example	Described	Stored	Reaches the agent
Bare base64	iVBORw0KGgo...	✅	✅	As a description, plus an attachment_id
data: URI, base64	data:image/png;base64,iVBOR...	✅	✅	As above. The URI's own media type wins over mime_type
data: URI, no base64 marker	data:text/plain,hello	❌	❌	Passed through untouched
http:// / https:// URL	https://cdn.example.com/a.png	❌	❌	Passed through untouched
s3:// URL	s3://bucket/key.pdf	❌	❌	Passed through untouched

"Passed through untouched" is the important row. The hook neither describes nor stores these, and — as of the source-form work — it no longer strips them either: the request travels on to the framework adapter with the URL intact, so an adapter whose provider can fetch it gets the chance to. Previously they were consumed and dropped, so the agent saw nothing at all.

Agent Kernel never fetches a remote attachment. The request travels on to the framework adapter, which hands the address to the model provider — so whether it resolves is a question for the app's configured model, not for Agent Kernel. If that model cannot fetch the address, the app should send the attachment as base64 instead — it is stored and described locally, so it does not depend on the provider fetching anything.

A data: URI with an empty payload (data:image/png;base64,) is dropped rather than forwarded, since there are no bytes to describe and nothing for an adapter to fetch. Scheme and media-type matching is case-insensitive.

The Described and Reaches the agent columns hold with Conversation Thread Support enabled too — ConversationThreadManager.store_attachments calls the same classifier. Stored is where thread mode differs: every attachment gets a store record and an attachment_id, so a thread's message history is uniform. Base64 is stored as bytes; a remote reference is stored as a url with no bytes, and the request still passes through untouched for the adapter to resolve.

Attachment Storage

Attachments are stored outside the session to prevent session bloat. The storage backend is independent of your session storage; you can use Redis sessions with in-memory attachment storage, or vice versa.

In-Memory (Default)

Fast, ephemeral storage. Attachments live in a module-level dictionary, not inside the session object.

export AK_MULTIMODAL__STORAGE_TYPE=in_memory

Trait	Value
Session bloat	❌ None
Persistence	❌ Lost on restart
Setup	✅ None required
Best for	Development, testing

Redis

Persistent storage for production. Requires a Redis server.

export AK_MULTIMODAL__STORAGE_TYPE=redis
export AK_MULTIMODAL__REDIS__URL=redis://localhost:6379
export AK_MULTIMODAL__REDIS__PREFIX=ak:attachments:
export AK_MULTIMODAL__REDIS__TTL=3600

Trait	Value
Session bloat	❌ None
Persistence	✅ Across restarts
Setup	🔧 Redis server
Best for	Containerized production

DynamoDB

Serverless storage for AWS deployments.

export AK_MULTIMODAL__STORAGE_TYPE=dynamodb
export AK_MULTIMODAL__DYNAMODB__TABLE_NAME=ak-attachments
export AK_MULTIMODAL__DYNAMODB__REGION=us-east-1
export AK_MULTIMODAL__DYNAMODB__TTL=3600

Trait	Value
Session bloat	❌ None
Persistence	✅ Fully managed
Setup	🔧 AWS account + table
Best for	AWS Lambda deployments

Session Cache (Legacy)

warning

This stores attachments inside the session object, causing session size to grow with each attachment. Use only for backward compatibility.

export AK_MULTIMODAL__STORAGE_TYPE=session_cache

The analyze_attachments System Tool

When multimodal is enabled, a system tool called analyze_attachments is automatically registered on all agents. This allows the agent to retrieve and re-analyze previously stored attachments.

# attachment_ids usually come from the multimodal storage layer
analyze_attachments(
    attachment_ids=["att_123", "att_456"],
    prompt="What breed is the dog?",
)

The tool:

1. Takes a list of attachment IDs (returned when attachments are stored by the multimodal pre-hook or storage backend; see the attachment_id in the sequence diagram above)
2. Fetches those attachments from storage
3. Sends them (with the prompt) to the vision LLM
4. Returns a detailed analysis

This enables multi-turn conversations about images:

User: [sends photo of a dog]
Agent: I see a golden retriever sitting in a park.

User: What breed is it exactly?
Agent: [calls analyze_attachments] It's a Golden Retriever, approximately 2-3 years old...

Configuration Reference

Full config.yaml Example

multimodal:
  enabled: true
  max_attachments: 10
  description_max_length: 200
  storage_type: in_memory          # in_memory | redis | dynamodb | session_cache

  redis:
    url: "redis://localhost:6379"
    prefix: "ak:attachments:"
    ttl: 3600

  dynamodb:
    table_name: "ak-attachments"
    region: "us-east-1"
    ttl: 3600

Environment Variables

# Core
export AK_MULTIMODAL__ENABLED=true
export AK_MULTIMODAL__MAX_ATTACHMENTS=10
export AK_MULTIMODAL__DESCRIPTION_MAX_LENGTH=200
export AK_MULTIMODAL__STORAGE_TYPE=in_memory

# Redis storage
export AK_MULTIMODAL__REDIS__URL=redis://localhost:6379
export AK_MULTIMODAL__REDIS__PREFIX=ak:attachments:
export AK_MULTIMODAL__REDIS__TTL=3600

# DynamoDB storage
export AK_MULTIMODAL__DYNAMODB__TABLE_NAME=ak-attachments
export AK_MULTIMODAL__DYNAMODB__REGION=us-east-1
export AK_MULTIMODAL__DYNAMODB__TTL=3600

Storage Backend Comparison

Feature	In-Memory	Redis	DynamoDB	Session Cache
Session Bloat	❌ None	❌ None	❌ None	⚠️ Yes
Persistence	❌ Lost on restart	✅ Persistent	✅ Persistent	✅ With session
Multi-Process	❌ Single process	✅ Distributed	✅ Distributed	Depends on session
Setup	✅ None	🔧 Redis server	🔧 AWS account	✅ None
Best For	Development	Production	Serverless	Legacy only

Supported Integrations

Multimodal attachments are supported on the following platforms:

Platform	Images	Files	Notes
Telegram	✅	✅	Photos + documents
Teams	✅	✅	Inline images + uploaded files; audio/video rejected, api.max_file_size enforced while streaming
REST API	✅	✅	Via AgentRequestImage / AgentRequestFile
CLI	❌	❌	Text only

Related Documentation

• Session Management: Session storage and caching
• Execution Hooks: How PreHooks and PostHooks work
• Configuration: Complete configuration reference
• Telegram Integration: Telegram-specific file handling
• Teams Integration: Teams-specific attachment handling