Skip to main content
Version: 0.10.1

LiveKit Voice Integration

LiveKit is a realtime voice integration for OpenAI Agents SDK and Google ADK agents. The LiveKitEdgeGateway joins a LiveKit room, sends the user's audio and chat text to the agent, and plays the model's streamed audio and transcript back to the room through Open Agent Kernel's realtime execution pipeline.

Overview​

The gateway runs in realtime execution mode. It works on every queue transport:

  • in_memory (single process): the gateway, the Agent Runner and the Response Handler run in one process. This is the simplest setup and the recommended first run.
  • A broker (kafka, nats, sqs): the gateway (IOHandler process) and the Agent Runner run as two processes that share the queues.

Only OpenAI Agents SDK and Google ADK agents can run in realtime mode. They are the only adapters with a realtime runner (OpenAIRealtimeRunner, GoogleADKRealtimeRunner). Use a native RealtimeAgent for OpenAI, or a live ADK LlmAgent for Gemini.

How It Works​

  1. A user joins the LiveKit room and starts speaking.
  2. The LiveKitEdgeGateway batches the microphone audio (see input_batch_ms) and puts it on the input queue. Chat text from the room's data channel goes the same way.
  3. The Agent Runner hands each chunk to the RealtimeConnectionPool, which keeps one native SDK live session per OAK session.
  4. The model's audio and transcript are paced at playback speed and put on the output queue.
  5. The Response Handler passes each chunk to the gateway, which plays the audio into the room and publishes the transcript as a chat message when the turn ends.

For the architecture behind these steps, including multi-agent handoffs, see Realtime Voice Internals.

Features​

  • Realtime voice: two-way streaming audio between the room and the model.
  • Barge-in: when the user talks over the model, the model's response is cancelled and the gateway clears the audio it has buffered.
  • Tools: the tools declared on the native agent run through the OpenAI Agents SDK or Google ADK, with the OAK context and system tools available as in other modes.
  • Multi-agent handoffs: a supervisor hands the call to a specialist and back with the framework's own handoffs, and every agent appears in the room as its own participant. See Multi-agent handoffs.

Configuration Steps​

1. LiveKit Cloud Setup​

  1. Create a free account at LiveKit Cloud.
  2. Create a new project.
  3. Obtain your server URL, API key, and API secret from the project settings.

2. Required Environment Variables​

export AK_LIVEKIT__URL="wss://<your-project>.livekit.cloud"
export AK_LIVEKIT__API_KEY="your_api_key"
export AK_LIVEKIT__API_SECRET="your_api_secret"

Secret providers: these credentials can also live in your configured secret provider (AWS SSM Parameter Store, mounted Kubernetes Secrets) instead of the environment; see Framework-owned credentials.

Implementation​

The gateway is hosted with GatewayRunner and passed to IOHandler.run(gateways=...). One gateway serves one room, and its session_id is the room name.

Single process (in_memory)​

# config.yaml
execution:
mode: realtime
from agents.realtime import RealtimeAgent

from agentkernel.framework.openai import OpenAIModule
from agentkernel.integration.adapter import GatewayRunner
from agentkernel.integration.livekit import LiveKitEdgeGateway
from agentkernel.pipeline import IOHandler

agent = RealtimeAgent(
name="voice_assistant",
instructions="You are a helpful voice assistant.",
)
OpenAIModule([agent]).run_options(agent, model_config={"initial_model_settings": {"model_name": "gpt-realtime", "voice": "alloy"}})

if __name__ == "__main__":
IOHandler.run(gateways=[GatewayRunner(LiveKitEdgeGateway(session_id="room_01"))])

Two processes (broker transport)​

With execution.queues.type set to kafka, nats or sqs, start the same module in two roles: the IOHandler with the gateway, and the Agent Runner.

from agentkernel.integration.adapter import GatewayRunner
from agentkernel.integration.livekit import LiveKitEdgeGateway
from agentkernel.pipeline import AgentRunner, IOHandler

# Process 1: the gateway and the Response Handler
IOHandler.run(gateways=[GatewayRunner(LiveKitEdgeGateway(session_id="room_01"))])

# Process 2: the Agent Runner, which holds native live sessions
AgentRunner.run()

Run the Agent Runner as a single replica: its connection pool is local to the process. Run one IOHandler process that hosts the gateway for every room, because replies are routed to the gateway registered in that process.

Execution Mode

LiveKit requires the OAK execution mode to be set to realtime. Standard async/sync text modes will not work. Ensure you have execution.mode: realtime set in your config.yaml.

Realtime pacing and audio batching​

Two execution.realtime knobs tune the audio pipeline:

  • playback_lead_ms (default 50): outbound model audio is paced so the edge holds this much audio ahead of playback. Low-latency transports (in-process, Kafka, NATS) are fine at the default; a store-and-forward broker such as SQS adds latency and jitter per chunk, so raise it to keep playback from underrunning (heard as choppy or missing audio).
  • input_batch_ms (default 100): inbound mic audio is accumulated to this many milliseconds per input-queue message. Batching keeps message count, broker cost, and per-message overhead low. Set it to 0 to pass every frame straight through for the lowest input latency (many more messages, so only sensible on a low-cost transport).
execution:
mode: realtime
realtime:
playback_lead_ms: 250
input_batch_ms: 100

Both are settable from the environment too (AK_EXECUTION__REALTIME__PLAYBACK_LEAD_MS, AK_EXECUTION__REALTIME__INPUT_BATCH_MS). A larger playback lead is smoother but adds a little latency at the start of each response.

Tools and framework context​

Declare realtime tools on the native RealtimeAgent or ADK LlmAgent. The SDK calls them and returns their results. Inside a tool, ToolContext.get(), Session.current() and Agent.current() work as they do in other execution modes: the pool sets them once when the call opens, and the SDK's own tasks inherit them. The session lock is not held while a tool runs, because a call is one long-lived session. OAK system tools (multimodal, sandbox, schedule, knowledge base) and their prompt guidance are attached as in other modes. An ADK Workflow root holds no tools of its own, so they attach to its speaking agents, each looked up by its own name for the agents allow-lists. A realtime session does not carry the OAK session's framework_context: the native session lives for the whole call, and the OAK session is not written back when it ends.

Multi-agent handoffs​

One call can move between several agents. Declare native live agents and register the entry agent:

  • OpenAI Agents SDK: RealtimeAgent(handoffs=[...]). The SDK executes each handoff and updates the active agent.
  • Google ADK: LlmAgent(sub_agents=[...]). ADK's own rules decide who can transfer to whom: an agent to its sub-agents, and back to its parent or across to its peers unless disallow_transfer_to_parent / disallow_transfer_to_peers is set.
from agents.realtime import RealtimeAgent

from agentkernel.framework.openai import OpenAIModule
from agentkernel.integration.adapter import GatewayRunner
from agentkernel.integration.livekit import LiveKitEdgeGateway
from agentkernel.pipeline import IOHandler

supervisor = RealtimeAgent(name="supervisor", instructions="Route the caller to billing or tech support.")
billing = RealtimeAgent(name="billing", instructions="Answer billing questions.", handoffs=[supervisor])
tech_support = RealtimeAgent(name="tech_support", instructions="Fix connection problems.", handoffs=[supervisor])
supervisor.handoffs = [billing, tech_support]

OpenAIModule([supervisor]).run_options(supervisor, model_config={"initial_model_settings": {"model_name": "gpt-realtime", "voice": "alloy"}})

if __name__ == "__main__":
IOHandler.run(gateways=[GatewayRunner(LiveKitEdgeGateway(session_id="room_01", agent_name="supervisor"))])

The conversation, and the context the caller has given, carries over to the agent that takes the call.

What the caller sees in the room​

Every agent the entry agent can reach through handoffs appears as a participant of its own (agent-supervisor, agent-billing, agent-tech_support), with its own audio track. There is nothing to configure: when the call starts, the conversation tells the gateway its whole team, and the other agents join the room about a second later.

  • Only the agent that has the call is heard and unmuted; the others stay connected and muted.
  • On a handoff, the previous agent finishes what it was saying before the next agent's participant takes over, so the room client's speaking indicator moves to it.
  • Each agent's words are published to the room chat from its own participant.
  • Each agent participant carries an ak.active attribute, "true" on the one that has the call, for a custom client UI to highlight.
  • Only the first participant listens to the caller; no agent's audio or chat is ever taken as the caller's input.

Each agent participant needs a token of its own, so the gateway needs AK_LIVEKIT__API_KEY and AK_LIVEKIT__API_SECRET rather than a single pre-made token; with only a token, the whole team speaks through one participant (a warning is logged). A single agent with no handoffs is one participant, as before.

How the two frameworks differ​

OpenAI Agents SDKGoogle ADK
Live sessionOne native RealtimeSession for the callOne native Runner.run_live() and LiveRequestQueue for the call
HandoffRealtimeAgent.handoffsADK agent transfers (sub_agents) and live Workflow steps
ToolsNative tools on each RealtimeAgentNative tools and toolsets on each LlmAgent
VoiceOne session voice through model_configPer-agent Gemini(speech_config=...) or session RunConfig.speech_config

For how handoffs flow through the pipeline, see Realtime Voice Internals.

Example Projects​

Complete working examples (with In-Memory, Kafka, NATS, and AWS SQS architectures) are available in the examples/api/livekit-voice directory. The openai/handoffs variant shows multi-agent handoffs with an internet support line (supervisor, billing and tech support). The google/handoffs variant shows two agents passing the turn back and forth with ADK transfers (a planner and a budget controller) before reporting to the caller. The google/workflow variant shows a fixed order of steps (intake, scheduler, confirmer) written as an ADK Workflow.

Limitations​

  • OAK hooks and guardrails do not automatically run on native realtime turns. Use native SDK live features where supported.
  • Human approval has no decision path in this LiveKit edge. Approval-requiring OpenAI tools and ADK confirmation requests fail the connection clearly.
  • No session lock around tools. The SDK runs the tools; OAK sets the tool context when the call opens but does not hold the session lock while a tool runs.
  • One room per gateway: each LiveKitEdgeGateway joins one room, and the room uses one OAK session. Many gateways can run in one process, one per room.
  • Connection lifetime: a failed or expired native session is closed. A later input can start a fresh session; OAK does not replay the old transcript.
  • One LiveKit connection per agent: with handoffs, every listed agent is a participant, and LiveKit bills connection minutes for each.
  • Single replicas: one Agent Runner replica and one IOHandler process, as described above.

Ship your first agent today

Free and open source under Apache 2.0. No licensing costs and no lock-in.