Skip to main content
Version: Next

Traceability and Observability

Track, monitor, and debug all agent operations with built-in tracing and observability features.

Overview​

Agent Kernel provides comprehensive observability capabilities through integration with popular tracing platforms. Monitor agent execution, debug issues, and gain insights into your AI agent systems.

Supported Platforms​

Agent Kernel supports the following observability platforms:

  • Langfuse - Open-source LLM engineering platform for tracing, evaluating, and monitoring AI applications
  • OpenLLMetry (Traceloop) - OpenTelemetry-based observability for LLM applications with support for multiple backends
  • Pydantic Logfire - OpenTelemetry-based observability from the Pydantic team, with a native OpenAI Agents SDK integration
  • AWS CloudWatch - OpenTelemetry traces exported straight to AWS X-Ray, searchable in CloudWatch Transaction Search

Getting Started with Langfuse​

Installation​

Install Agent Kernel with Langfuse support:

pip install agentkernel[langfuse]

Or if you need multiple framework integrations:

# OpenAI with Langfuse
pip install agentkernel[openai,langfuse]

# LangGraph with Langfuse
pip install agentkernel[langgraph,langfuse]

# CrewAI with Langfuse
pip install agentkernel[crewai,langfuse]

# Google ADK with Langfuse
pip install agentkernel[adk,langfuse]

Configuration​

Method 1: Configuration File​

Create or update config.yaml:

trace:
  enabled: true
  type: langfuse

Method 2: Environment Variables​

export AK_TRACE__ENABLED=true
export AK_TRACE__TYPE=langfuse

With tracing enabled in config, all agent interactions will be automatically traced to Langfuse.

Langfuse Credentials​

Configure Langfuse credentials via environment variables:

export LANGFUSE_PUBLIC_KEY=pk-lf-...
export LANGFUSE_SECRET_KEY=sk-lf-...
export LANGFUSE_HOST=https://cloud.langfuse.com  # or your self-hosted instance

Or add them to your .env file:

LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...
LANGFUSE_HOST=https://cloud.langfuse.com

Getting Langfuse Credentials​

  1. Sign up for a free account at https://cloud.langfuse.com
  2. Create a new project
  3. Navigate to Settings → API Keys
  4. Copy your Public Key and Secret Key

For self-hosted Langfuse, see the Langfuse documentation.

Getting Started with OpenLLMetry (Traceloop)​

Installation​

Install Agent Kernel with OpenLLMetry support:

pip install agentkernel[openllmetry]

Or if you need multiple framework integrations:

# OpenAI with OpenLLMetry
pip install agentkernel[openai,openllmetry]

# LangGraph with OpenLLMetry
pip install agentkernel[langgraph,openllmetry]

# CrewAI with OpenLLMetry
pip install agentkernel[crewai,openllmetry]

# Google ADK with OpenLLMetry
pip install agentkernel[adk,openllmetry]

Configuration​

Method 1: Configuration File​

Create or update config.yaml:

trace:
  enabled: true
  type: openllmetry

Method 2: Environment Variables​

export AK_TRACE__ENABLED=true
export AK_TRACE__TYPE=openllmetry

OpenLLMetry Credentials​

Configure Traceloop credentials via environment variables:

export TRACELOOP_API_KEY=your-api-key
# Optional: for self-hosted instances
export TRACELOOP_BASE_URL=https://api.traceloop.com

Or add them to your .env file:

TRACELOOP_API_KEY=your-api-key
TRACELOOP_BASE_URL=https://api.traceloop.com

Getting Traceloop Credentials​

  1. Sign up for an account at https://www.traceloop.com
  2. Create a new project
  3. Navigate to Settings → API Keys
  4. Copy your API key

For self-hosted deployment or other backends (like Datadog, New Relic, Honeycomb), see the Traceloop documentation.

OpenLLMetry Features​

OpenLLMetry provides:

  • OpenTelemetry Standards: Industry-standard telemetry data
  • Multiple Backends: Send traces to Traceloop, Datadog, New Relic, Honeycomb, and more
  • Automatic Instrumentation: Zero-code instrumentation for popular LLM frameworks
  • Performance Monitoring: Track latency, token usage, and costs
  • Distributed Tracing: Follow requests across multiple services

What Gets Traced​

When tracing is enabled, Agent Kernel automatically captures:

Execution Metrics​

  • Request/Response: Full input prompts and agent responses
  • Latency: Execution time for each agent interaction
  • Token Usage: Input and output tokens consumed

Agent Context​

  • Agent Name: Which agent handled the request
  • Session ID: Conversation session identifier
  • Metadata: Custom metadata and tags

LLM Interactions​

  • Model Calls: All LLM API calls with parameters
  • Prompt Templates: Resolved prompts sent to models
  • Completions: Model responses and reasoning

Tool Usage​

  • Tool Invocations: Which tools were called
  • Tool Parameters: Arguments passed to tools
  • Tool Results: Return values and execution status

Viewing Traces​

In Langfuse​

After running your agents with Langfuse tracing enabled:

  1. Log in to your Langfuse dashboard
  2. Navigate to Traces
  3. View detailed execution traces including:
    • Full conversation flow
    • LLM calls with prompts and completions
    • Tool invocations
    • Performance metrics
    • Token consumption

Langfuse Features:

  • Search and Filter: Find specific traces by agent, session, or metadata
  • Timeline View: See execution flow and timing
  • Cost Analysis: Track token usage and estimated costs
  • Error Tracking: Debug failed executions
  • Performance Metrics: Latency analysis and bottleneck identification

In OpenLLMetry/Traceloop​

After running your agents with OpenLLMetry tracing enabled:

  1. Log in to your Traceloop dashboard (or your configured backend)
  2. Navigate to Traces or Observability
  3. View detailed execution traces including:
    • Distributed trace timeline
    • LLM API calls with full context
    • Token usage and costs
    • Performance metrics
    • Error details

OpenLLMetry/Traceloop Features:

  • OpenTelemetry Standards: Compatible with any OpenTelemetry backend
  • Multi-Backend Support: View traces in Traceloop, Datadog, New Relic, etc.
  • Distributed Tracing: Track requests across multiple services
  • Custom Metrics: Add custom spans and metrics
  • Real-time Monitoring: Live trace streaming and alerts

Troubleshooting​

Langfuse Issues​

Traces Not Appearing:

  1. Check Credentials: Verify LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, and LANGFUSE_HOST are set correctly
  2. Verify Configuration: Ensure trace.enabled is set to true and trace.type is langfuse
  3. Check Installation: Confirm agentkernel[langfuse] is installed
  4. Review Logs: Look for trace initialization messages in your application logs

Authentication Errors:

# Verify your Langfuse credentials
python -c "from langfuse import Langfuse; client = Langfuse(); print(client.auth_check())"

This should return True if credentials are valid.

OpenLLMetry Issues​

Traces Not Appearing:

  1. Check Credentials: Verify TRACELOOP_API_KEY is set correctly
  2. Verify Configuration: Ensure trace.enabled is set to true and trace.type is openllmetry
  3. Check Installation: Confirm agentkernel[openllmetry] is installed
  4. Review Logs: Look for Traceloop initialization messages in your application logs

Connection Errors:

# Verify your Traceloop setup
python -c "from traceloop.sdk import Traceloop; Traceloop.init(app_name='test'); print('Success')"

Logfire Issues​

Traces Not Appearing:

  1. Check Token: Verify LOGFIRE_TOKEN is set — without it, Logfire runs locally and does not ship traces
  2. Verify Configuration: Ensure trace.enabled is true and trace.type is logfire
  3. Check Installation: Confirm agentkernel[logfire] is installed
  4. Review Logs: Look for the ak.trace.logfire "Logfire configured" debug message

Setup Check:

python -c "import logfire; logfire.configure(send_to_logfire='if-token-present'); print('Success')"

CloudWatch Issues​

Startup fails with "trace.type: cloudwatch needs an AWS region": set AWS_REGION, or point OTEL_EXPORTER_OTLP_TRACES_ENDPOINT at a collector or the CloudWatch agent.

Traces Not Appearing:

  1. Check Transaction Search: an ak.trace.cloudwatch error X-Ray in <region> rejected the span export (HTTP 400): The OTLP API is supported with CloudWatch Logs as a Trace Segment Destination... means it is not enabled in the region named in the message (which is the region exports go to: AWS_REGION, else your profile's default) — aws xray get-trace-segment-destination must report "Destination": "CloudWatchLogs" and "Status": "ACTIVE"
  2. Check Permissions: X-Ray in <region> rejected the span export (HTTP 403) means the role lacks AWSXrayWriteOnlyAccess
  3. Check Credentials: Exception while exporting Span. with NoCredentialsError means no AWS credentials resolved in the process
  4. Check Region: spans land in the region named by AWS_REGION (or the endpoint URL), so open the console in that region
  5. Review Logs: look for the ak.trace.cloudwatch "CloudWatch tracing configured" debug message

Setup Check:

aws sts get-caller-identity && aws xray get-trace-segment-destination

Performance Impact​

Tracing adds minimal overhead:

  • Latency: < 50ms per request
  • Memory: Negligible (~1-2MB)
  • Network: Async batched uploads to minimize impact

Best Practices​

  1. Enable in All Environments: Use tracing in development, staging, and production
  2. Set Appropriate TTLs: Configure data retention in Langfuse settings
  3. Use Tags: Add meaningful tags for better organization
  4. Monitor Costs: Track token usage to manage LLM costs
  5. Review Regularly: Set up alerts for errors and performance degradation

Privacy and Security​

Data Handling​

When tracing is enabled:

  • All prompts and completions are sent to your chosen tracing platform
  • Langfuse: Data sent to Langfuse cloud or self-hosted instance
  • OpenLLMetry: Data sent to Traceloop or configured OpenTelemetry backend
  • Logfire: Data sent to Logfire when LOGFIRE_TOKEN is set; otherwise kept local
  • CloudWatch: Data sent to AWS X-Ray and stored in your account's CloudWatch Logs (aws/spans), in the region you export to
  • Ensure compliance with your data privacy requirements
  • Consider self-hosting for sensitive data

Security Best Practices​

  1. Protect Credentials: Never commit API keys to version control
  2. Use Environment Variables: Store credentials securely
  3. Rotate Keys: Regularly rotate API keys
  4. Network Security: Use TLS/SSL for all connections (enabled by default)

Compliance​

For regulated industries:

  • Use self-hosted instances within your infrastructure
  • Configure appropriate data retention policies
  • Implement access controls in your tracing platform
  • Review and redact sensitive information before tracing

Self-Hosting Options​

Self-Hosting Langfuse​

For maximum control and privacy, self-host Langfuse:

  1. Follow the Langfuse self-hosting guide
  2. Update your configuration:
export LANGFUSE_HOST=https://langfuse.your-domain.com

OpenLLMetry with Custom Backends​

OpenLLMetry supports multiple backends via OpenTelemetry:

Datadog:

export TRACELOOP_BASE_URL=https://api.datadoghq.com
export DD_API_KEY=your-datadog-api-key

New Relic:

export TRACELOOP_BASE_URL=https://otlp.nr-data.net
export NEW_RELIC_LICENSE_KEY=your-license-key

Honeycomb:

export TRACELOOP_BASE_URL=https://api.honeycomb.io
export HONEYCOMB_API_KEY=your-api-key

Self-Hosted OpenTelemetry Collector:

export TRACELOOP_BASE_URL=http://your-otel-collector:4318

See OpenLLMetry documentation for more backend options.

Getting Started with Pydantic Logfire​

Pydantic Logfire is an OpenTelemetry-based observability platform from the Pydantic team.

Installation​

pip install agentkernel[logfire]

Or with a framework integration:

# OpenAI with Logfire
pip install agentkernel[openai,logfire]

# CrewAI with Logfire
pip install agentkernel[crewai,logfire]

# Google ADK with Logfire
pip install agentkernel[adk,logfire]

Configuration​

Method 1: Configuration File​

trace:
  enabled: true
  type: logfire

Method 2: Environment Variables​

export AK_TRACE__ENABLED=true
export AK_TRACE__TYPE=logfire

Logfire Credentials​

Set your Logfire write token via an environment variable:

export LOGFIRE_TOKEN=your-write-token

Without LOGFIRE_TOKEN, Logfire runs locally and does not ship traces (Agent Kernel configures it with send_to_logfire="if-token-present"), so your agents still run.

Getting a Logfire Token​

  1. Sign up at https://logfire.pydantic.dev
  2. Create a new project
  3. Copy the project's write token

Framework Coverage​

Logfire configures the global OpenTelemetry tracer provider, and each non-streaming agent run (execution.mode: rest_sync) is wrapped in a Logfire span carrying the session_id. Streaming runs (execution.mode: stream) are not wrapped in a session span — matching the Langfuse and OpenLLMetry runners — though deep instrumentation still emits LLM/tool spans where it is active. Deep (LLM/tool-level) instrumentation depends on the framework:

FrameworkDepth under Logfire
OpenAI Agents SDKFull — native logfire.instrument_openai_agents()
CrewAIFull — OpenInference CrewAI + LiteLLM instrumentors
Google ADKFull — OpenInference Google ADK instrumentor
Microsoft Agent FrameworkSession span, plus MAF's own OpenTelemetry agent and chat spans (MAF instruments itself; Agent Kernel activates nothing extra)
LangGraphSession span only
SmolagentsSession span only

Microsoft Agent Framework's own spans carry metadata only by default (operation names, token counts, durations). To include prompts, completions, tool arguments and tool results in them, set MAF's ENABLE_SENSITIVE_DATA=true, and only when the backend receiving the spans is secured for that data.

Getting Started with AWS CloudWatch​

Amazon CloudWatch ingests OpenTelemetry traces through AWS X-Ray and makes every span searchable with Transaction Search. Agent Kernel exports spans over OTLP/HTTP straight to the regional X-Ray endpoint, signed with your AWS credentials (SigV4), so no collector or sidecar is needed. That makes it work the same way on AWS Lambda, ECS, EKS, and EC2.

Installation​

pip install agentkernel[cloudwatch]

Or with a framework integration:

# OpenAI with CloudWatch
pip install agentkernel[openai,cloudwatch]

# CrewAI with CloudWatch
pip install agentkernel[crewai,cloudwatch]

# Google ADK with CloudWatch
pip install agentkernel[adk,cloudwatch]

Configuration​

Method 1: Configuration File​

trace:
  enabled: true
  type: cloudwatch

Method 2: Environment Variables​

export AK_TRACE__ENABLED=true
export AK_TRACE__TYPE=cloudwatch

AWS Prerequisites​

  1. Enable Transaction Search once per account and region. In the CloudWatch console, choose Application Signals → Transaction Search → Enable Transaction Search. To do it with the AWS CLI instead, follow the AWS guide: add a CloudWatch Logs resource policy for xray.amazonaws.com, then run aws xray update-trace-segment-destination --destination CloudWatchLogs. aws xray get-trace-segment-destination reports "Status": "ACTIVE" once spans are being ingested, which can take about 10 minutes.
  2. Grant write access: attach the AWS managed policy AWSXrayWriteOnlyAccess to the identity that runs your agent (the Lambda execution role, ECS task role, or your local profile). Agent Kernel's Terraform modules do not attach it for you.
  3. Region and credentials come from the standard AWS SDK chain. The region is read from AWS_REGION, then AWS_DEFAULT_REGION or the active profile; Lambda and ECS set it for you. Credentials can come from environment variables, a profile, SSO, or an instance or task role.

Export Endpoint​

The exporter uses the standard OpenTelemetry environment variables, so no Agent Kernel configuration beyond trace.type is needed:

SettingBehaviour
No endpoint variable setExports to https://xray.<region>.amazonaws.com/v1/traces, SigV4-signed (amazonaws.com.cn for cn-* regions)
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT set to https://xray.<region>.amazonaws.com/v1/traces, or OTEL_EXPORTER_OTLP_ENDPOINT set to https://xray.<region>.amazonaws.comExports to X-Ray, signed for the region in the URL
Either variable set to any other URL, such as the CloudWatch agent or an OpenTelemetry collector on http://localhost:4318/v1/tracesExports there unsigned, and the agent or collector handles authentication
OTEL_SERVICE_NAMEService name in CloudWatch (default AgentKernel)
OTEL_RESOURCE_ATTRIBUTESExtra resource attributes, e.g. deployment.environment.name=prod

The other standard OTEL_* variables (headers, timeout, compression, sampler, batch processor, attribute limits) apply as usual.

If an OpenTelemetry tracer provider is already installed, Agent Kernel reuses it rather than registering a second one. This happens, for example, when you run under opentelemetry-instrument with the AWS Distro for OpenTelemetry, or inside Bedrock AgentCore Runtime. In that case it adds only its session processor, and the existing provider's exporter configuration applies.

What Gets Traced​

Each non-streaming agent run is wrapped in a span named Agent Kernel <Framework>. The span carries:

  • session.id: the Agent Kernel session ID. It is also copied onto every span the framework instrumentation emits during the run, so you can filter a whole conversation in Transaction Search.
  • input.value / output.value: the prompt and the agent's reply.

The service resource defaults to service.name=AgentKernel and aws.service.type=gen_ai_agent. Values set through OTEL_SERVICE_NAME or OTEL_RESOURCE_ATTRIBUTES win.

Streaming runs (execution.mode: stream) are not wrapped in a session span, which matches the other providers, though deep instrumentation still emits LLM and tool spans where it is active. How deep the LLM- and tool-level instrumentation goes depends on the framework:

FrameworkDepth under CloudWatch
OpenAI Agents SDKFull — OpenInference OpenAI Agents instrumentor
CrewAIFull — OpenInference CrewAI + LiteLLM instrumentors
Google ADKFull — OpenInference Google ADK instrumentor
Pydantic AIFull — Pydantic AI instrumentation + OpenInference span processor
Microsoft Agent FrameworkSession span, plus MAF's own OpenTelemetry agent and chat spans (MAF instruments itself; Agent Kernel activates nothing extra)
LangGraphSession span only
SmolagentsSession span only

Microsoft Agent Framework's own spans carry metadata only by default (operation names, token counts, durations). To include prompts, completions, tool arguments and tool results in them, set MAF's ENABLE_SENSITIVE_DATA=true, and only when the backend receiving the spans is secured for that data.

X-Ray rejects spans larger than 200 KB. If your prompts or replies are very large, cap attribute sizes with OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT.

Running on AWS Lambda​

The batch span processor normally exports in the background, but Lambda freezes the execution environment as soon as an invocation returns. When AWS_LAMBDA_FUNCTION_NAME is set, Agent Kernel therefore flushes pending spans at the end of every agent run, including failed runs. The cost is one export round trip added to each invocation. Agent spans are not parented under the function's own X-Ray segment when Lambda active tracing is on: with X-Ray sampling, that parent would cause most agent spans to be dropped.

Viewing Traces in CloudWatch​

  1. Open the CloudWatch console in the region you export to.
  2. Go to Application Signals → Transaction Search and filter spans by session.id or by service AgentKernel. Open a trace to see the run span with its agent, LLM, and tool spans nested under it.
  3. Optionally, check GenAI Observability. Agent Kernel sets the aws.service.type=gen_ai_agent resource attribute and session.id that the AWS Distro for OpenTelemetry sets, but AWS does not document whether its agent and session views list agents that export without the distro, so treat Transaction Search as the supported view.

Integrate with Your Own Traceability Platform​

Agent Kernel's plugin architecture makes it easy to integrate your own observability platform. If you're already using a different monitoring solution or have specific requirements, you can add support in just a few steps.

How to Add Your Own Platform​

  1. Create Framework-Specific Runners

Runner wraps the framework's execution with your tracing logic:

from agentkernel.openai.openai import OpenAIRunner
from agentkernel.core import Session

class MyCustomOpenAIRunner(OpenAIRunner):
    def __init__(self, client):
        super().__init__()
        self._client = client
    
    async def run(self, agent, session: Session, requests):
        # Start a trace span
        with self._client.start_span("agent-execution") as span:
            span.set_attribute("session_id", session.id)

            # Run the agent
            result = await super().run(agent, session, requests)

            span.set_attribute("prompt", result.prompt)
            span.set_attribute("result", str(result))
            return result

A custom runner is for behaviour that wraps the native call. To pass the framework's own run arguments (OpenAI's hooks, run_config and max_turns, LangGraph's config, ADK's plugins, ...) declare them per agent with Module.run_options instead; the base runner merges them into the call, so they also reach the SDK through the traced runners above without a subclass.

  1. Initialize the module with the custom runner

Initialize the module with your custom runner

OpenAIModule([general_agent], runner=MyCustomOpenAIRunner())

Custom runner supersedes all other trace configurations

export MY_TRACE_API_KEY=your-api-key

Your custom observability platform is now integrated with Agent Kernel.

Example Use Cases​

  • Proprietary Monitoring Systems: Integrate with your company's internal monitoring tools
  • Compliance Requirements: Route traces to approved, compliant systems
  • Custom Aggregation: Combine multiple backends or add custom processing
  • Cost Optimization: Use cheaper or self-hosted alternatives

The extensible architecture ensures you're never locked into a specific platform.

Roadmap​

Upcoming observability features:

  • Custom Spans: Manual span creation for custom tracking
  • Trace Sampling: Configurable sampling strategies
  • Metric Exports: Prometheus and StatsD integrations
  • Real-time Dashboards: Built-in monitoring dashboards
  • Anomaly Detection: Automatic detection of unusual patterns
  • Enhanced Correlation: Better correlation between traces and logs

Summary​

  • Enable observability with simple configuration
  • Four Platform Options: Choose between Langfuse, OpenLLMetry, Pydantic Logfire, and AWS CloudWatch
  • Langfuse: Specialized LLM observability platform with rich analytics
  • OpenLLMetry: OpenTelemetry-based solution with multi-backend support
  • Pydantic Logfire: OpenTelemetry-based platform with a native OpenAI Agents SDK integration
  • AWS CloudWatch: OpenTelemetry traces in your own AWS account via X-Ray, with no collector needed
  • Comprehensive trace data including LLM calls, tools, and performance
  • Minimal performance impact
  • Self-hosting and custom backend options for data privacy
  • Production-ready for all framework integrations (OpenAI, LangGraph, CrewAI, Google ADK)

Ready to Ship Your
First Agent?

Free, open-source, Apache 2.0. No licensing costs, no vendor lock-in. Join hundreds of developers building production AI agents with Agent Kernel.

Quick StartView On GitHub
Agent Kernel
Ask Agent Kernel