Skip to main content
Version: 0.2.8

Traceability and Observability

Track, monitor, and debug all agent operations with built-in tracing and observability features.

Overview​

Agent Kernel provides comprehensive observability capabilities through integration with popular tracing platforms. Monitor agent execution, debug issues, and gain insights into your AI agent systems.

Supported Platforms​

Agent Kernel supports the following observability platforms:

  • Langfuse - Open-source LLM engineering platform for tracing, evaluating, and monitoring AI applications
  • OpenLLMetry (Traceloop) - OpenTelemetry-based observability for LLM applications with support for multiple backends

Getting Started with Langfuse​

Installation​

Install Agent Kernel with Langfuse support:

pip install agentkernel[langfuse]

Or if you need multiple framework integrations:

# OpenAI with Langfuse
pip install agentkernel[openai,langfuse]

# LangGraph with Langfuse
pip install agentkernel[langgraph,langfuse]

# CrewAI with Langfuse
pip install agentkernel[crewai,langfuse]

# Google ADK with Langfuse
pip install agentkernel[adk,langfuse]

Configuration​

Method 1: Configuration File​

Create or update config.yaml:

trace:
  enabled: true
  type: langfuse

Method 2: Environment Variables​

export AK_TRACE__ENABLED=true
export AK_TRACE__TYPE=langfuse

With tracing enabled in config, all agent interactions will be automatically traced to Langfuse.

Langfuse Credentials​

Configure Langfuse credentials via environment variables:

export LANGFUSE_PUBLIC_KEY=pk-lf-...
export LANGFUSE_SECRET_KEY=sk-lf-...
export LANGFUSE_HOST=https://cloud.langfuse.com  # or your self-hosted instance

Or add them to your .env file:

LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...
LANGFUSE_HOST=https://cloud.langfuse.com

Getting Langfuse Credentials​

  1. Sign up for a free account at https://cloud.langfuse.com
  2. Create a new project
  3. Navigate to Settings → API Keys
  4. Copy your Public Key and Secret Key

For self-hosted Langfuse, see the Langfuse documentation.

Getting Started with OpenLLMetry (Traceloop)​

Installation​

Install Agent Kernel with OpenLLMetry support:

pip install agentkernel[openllmetry]

Or if you need multiple framework integrations:

# OpenAI with OpenLLMetry
pip install agentkernel[openai,openllmetry]

# LangGraph with OpenLLMetry
pip install agentkernel[langgraph,openllmetry]

# CrewAI with OpenLLMetry
pip install agentkernel[crewai,openllmetry]

# Google ADK with OpenLLMetry
pip install agentkernel[adk,openllmetry]

Configuration​

Method 1: Configuration File​

Create or update config.yaml:

trace:
  enabled: true
  type: openllmetry

Method 2: Environment Variables​

export AK_TRACE__ENABLED=true
export AK_TRACE__TYPE=openllmetry

OpenLLMetry Credentials​

Configure Traceloop credentials via environment variables:

export TRACELOOP_API_KEY=your-api-key
# Optional: for self-hosted instances
export TRACELOOP_BASE_URL=https://api.traceloop.com

Or add them to your .env file:

TRACELOOP_API_KEY=your-api-key
TRACELOOP_BASE_URL=https://api.traceloop.com

Getting Traceloop Credentials​

  1. Sign up for an account at https://www.traceloop.com
  2. Create a new project
  3. Navigate to Settings → API Keys
  4. Copy your API key

For self-hosted deployment or other backends (like Datadog, New Relic, Honeycomb), see the Traceloop documentation.

OpenLLMetry Features​

OpenLLMetry provides:

  • OpenTelemetry Standards: Industry-standard telemetry data
  • Multiple Backends: Send traces to Traceloop, Datadog, New Relic, Honeycomb, and more
  • Automatic Instrumentation: Zero-code instrumentation for popular LLM frameworks
  • Performance Monitoring: Track latency, token usage, and costs
  • Distributed Tracing: Follow requests across multiple services

What Gets Traced​

When tracing is enabled, Agent Kernel automatically captures:

Execution Metrics​

  • Request/Response: Full input prompts and agent responses
  • Latency: Execution time for each agent interaction
  • Token Usage: Input and output tokens consumed

Agent Context​

  • Agent Name: Which agent handled the request
  • Session ID: Conversation session identifier
  • Metadata: Custom metadata and tags

LLM Interactions​

  • Model Calls: All LLM API calls with parameters
  • Prompt Templates: Resolved prompts sent to models
  • Completions: Model responses and reasoning

Tool Usage​

  • Tool Invocations: Which tools were called
  • Tool Parameters: Arguments passed to tools
  • Tool Results: Return values and execution status

Viewing Traces​

In Langfuse​

After running your agents with Langfuse tracing enabled:

  1. Log in to your Langfuse dashboard
  2. Navigate to Traces
  3. View detailed execution traces including:
    • Full conversation flow
    • LLM calls with prompts and completions
    • Tool invocations
    • Performance metrics
    • Token consumption

Langfuse Features:

  • Search and Filter: Find specific traces by agent, session, or metadata
  • Timeline View: See execution flow and timing
  • Cost Analysis: Track token usage and estimated costs
  • Error Tracking: Debug failed executions
  • Performance Metrics: Latency analysis and bottleneck identification

In OpenLLMetry/Traceloop​

After running your agents with OpenLLMetry tracing enabled:

  1. Log in to your Traceloop dashboard (or your configured backend)
  2. Navigate to Traces or Observability
  3. View detailed execution traces including:
    • Distributed trace timeline
    • LLM API calls with full context
    • Token usage and costs
    • Performance metrics
    • Error details

OpenLLMetry/Traceloop Features:

  • OpenTelemetry Standards: Compatible with any OpenTelemetry backend
  • Multi-Backend Support: View traces in Traceloop, Datadog, New Relic, etc.
  • Distributed Tracing: Track requests across multiple services
  • Custom Metrics: Add custom spans and metrics
  • Real-time Monitoring: Live trace streaming and alerts

Troubleshooting​

Langfuse Issues​

Traces Not Appearing:

  1. Check Credentials: Verify LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, and LANGFUSE_HOST are set correctly
  2. Verify Configuration: Ensure trace.enabled is set to true and trace.type is langfuse
  3. Check Installation: Confirm agentkernel[langfuse] is installed
  4. Review Logs: Look for trace initialization messages in your application logs

Authentication Errors:

# Verify your Langfuse credentials
python -c "from langfuse import Langfuse; client = Langfuse(); print(client.auth_check())"

This should return True if credentials are valid.

OpenLLMetry Issues​

Traces Not Appearing:

  1. Check Credentials: Verify TRACELOOP_API_KEY is set correctly
  2. Verify Configuration: Ensure trace.enabled is set to true and trace.type is openllmetry
  3. Check Installation: Confirm agentkernel[openllmetry] is installed
  4. Review Logs: Look for Traceloop initialization messages in your application logs

Connection Errors:

# Verify your Traceloop setup
python -c "from traceloop.sdk import Traceloop; Traceloop.init(app_name='test'); print('Success')"

Performance Impact​

Tracing adds minimal overhead:

  • Latency: < 50ms per request
  • Memory: Negligible (~1-2MB)
  • Network: Async batched uploads to minimize impact

Best Practices​

  1. Enable in All Environments: Use tracing in development, staging, and production
  2. Set Appropriate TTLs: Configure data retention in Langfuse settings
  3. Use Tags: Add meaningful tags for better organization
  4. Monitor Costs: Track token usage to manage LLM costs
  5. Review Regularly: Set up alerts for errors and performance degradation

Privacy and Security​

Data Handling​

When tracing is enabled:

  • All prompts and completions are sent to your chosen tracing platform
  • Langfuse: Data sent to Langfuse cloud or self-hosted instance
  • OpenLLMetry: Data sent to Traceloop or configured OpenTelemetry backend
  • Ensure compliance with your data privacy requirements
  • Consider self-hosting for sensitive data

Security Best Practices​

  1. Protect Credentials: Never commit API keys to version control
  2. Use Environment Variables: Store credentials securely
  3. Rotate Keys: Regularly rotate API keys
  4. Network Security: Use TLS/SSL for all connections (enabled by default)

Compliance​

For regulated industries:

  • Use self-hosted instances within your infrastructure
  • Configure appropriate data retention policies
  • Implement access controls in your tracing platform
  • Review and redact sensitive information before tracing

Self-Hosting Options​

Self-Hosting Langfuse​

For maximum control and privacy, self-host Langfuse:

  1. Follow the Langfuse self-hosting guide
  2. Update your configuration:
export LANGFUSE_HOST=https://langfuse.your-domain.com

OpenLLMetry with Custom Backends​

OpenLLMetry supports multiple backends via OpenTelemetry:

Datadog:

export TRACELOOP_BASE_URL=https://api.datadoghq.com
export DD_API_KEY=your-datadog-api-key

New Relic:

export TRACELOOP_BASE_URL=https://otlp.nr-data.net
export NEW_RELIC_LICENSE_KEY=your-license-key

Honeycomb:

export TRACELOOP_BASE_URL=https://api.honeycomb.io
export HONEYCOMB_API_KEY=your-api-key

Self-Hosted OpenTelemetry Collector:

export TRACELOOP_BASE_URL=http://your-otel-collector:4318

See OpenLLMetry documentation for more backend options.

Integrate with Your Own Traceability Platform​

Agent Kernel's plugin architecture makes it easy to integrate your own observability platform. If you're already using a different monitoring solution or have specific requirements, you can add support in just a few steps.

How to Add Your Own Platform​

  1. Create Framework-Specific Runners

Runner wraps the framework's execution with your tracing logic:

from agentkernel.openai.openai import OpenAIRunner
from agentkernel.core import Session

class MyCustomOpenAIRunner(OpenAIRunner):
    def __init__(self, client):
        super().__init__()
        self._client = client
    
    async def run(self, agent, session: Session, prompt):
        # Start a trace span
        with self._client.start_span("agent-execution") as span:
            span.set_attribute("session_id", session.id)
            span.set_attribute("prompt", prompt)
            
            # Run the agent
            result = await super().run(agent=agent, prompt=prompt, session=session)
            
            span.set_attribute("result", result)
            return result
  1. Initialize the module with the custom runner

Initialize the module with your custom runner

OpenAIModule([general_agent], runner=MyCustomOpenAIRunner())

Custom runner supersedes all other trace configurations

export MY_TRACE_API_KEY=your-api-key

Your custom observability platform is now integrated with Agent Kernel.

Example Use Cases​

  • Proprietary Monitoring Systems: Integrate with your company's internal monitoring tools
  • Compliance Requirements: Route traces to approved, compliant systems
  • Custom Aggregation: Combine multiple backends or add custom processing
  • Cost Optimization: Use cheaper or self-hosted alternatives

The extensible architecture ensures you're never locked into a specific platform.

Roadmap​

Upcoming observability features:

  • Custom Spans: Manual span creation for custom tracking
  • Trace Sampling: Configurable sampling strategies
  • Metric Exports: Prometheus and StatsD integrations
  • Real-time Dashboards: Built-in monitoring dashboards
  • Anomaly Detection: Automatic detection of unusual patterns
  • Enhanced Correlation: Better correlation between traces and logs

Summary​

  • Enable observability with simple configuration
  • Two Platform Options: Choose between Langfuse and OpenLLMetry
  • Langfuse: Specialized LLM observability platform with rich analytics
  • OpenLLMetry: OpenTelemetry-based solution with multi-backend support
  • Comprehensive trace data including LLM calls, tools, and performance
  • Minimal performance impact
  • Self-hosting and custom backend options for data privacy
  • Production-ready for all framework integrations (OpenAI, LangGraph, CrewAI, Google ADK)
💬 Ask AI Assistant

Get instant help with Agent Kernel documentation, examples, and more

AI Assistant