Example - Tracing and Evaluation for the OpenAI-Agents SDK - Langfuse

Evaluation for the OpenAI-Agents SDK

In this tutorial, we will learn how to monitor the internal steps (traces) of the OpenAI agent SDK and evaluate its performance using Langfuse and Hugging Face Datasets.

This guide covers online and offline evaluation metrics used by teams to bring agents to production fast and reliably. To learn more about evaluation strategies, check out our blog post.

Why AI agent Evaluation is important:

Step 0: Install the Required Libraries

Below we install the openai-agents library (the OpenAI Agents SDK link text), the openinference OpenTelemetry instrumentation, langfuse and the Hugging Face datasets library

%pip install openai-agents nest_asyncio openinference-instrumentation-openai-agents langfuse datasets -q

Note: you may need to restart the kernel to use updated packages.

Step 1: Instrument Your Agent

In this notebook, we will use Langfuse to trace, debug and evaluate our agent.

Note: If you use another framework (such as LangGraph, LlamaIndex, or CrewAI), you can find documentation on instrumenting them in our integration section.

import os

# Get keys for your project from the project settings page: https://cloud.langfuse.com
os.environ.setdefault("LANGFUSE_PUBLIC_KEY", "pk-lf-***")
os.environ.setdefault("LANGFUSE_SECRET_KEY", "sk-lf-***")
os.environ.setdefault("LANGFUSE_BASE_URL", "https://cloud.langfuse.com") # 🇪🇺 EU region
# Other Langfuse data regions include 🇺🇸 US: https://us.cloud.langfuse.com, 🇯🇵 Japan: https://jp.cloud.langfuse.com and ⚕️ HIPAA: https://hipaa.cloud.langfuse.com

# Your openai key
os.environ.setdefault("OPENAI_API_KEY", "sk-proj-***")

With the environment variables set, we can now initialize the Langfuse client. get_client() initializes the Langfuse client using the credentials provided in the environment variables.

from langfuse import get_client

langfuse = get_client()

# Verify connection
if langfuse.auth_check():
    print("Langfuse client is authenticated and ready!")
else:
    print("Authentication failed. Please check your credentials and host.")

Now, we initialize the OpenInference OpenAI Agents instrumentation. This third-party instrumentation automatically captures OpenAI Agents operations and exports OpenTelemetry (OTel) spans to Langfuse.

Note:nest_asyncio.apply() is not compatible with uvloop, which is commonly used with FastAPI to manage the event loop. If your application uses uvloop and you require nest_asyncio (e.g., for certain instrumentation or tracing libraries), you'll need to disable uvloop in the affected parts of your codebase and fall back to Python’s standard asyncio event loop.

import nest_asyncio
nest_asyncio.apply()

from openinference.instrumentation.openai_agents import OpenAIAgentsInstrumentor

OpenAIAgentsInstrumentor().instrument()

Step 2: Test Your Instrumentation

Here is a simple Q&A agent. We run it to confirm that the instrumentation is working correctly. If everything is set up correctly, you will see logs/spans in your observability dashboard.

import asyncio
from agents import Agent, Runner

async def main():
    agent = Agent(
        name="Assistant",
        instructions="You are a senior software engineer",
    )

result = await Runner.run(agent, "Tell me why it is important to evaluate AI agents.")
    print(result.final_output)

loop = asyncio.get_running_loop()
await loop.create_task(main())

langfuse.flush()

Check your Langfuse Traces Dashboard to confirm that the spans and logs have been recorded.

Step 3: Observe and Evaluate a More Complex Agent

Now that you have confirmed your instrumentation works, let's try a more complex query so we can see how advanced metrics (token usage, latency, costs, etc.) are tracked.

import asyncio
from agents import Agent, Runner, function_tool

# Example function tool.
@function_tool
def get_weather(city: str) -> str:
    return f"The weather in {city} is sunny."

agent = Agent(
    name="Hello world",
    instructions="You are a helpful agent.",
    tools=[get_weather],
)

async def main():
    result = await Runner.run(agent, input="What's the weather in Berlin?")
    print(result.final_output)

loop = asyncio.get_running_loop()
await loop.create_task(main())

Trace Structure

Langfuse records a trace that contains spans, which represent each step of your agent’s logic. Here, the trace contains the overall agent run and sub-spans for:

You can inspect these to see precisely where time is spent, how many tokens are used, and so on:

Online Evaluation

Online Evaluation refers to evaluating the agent in a live, real-world environment, i.e. during actual usage in production. This involves monitoring the agent’s performance on real user interactions and analyzing outcomes continuously.

We have written down a guide on different evaluation techniques here.

Common Metrics to Track in Production

  1. Costs — The instrumentation captures token usage, which you can transform into approximate costs by assigning a price per token.
  2. Latency — Observe the time it takes to complete each step, or the entire run.
  3. User Feedback — Users can provide direct feedback (thumbs up/down) to help refine or correct the agent.
  4. LLM-as-a-Judge — Use a separate LLM to evaluate your agent’s output in near real-time (e.g., checking for toxicity or correctness).

Offline Evaluation

Online evaluation is essential for live feedback, but you also need offline evaluation—systematic checks before or during development. This helps maintain quality and reliability before rolling changes into production.

Dataset Evaluation

In offline evaluation, you typically:

  1. Have a benchmark dataset (with prompt and expected output pairs)
  2. Run your agent on that dataset
  3. Compare outputs to the expected results or use an additional scoring mechanism

Running the Agent on the Dataset

We use the experiment runner SDK to run our agent against each dataset item. The experiment runner handles concurrent execution, automatic tracing, and evaluation.

You can repeat this process with different:

Check out the Langfuse docs to learn more ways to evaluate and debug your agent.