# Evaluation for the OpenAI-Agents SDK

In this tutorial, we will learn how to **monitor the internal steps (traces) of the [OpenAI agent SDK](https://github.com/openai/openai-agents-python)** and **evaluate its performance** using [Langfuse](/content/site-root.html) and [Hugging Face Datasets](https://huggingface.co/datasets).

This guide covers online and offline evaluation metrics used by teams to bring agents to production fast and reliably. To learn more about evaluation strategies, check out our [blog post](/content/blog/2025-03-04-llm-evaluation-101-best-practices-and-challenges/index.html).

**Why AI agent Evaluation is important:**

- Debugging issues when tasks fail or produce suboptimal results
- Monitoring costs and performance in real-time
- Improving reliability and safety through continuous feedback

## [Step 0: Install the Required Libraries](/content/guides/cookbook/example_evaluating_openai_agents#step-0-install-the-required-libraries/index.html)

Below we install the `openai-agents` library (the OpenAI Agents SDK [link text](https://github.com/openai/openai-agents-python)), the `openinference` OpenTelemetry instrumentation, `langfuse` and the Hugging Face `datasets` library

```
%pip install openai-agents nest_asyncio openinference-instrumentation-openai-agents langfuse datasets -q
```

```
Note: you may need to restart the kernel to use updated packages.
```

## [Step 1: Instrument Your Agent](/content/guides/cookbook/example_evaluating_openai_agents#step-1-instrument-your-agent/index.html)

In this notebook, we will use [Langfuse](/content/site-root.html) to trace, debug and evaluate our agent.

**Note:** If you use another framework (such as LangGraph, LlamaIndex, or CrewAI), you can find documentation on instrumenting them in our [integration section](/content/integrations/index.html).

```
import os

# Get keys for your project from the project settings page: https://cloud.langfuse.com
os.environ.setdefault("LANGFUSE_PUBLIC_KEY", "pk-lf-***")
os.environ.setdefault("LANGFUSE_SECRET_KEY", "sk-lf-***")
os.environ.setdefault("LANGFUSE_BASE_URL", "https://cloud.langfuse.com") # 🇪🇺 EU region
# Other Langfuse data regions include 🇺🇸 US: https://us.cloud.langfuse.com, 🇯🇵 Japan: https://jp.cloud.langfuse.com and ⚕️ HIPAA: https://hipaa.cloud.langfuse.com

# Your openai key
os.environ.setdefault("OPENAI_API_KEY", "sk-proj-***")
```

With the environment variables set, we can now initialize the Langfuse client. `get_client()` initializes the Langfuse client using the credentials provided in the environment variables.

```
from langfuse import get_client

langfuse = get_client()

# Verify connection
if langfuse.auth_check():
    print("Langfuse client is authenticated and ready!")
else:
    print("Authentication failed. Please check your credentials and host.")
```

Now, we initialize the [OpenInference OpenAI Agents instrumentation](https://github.com/Arize-ai/openinference/tree/main/python/instrumentation/openinference-instrumentation-openai-agents). This third-party instrumentation automatically captures OpenAI Agents operations and exports OpenTelemetry (OTel) spans to Langfuse.

**Note:**`nest_asyncio.apply()` is not compatible with `uvloop`, which is commonly used with FastAPI to manage the event loop. If your application uses `uvloop` and you require nest_asyncio (e.g., for certain instrumentation or tracing libraries), you'll need to disable `uvloop` in the affected parts of your codebase and fall back to Python’s standard asyncio event loop.

```
import nest_asyncio
nest_asyncio.apply()
```

```
from openinference.instrumentation.openai_agents import OpenAIAgentsInstrumentor

OpenAIAgentsInstrumentor().instrument()
```

## [Step 2: Test Your Instrumentation](/content/guides/cookbook/example_evaluating_openai_agents#step-2-test-your-instrumentation/index.html)

Here is a simple Q&A agent. We run it to confirm that the instrumentation is working correctly. If everything is set up correctly, you will see logs/spans in your observability dashboard.

```
import asyncio
from agents import Agent, Runner

async def main():
    agent = Agent(
        name="Assistant",
        instructions="You are a senior software engineer",
    )

result = await Runner.run(agent, "Tell me why it is important to evaluate AI agents.")
    print(result.final_output)

loop = asyncio.get_running_loop()
await loop.create_task(main())

langfuse.flush()
```

Check your Langfuse Traces Dashboard to confirm that the spans and logs have been recorded.

## [Step 3: Observe and Evaluate a More Complex Agent](/content/guides/cookbook/example_evaluating_openai_agents#step-3-observe-and-evaluate-a-more-complex-agent/index.html)

Now that you have confirmed your instrumentation works, let's try a more complex query so we can see how advanced metrics (token usage, latency, costs, etc.) are tracked.

```
import asyncio
from agents import Agent, Runner, function_tool

# Example function tool.
@function_tool
def get_weather(city: str) -> str:
    return f"The weather in {city} is sunny."

agent = Agent(
    name="Hello world",
    instructions="You are a helpful agent.",
    tools=[get_weather],
)

async def main():
    result = await Runner.run(agent, input="What's the weather in Berlin?")
    print(result.final_output)

loop = asyncio.get_running_loop()
await loop.create_task(main())
```

### [Trace Structure](/content/guides/cookbook/example_evaluating_openai_agents#trace-structure/index.html)

Langfuse records a **trace** that contains **spans**, which represent each step of your agent’s logic. Here, the trace contains the overall agent run and sub-spans for:

- The tool call (get_weather)
- The LLM calls (Responses API with 'gpt-4o')

You can inspect these to see precisely where time is spent, how many tokens are used, and so on:

## [Online Evaluation](/content/guides/cookbook/example_evaluating_openai_agents#online-evaluation/index.html)

Online Evaluation refers to evaluating the agent in a live, real-world environment, i.e. during actual usage in production. This involves monitoring the agent’s performance on real user interactions and analyzing outcomes continuously.

We have written down a guide on different evaluation techniques [here](/content/blog/2025-03-04-llm-evaluation-101-best-practices-and-challenges/index.html).

### [Common Metrics to Track in Production](/content/guides/cookbook/example_evaluating_openai_agents#common-metrics-to-track-in-production/index.html)

1. **Costs** — The instrumentation captures token usage, which you can transform into approximate costs by assigning a price per token.
2. **Latency** — Observe the time it takes to complete each step, or the entire run.
3. **User Feedback** — Users can provide direct feedback (thumbs up/down) to help refine or correct the agent.
4. **LLM-as-a-Judge** — Use a separate LLM to evaluate your agent’s output in near real-time (e.g., checking for toxicity or correctness).

### [Offline Evaluation](/content/guides/cookbook/example_evaluating_openai_agents#offline-evaluation/index.html)

Online evaluation is essential for live feedback, but you also need **offline evaluation**—systematic checks before or during development. This helps maintain quality and reliability before rolling changes into production.

### [Dataset Evaluation](/content/guides/cookbook/example_evaluating_openai_agents#dataset-evaluation/index.html)

In offline evaluation, you typically:

1. Have a benchmark dataset (with prompt and expected output pairs)
2. Run your agent on that dataset
3. Compare outputs to the expected results or use an additional scoring mechanism

### [Running the Agent on the Dataset](/content/guides/cookbook/example_evaluating_openai_agents#running-the-agent-on-the-dataset/index.html)

We use the [experiment runner SDK](/content/docs/evaluation/experiments/experiments-via-sdk#experiment-runner-sdk/index.html) to run our agent against each dataset item. The experiment runner handles concurrent execution, automatic tracing, and evaluation.

You can repeat this process with different:

- Search tools (e.g. different context sized for OpenAI's `WebSearchTool`)
- Models (gpt-5.2, gpt-5.2-mini, etc.)
- Tools (search vs. no search)

* * *

Check out the [Langfuse docs](/content/docs/index.html) to learn more ways to evaluate and debug your agent.