Upgrade trace-level evaluators

Trace-level evaluators use Langfuse's legacy trace-centric data model and are being deprecated as part of the Langfuse v4 upgrade. Existing evaluators continue to run while you upgrade. Based on existing configurations, the overwhelming majority can be upgraded by targeting the observation that already contains the required input, output, and context.

Setting up a new evaluator? Start with an observation-level evaluator in the LLM-as-a-Judge setup guide. This page is only for upgrading existing trace-level evaluators.

Why this upgrade is needed

Real-time execution: The legacy engine waits for and assembles a full trace before evaluating it. Observation-level evaluators run as observations are ingested.
Scalable by design: Evaluating one observation avoids loading the full trace and scales with larger traces and higher evaluation volume.
Required for future capabilities: Any future multi-span support will also use the observation-centric data model. Upgrading now is the foundation for it.

How to upgrade

First, upgrade your SDK or ingestion path. Do this before changing evaluator configurations.

Observation-level evaluators run in real time with:

Python SDK: v4.7.0+ (upgrade guide)
JS/TS SDK: v5.4.0+ (upgrade guide)
Direct OpenTelemetry ingestion: x-langfuse-ingestion-version: 4 (setup guide)

After upgrading to the v4 data model, input, output, and context are no longer available as separate trace fields. The equivalent values must be available on an observation.

1. Start the upgrade in Langfuse

Open Evaluation > LLM-as-a-Judge. You will be greeted with the upgrade screen, which walks you through all evaluators that need to be upgraded and recommends the appropriate assistant for each one.

2. Follow the recommended assistant path

Langfuse Assistant: Recommended when the evaluator can be upgraded by selecting an existing observation and translating its configuration. This is the fastest path and does not require manual UI setup.
Coding assistant: Recommended when the SDK or instrumentation must change first. Langfuse provides the relevant context and prompt to use in your coding editor.

The upgrade screen and recommended assistant guide you through the full process. You do not need to recreate evaluator configurations manually in the UI.

3. Review and create the evaluator

Review the proposed target observation, filters, and variable mappings. When creating the observation-level evaluator, you can keep the legacy evaluator active to compare results or deactivate it immediately.

How resulting evaluation scores might look different

A trace-level evaluator produces one score per matching trace. An observation-level evaluator produces one score per matching observation.

If three observations in one trace match the filters, the upgraded evaluator produces three scores. This is expected. To keep one score per trace, narrow the filters to one observation using its name, type, or root-observation status.

Validate the upgrade

Confirm that the new evaluator receives the expected input, output, and context.
Check that the number of matched observations and scores per trace is intentional.
If you kept the legacy evaluator active, compare the results and deactivate it when ready.

If you need to revert, deactivate the new evaluator and reactivate the existing legacy evaluator. Historical evaluation results remain accessible.

Optional: How configurations might change

The upgrade screen and recommended assistant guide you through every change. You do not need to classify or update configurations manually. Expand this section only if you want to understand the technical details.

View technical configuration cases and a before/after example

Langfuse groups existing configurations by their variable mappings and recommends an assistant based on the required change:

Current variable mapping	Typical configuration change
`Observation` for every variable, all from one observation	Target the same span or generation and translate the filters and variable mappings.
`Trace` for every variable	Select the observation holding the equivalent input, output, and metadata. For end-to-end evaluations, this is often a root observation.
A mix of `Trace` and one `Observation`	Target that observation and make the trace fields available there via `propagate_attributes()` or an instrumentation update.
More than one `Observation`, with or without `Trace`	Write the required values to a root or dedicated evaluation observation, then target it directly. Any future multi-span support will also use the observation-centric model.

Before and after example

This example upgrades a trace-level evaluator that reads trace input and output to target the span containing the same values.

Before: trace-level evaluator

Target: Trace
Filters:
  environment not in ["sdk-experiment"]
  name = "user-workflow"
Variable mapping:
  query = trace.input.key
  generation = trace.output

After: observation-level evaluator

Target: Observation
Filters:
  environment not in ["sdk-experiment"]
  traceName = "user-workflow"
  type = "SPAN"
Variable mapping:
  query = observation.input.key
  generation = observation.output

Getting help

Troubleshooting: Why is my observation-level evaluator not executing?
Documentation: LLM-as-a-Judge step-by-step set up guide
GitHub: Report issues at github.com/langfuse/langfuse
Support: Contact support@langfuse.com for enterprise customers

Last updated: July 17, 2026

Was this page helpful?

On this page