How do you turn traces into a training dataset?

Turning agent traces into a training dataset takes four stages. Capture every run as a trace. Sample the runs worth a human’s attention. Label each sampled run against a written spec of what good looks like. Build the survivors into the format your training method needs, then rebuild weekly as production keeps producing new runs. Overmind does all four from the traces your agent already emits, then trains an open-weight model on the result.

Every agentic AI system you run is quietly writing its own training set, and most teams never use it.

When an agent runs, it emits a stream of inputs and outputs: the prompt, the tool calls, the reasoning and the final answer. That stream is a trace, and once you standardise how you capture it (the OpenTelemetry GenAI semantic conventions are the default schema here) you have raw telemetry. A training dataset is narrower: a curated set of runs, scored against a spec, designed for fine-tuning jobs.

Building a dataset like this is a four-stage pipeline.

Here's a worked example, using a refund agent, to show what each stage actually involves.

Step 1: how do you capture an agent trace?

You capture an agent trace by instrumenting every model call and tool call, so each run lands as one OpenTelemetry trace with its goal, its calls and its outcome attached.

If your agent is instrumented at all, you already have this part. Every run generates a trace: the goal it was given, every model call, every tool invocation, every observation from the environment, and the eventual outcome. That trace is the raw material everything downstream depends on.

The industry is converging on a shared vocabulary for this. OpenTelemetry’s GenAI working group has been building standard span attributes for LLM calls since 2024, covering model name, token counts, latency, and tool execution as first-class fields rather than something each team invents from scratch. That matters more than it sounds like it should, because a trace schema you have to redesign every time you switch observability vendors is a trace schema nobody trusts enough to build a dataset on top of.

Overmind captures this stage for you. Overmind is the model training platform for AI teams, and the whole product starts here, turning your production traces into specialized models you own, trained, benchmarked and served. One overmind.init(providers="auto")call in a Python agent captures every LLM invocation across OpenAI, Anthropic, Google Gemini, Agno and LangChain or LangGraph, logging inputs, outputs, latency, token counts and errors without extra plumbing on your side. Anything else that speaks OpenTelemetry can post its traces straight to the Observability endpoint, and traces already sitting in Langfuse, LangSmith, Braintrust or Galileo import through a connector. The point isn’t the SDK itself. It’s that capture has to be automatic and total, or the sampling and labeling stages downstream never get the raw material they need.

Step 2: how do you choose which runs to label?

You choose runs to label with a random slice for a baseline, a stratified slice for the segments you care about, and every run that failed or scored badly.

You should not try to label every run. Most of what an agent does in a given week is unremarkable, and reviewing all of it teaches a labeling team nothing it didn’t already know. The job at this stage is picking which runs are worth a human’s attention.

Three strategies cover most cases.

• Random gives you an honest baseline of what the agent does on an ordinary day. Use it every cycle, as the control slice you compare everything else against.
• Stratified gives you deliberate coverage of cases you already care about, such as a specific refund reason, a customer tier, or a tool that keeps timing out. Use it when a known segment matters more than the average run.
• Failure-weighted gives you the most signal per run, because the runs that went wrong carry the most information. Use it when you have an error flag or a satisfaction score to sort on.

Out of the refund agent’s 50,000 weekly runs, a reasonable pull is around 500: a random slice plus every run that hit an error or a low satisfaction score. Whatever strategy you pick, run it on a cadence, weekly to start, rather than treating it as a one-off export you remember to do after something breaks in production.

Step 3: how do you label runs against a spec?

You label runs by writing down what good looks like as a checklist of criteria, then scoring every sampled run against that checklist instead of by gut.

This is the step almost everyone skips, and it’s the one that decides whether the dataset is worth anything.

The instinct is to label by gut: skim a run, decide it feels fine, move to the next one. That works at a hundred runs and falls apart completely at ten thousand, mostly because “feels fine” means something slightly different to every reviewer and drifts over time even for the same reviewer. The alternative is writing down, explicitly, what good looks like, and scoring every sampled run against that spec.

You’re not starting from nothing here. Your agent’s own codebase already encodes most of what it’s supposed to do: the outputs it should produce, the tools it’s allowed to call, the checks it runs before acting, the paths it should and shouldn’t take. Read the code first and most of the labeling criteria fall directly out of it.

For the refund agent, that read produces a checklist along these lines.

• Refund within terms. Amount within the order value and the 30-day window. Violation example: refunded a 90-day-old order.
• No invented terms. Only cites the published refund terms. Violation example: quoted a returns rule that does not exist.
• Escalate disputes. Hands chargebacks off to a human. Violation example: auto-refunded a disputed charge.

Automating this scoring step is where LLM-as-judge techniques have become the practical default, since manually reviewing thousands of runs against a rubric doesn’t scale. The approach has real limits worth knowing before you lean on it: research comparing LLM judges against human-labeled relevance data found strong rank correlation but only fair agreement on exact labels, and accuracy drops sharply on the more nuanced categories rather than the easy pass/fail calls. A survey of LLM-as-judge methods notes it emerged specifically because manually assessing helpfulness in training data got too expensive to do at scale by hand, which is exactly the tradeoff a labeling pipeline is making. Overmind runs this scoring as evaluators: LLM judges that each check one sentence of your rubric, and deterministic checks in code. In an eval run every variant is scored against a baseline you choose, so a changed rubric or a changed prompt is a comparison rather than a new number.

The runs that fail a criterion, whatever the customer clicked afterward, are the highest-value training data you have. The labels encode your judgment about what good looks like, and that judgment is the one thing no generic tool can supply for you. Do the first pass yourself for the first month. It’s the fastest way to find out what your criteria actually are, as opposed to what you assumed they were when you wrote the checklist.

Step 4: how do you build the training set?

You build the training set by keeping the runs your labels approve, shaping each one into the format your training method needs, and attaching provenance to every row.

Labeled runs aren’t training examples yet. The last step shapes them for whatever method you’re about to run, and the method decides the shape.

• Which runs you keep. SFT: runs that passed every criterion. RL: sampled runs with their criterion scores attached.
• What a row holds. SFT: an input-output pair of the behavior you want repeated. RL: the run plus the scores, used as a reward signal.
• What the model learns. SFT: to imitate a fixed set of good examples. RL: to produce runs that score higher.
• What labeling has to produce. SFT: a pass/fail verdict per criterion. RL: a usable score per criterion, not just pass/fail.

Either way, the format needs to be consistent and machine-readable, with the goal and outcome attached to every run. Provenance belongs in the format too. In Overmind’s Datasets, every row keeps its link to the trace it came from, so you can always answer where a training example came from, and training uses that link to warn you when an evaluation row shares a trace with a training row. Overmind trains with supervised fine-tuning, LoRA or full, so the left-hand column is the shape its training datasets take.

This is also the stage where the case for smaller, specialized models gets concrete. Overmind’s own research argues that every model invocation inside an agentic workflow is a natural source of high-quality training data, precisely because the prompts are narrow and well-defined and the pass/fail signal is clean, unlike open-ended chat data. A team that instruments its model calls, clusters the resulting patterns, and fine-tunes a specialist model on them ends up with a system that improves with every production run instead of one that’s frozen at whatever a general-purpose model happened to learn at pretraining time.

Which method you feed, supervised fine-tuning or RL, is its own separate decision. The dataset from steps 1 through 3 is what feeds either one.

Which tools turn production traces into datasets?

Overmind builds evaluation and training datasets from production traces and trains an open-weight model on them in the same product. LangSmith and Langfuse add production traces to datasets for evaluation and export them for training elsewhere. Label Studio is a labeling tool you feed exported runs into.

Here is what each one does at this stage, taken from its own documentation.

• Overmind selects production traces into a dataset from Observability, or imports them from a Langfuse, LangSmith, Braintrust or Galileo connector. Each run lands as one row with its input, delivered output, full transcript, tool calls, model, tokens, cost and score. Your coding agent then prepares the rows as readable Python steps that Overmind runs and records, before the dataset feeds an eval run or a training job on an open-weight base model (datasets docs, training docs).
• LangSmith datasets are versioned, the LangSmith Engine can generate ground-truth examples from production traces, and a dataset exports to CSV, JSONL or OpenAI’s fine-tuning format; training happens elsewhere (manage datasets docs).
• Langfuse datasets add items from a trace or in batch from the observations table, keep a source_trace_id on each item, and export CSV or JSON from the UI; Langfuse is open source and does not train models (datasets docs, export docs).
• Label Studio is an open-source labeling platform that now lists agent traces and LLM evals among its data types; you export runs into it, label them there, and export the labels back out. It does not capture traces or train models (labelstud.io).

Prices and formats move; every claim above was checked against the linked page on 5 October 2026.

How do you do it in Overmind?

Four steps, from a running agent to a dataset a training job can read.

1. Capture. Add overmind.init(providers="auto") to your Python agent, or point any OpenTelemetry exporter at Overmind. From Cursor, Claude Code, OpenCode or Codex, /overmind ensure-tracing checks the instrumentation for you. Traces already in Langfuse, LangSmith, Braintrust or Galileo come in through a connector.
2. Select. In Observability, filter the root traces you want and choose Add to dataset, or Data from traces to take everything matching the filter. Pick Evaluation, Training, or Train + eval, which splits the source with a 30% evaluation share by default and keeps matching inputs, traces and conversations on the same side.
3. Shape. Open the dataset in the Data Workshop. Your coding agent prepares the rows as Python steps, and each step is a version, frozen when a run uses it. Start with /overmind dataset.
4. Use. Start an eval run on the evaluation half, or /overmind finetune on the training half. The datasets docs cover every option, and the quickstart gets a first trace in within a few minutes.

Why is the dataset the hard part?

The dataset is the hard part because it has to be rebuilt every week against new traces and sharper criteria, and most tooling stops at handing you the traces.

None of these four steps is exotic on its own. What makes the whole thing difficult is that it never stops. Production keeps producing new traces, your criteria keep getting sharper as you find edge cases you didn’t anticipate, and the dataset has to be rebuilt against what users actually did this week, not what they did last quarter. The dotted feedback line on that diagram at the top is the entire job.

It’s also exactly the gap most observability tooling leaves open. Datadog’s own writeup on GenAI tracing gets at this directly: teams are encouraged to promote interesting production traces into curated, version-controlled “golden” datasets and layer evaluation metadata on top, which is essentially this same capture-to-label pipeline described from the observability side. An observability platform hands you the traces and stops there. Stitching production traces to behavioral training data to a deployed, improved model is work that mostly happens in spreadsheets and one-off scripts today, and Overmind is built to close that gap, running capture, dataset, eval and training in one open-source platform instead of leaving it as a manual export.

Owning that labeled dataset matters because differentiation in agentic AI increasingly lives in proprietary behavioral data, not in which foundation model you call. Anyone can capture traces. The labeled dataset built from them, tuned to your own definition of correct, is the thing actually worth owning.

Frequently asked questions

What is the difference between a trace and a training example?

A trace is the full record of one agent run, from the input through every model and tool call to the output and its timing. A training example is a curated row the model should learn from, usually an input paired with the output you want. Most traces never become training examples; sampling and labeling decide which do.

Should failed runs go into a fine-tuning dataset?

Not as they are. Supervised fine-tuning teaches a model to copy its examples, so failed runs teach failure. Use them as evaluation cases, or correct the output first and train on the corrected version. Keep the successful runs that represent the work you want repeated.

Can an LLM label my traces for me?

For a first pass, yes. An LLM judge scoring each run against a short written rubric is the usual start. Check a sample against human labels before trusting it, and break the rubric into small yes/no checks so you can see which one disagrees. In Overmind, each rubric sentence is its own judged check.

How do i keep evaluation data out of my training data?

Split by run, not by row, so steps from one trace or conversation never land on both sides. In Overmind, a Train + eval dataset splits with a 30% evaluation share by default and keeps matching inputs, traces and conversations together; training reports any overlap it finds.

Which tools turn production traces into training datasets?

Overmind builds evaluation and training datasets from production traces, then trains an open-weight model on them. LangSmith and Langfuse can add production traces to datasets for evaluation, and LangSmith exports datasets in OpenAI’s fine-tuning format. Label Studio is a labeling tool you feed exported runs into.

Last updated 5 October 2026.

Overmind is the model training platform for AI teams. It turns your production traces into specialized models you own. Get started.