Every agentic system can create its own training data.
Every agentic AI system you run is quietly writing its own training set, and most teams never use it.
When an agent runs, it emits a stream of inputs and outputs: the prompt, the tool calls, the reasoning and the final answer. That stream is a trace, and once you standardise how you capture it (the OpenTelemetry GenAI semantic conventions is the default schema here) you have raw telemetry. A training dataset is narrower: a curated set of runs, scored against a spec, designed for fine-tuning jobs.
Building a dataset like this is a four-stage pipeline.
Here's a worked example, using a refund agent, to show what each stage actually involves.
Step 1: Capture every run as a trace
If your agent is instrumented at all, you already have this part. Every run generates a trace: the goal it was given, every model call, every tool invocation, every observation from the environment, and the eventual outcome. That trace is the raw material everything downstream depends on.
The industry is converging on a shared vocabulary for this. OpenTelemetry's GenAI working group has been building standard span attributes for LLM calls since 2024, covering model name, token counts, latency, and tool execution as first-class fields rather than something each team invents from scratch. That matters more than it sounds like it should, because a trace schema you have to redesign every time you switch observability vendors is a trace schema nobody trusts enough to build a dataset on top of.
This is also where a tool like Overmind tends to sit. Its SDK wraps the model call interface directly, so a single overmind.init() call captures every LLM invocation across OpenAI, Anthropic, Google Gemini and Agno, logging inputs, outputs, latency, token counts and errors without extra plumbing on your side. The point isn't the SDK itself. It's that capture has to be automatic and total, or the sampling and labelling stages downstream never get the raw material they need.
Step 2: Sample, because you cannot label everything
You should not try to label every run. Most of what an agent does in a given week is unremarkable, and reviewing all of it teaches a labelling team nothing it didn't already know. The job at this stage is picking which runs are worth a human's attention.
Three strategies cover most cases:
- Random sampling gives you an honest baseline of what the agent does on an ordinary day.
- Stratified sampling pulls deliberately from cases you care about: a specific refund reason, a customer tier, a tool that keeps timing out.
- Failure-weighted sampling oversamples the runs that went wrong, because those carry the most signal per run.
Out of the refund agent's 50,000 weekly runs, a reasonable pull is around 500: a random slice plus every run that hit an error or a low satisfaction score. Whatever strategy you pick, run it on a cadence, weekly to start, rather than treating it as a one-off export you remember to do after something breaks in production.
Step 3: Label against what good actually looks like
This is the step almost everyone skips, and it's the one that decides whether the dataset is worth anything.
The instinct is to label by gut: skim a run, decide it feels fine, move to the next one. That works at a hundred runs and falls apart completely at ten thousand, mostly because "feels fine" means something slightly different to every reviewer and drifts over time even for the same reviewer. The alternative is writing down, explicitly, what good looks like, and scoring every sampled run against that spec.
You're not starting from nothing here. Your agent's own codebase already encodes most of what it's supposed to do: the outputs it should produce, the tools it's allowed to call, the checks it runs before acting, the paths it should and shouldn't take. Read the code first and most of the labelling criteria fall directly out of it.
For the refund agent, that read produces a checklist along these lines:
- Refund within terms. Amount within the order value and the 30-day window. Example violation: refunded a 90-day-old order.
- No invented terms. Only cites the published refund terms. Example violation: quoted a returns rule that does not exist.
- Escalate disputes. Hands off chargebacks to a human. Example violation: auto-refunded a disputed charge.
Automating this scoring step is where LLM-as-judge techniques have become the practical default, since manually reviewing thousands of runs against a rubric doesn't scale. The approach has real limits worth knowing before you lean on it: research comparing LLM judges against human-labelled relevance data found strong rank correlation but only fair agreement on exact labels, and accuracy drops sharply on the more nuanced categories rather than the easy pass/fail calls. A survey of LLM-as-judge methods notes it emerged specifically because manually assessing helpfulness in training data got too expensive to do at scale by hand, which is exactly the tradeoff a labelling pipeline is making. Overmind runs this scoring as evaluators against a rubric you write. Six evaluator kinds cover it, from a deterministic check to an LLM judge, and every run is scored against a baseline before the result counts.
The runs that fail a criterion, whatever the customer clicked afterward, are the highest-value training data you have. The labels encode your judgment about what good looks like, and that judgment is the one thing no generic tool can supply for you. Do the first pass yourself for the first month. It's the fastest way to find out what your criteria actually are, as opposed to what you assumed they were when you wrote the checklist.
Step 4: Build the training set
Labelled runs aren't training examples yet. The last step shapes them for whatever method you're about to run, and the method decides the shape.
- For supervised fine-tuning, take the runs that passed every criterion and turn them into input-output pairs of the behaviour you want repeated.
- For reinforcement learning, keep the criterion scores themselves as the reward signal, so the model learns to produce runs that score higher rather than just imitating a fixed set of examples.
- Either way, the format needs to be consistent and machine-readable, with the goal and outcome attached to every run.
This is also the stage where the case for smaller, specialised models gets concrete. Overmind's own research argues that every model invocation inside an agentic workflow is a natural source of high-quality training data, precisely because the prompts are narrow and well-defined and the pass/fail signal is clean, unlike open-ended chat data. A team that instruments its model calls, clusters the resulting patterns, and fine-tunes a specialist model on them ends up with a system that improves with every production run instead of one that's frozen at whatever a general-purpose model happened to learn at pretraining time.
Which method you feed, supervised fine-tuning or RL, is its own separate decision. The dataset from steps 1 through 3 is what feeds either one.
Why this is the hard part
None of these four steps is exotic on its own. What makes the whole thing difficult is that it never stops. Production keeps producing new traces, your criteria keep getting sharper as you find edge cases you didn't anticipate, and the dataset has to be rebuilt against what users actually did this week, not what they did last quarter. The dotted feedback line on that diagram at the top is the entire job.
It's also exactly the gap most observability tooling leaves open. Datadog's own writeup on GenAI tracing gets at this directly: teams are encouraged to promote interesting production traces into curated, version-controlled "golden" datasets and layer evaluation metadata on top, which is essentially this same capture-to-label pipeline described from the observability side. An observability platform hands you the traces and stops there. Stitching production traces to behavioural training data to a deployed, improved model is work that mostly happens in spreadsheets and one-off scripts today, and it's the specific gap platforms like Overmind are built to close, running the optimise-evaluate-accept loop end to end instead of leaving it as a manual export.
Owning that labelled dataset matters because differentiation in agentic AI increasingly lives in proprietary behavioural data, not in which foundation model you call. Anyone can capture traces. The labelled dataset built from them, tuned to your own definition of correct, is the thing actually worth owning.
Overmind is the model training platform for AI teams. It turns your production traces into specialised models you own. Get started.
