Traces from an observability tool can train a model that does one of your agent's jobs, but only a small slice is worth training on. Keep runs with a good outcome, remove duplicates and personal data, and test on data the model never saw. This guide shows each step, with export code for Langfuse and LangSmith.
If you run an agent in production with Langfuse or LangSmith attached, you already hold a record of it doing its job thousands of times. Each trace has the input, every model and tool call, and the output. The export is the easy part of training on that record. The work is in choosing which traces to keep and cleaning them, then testing the result honestly.
The steps below assume your traces already exist. If you're still deciding what to capture or how to label it, start with how to turn traces into a training dataset.
Train one step, not the whole agent
An agent run usually contains several model calls with different jobs, such as classifying the request and writing the reply. Train a model for one of those calls at a time. A single step has a defined input and output and an obvious test of success, so the dataset is smaller and the result is easy to measure and swap in.
Pick the step that runs most often, or the one where mistakes cost the most. In your tracer it shows up as an observation or run with a stable name, such as classify_ticket or extract_invoice. The export filters below all start from that name.
Decide which traces are worth training on
Supervised fine-tuning teaches a model to copy its examples, mistakes included. So the question for each trace is whether its output is one you want repeated. Your tracer usually holds four signals that answer it, from weakest to strongest.
- No error. The call returned valid output without a retry. Necessary, but a confident wrong answer passes too.
- An eval score. An LLM judge or a code check scored the output. Useful at volume, once you have checked a sample of its verdicts against human ones.
- User feedback. A thumbs up, or a ticket closed without escalation.
- A human correction. Someone fixed the output. The corrected version is the best training target you can get, because it is exactly what you wanted.
Keep traces that pass the signals you trust, and put failures in your test set rather than your training set. A failure with a human correction is the exception: train on the corrected output.
Check where the outputs came from before you train on them. When a GPT or Claude model wrote them, the vendor's terms apply. OpenAI's usage policies, effective November 20, 2026, bar using its outputs "to generate, prepare, select, or supply training material or signals for models that are intended as alternatives to or substitutes for OpenAI models or services." Anthropic's usage policy, effective November 12, 2026, prohibits using outputs to train an AI model without Anthropic's prior authorization. Outputs your team wrote or corrected are yours. If you're unsure, ask your legal team before you train. Teams moving off OpenAI fine-tuning face the same question, and the migration playbook covers what they can take with them.
Export the traces
Each exported model call holds the messages your agent sent and the reply it got. You'll turn each one into a chat row, with the accepted or corrected output as the final assistant message.
Both tools changed their read APIs in 2026, so tutorials that call langfuse.api.trace.list or client.list_runs are out of date. Check your retention window before anything else. LangSmith's base tier keeps traces for 14 days and Langfuse's Hobby plan shows 30 days of history, so a slow start can cost you data.
From Langfuse
Langfuse stores each trace as a tree of observations, and its Observations API returns them in pages. It can't filter observations by score, so fetch the scores you trust from the Scores API first, then keep the model calls from those traces. Human corrections are stored as scores with the data type CORRECTION, and they make the best training targets.
from datetime import datetime, timedelta, timezone
from langfuse import get_client # langfuse>=5, Python 3.11+
langfuse = get_client() # reads LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY
end = datetime.now(timezone.utc)
start = end - timedelta(days=30)
# 1. Traces that got positive user feedback (100 scores a page at most).
good_traces, cursor = set(), None
while True:
page = langfuse.api.scores.get_many(
name="user-feedback", data_type="NUMERIC", value="1", # 1 = thumbs up
fields="subject", from_timestamp=start, to_timestamp=end,
limit=100, cursor=cursor,
)
for score in page.data:
if score.subject and score.subject.kind == "trace":
good_traces.add(score.subject.id)
elif score.subject and score.subject.trace_id:
good_traces.add(score.subject.trace_id)
cursor = page.meta.cursor
if not cursor:
break
# 2. The model calls for one step, kept only from those traces.
calls, cursor = [], None
while True:
page = langfuse.api.observations.get_many(
type="GENERATION", name="classify_ticket",
from_start_time=start, to_start_time=end,
fields="core,basic,io", limit=500, cursor=cursor,
)
calls += [obs for obs in page.data if obs.trace_id in good_traces]
cursor = page.meta.cursor
if not cursor:
break
print(len(calls), "calls from", len(good_traces), "traces")input and output come back as raw strings, so json.loads them before you build chat rows. If you'd rather not write code, filter a Langfuse table by score in the UI and export it as CSV or JSON. Paid plans can also schedule exports to S3, Google Cloud Storage or Azure. Langfuse doesn't train models itself. Its Fireworks guide curates a dataset by score and converts it to chat JSONL for training on Fireworks.
From LangSmith
LangSmith's runs.query replaces list_runs, which LangSmith Cloud removes on January 31, 2027. Its filter language can match feedback directly, and trace_filter keeps the model calls from traces whose root run got a thumbs up.
import asyncio
from datetime import datetime, timedelta, timezone
from langsmith import Client # langsmith>=0.10.15
async def export_calls():
client = Client() # reads LANGSMITH_API_KEY
project = client.read_project(project_name="support-agent")
calls = []
async for run in client.runs.query(
project_ids=[str(project.id)],
# Without this, runs.query only looks back one day.
min_start_time=datetime.now(timezone.utc) - timedelta(days=14),
run_type="LLM",
filter='eq(name, "classify_ticket")',
trace_filter='and(eq(feedback_key, "user_score"), eq(feedback_score, 1))',
selects=["ID", "TRACE_ID", "START_TIME", "INPUTS", "OUTPUTS"],
):
calls.append(run)
return calls
calls = asyncio.run(export_calls())
print(len(calls), "calls")If you've already curated a LangSmith dataset, you can download it in OpenAI's fine-tuning format from the UI or with client.read_dataset_openai_finetuning(dataset_name=...). Bulk export of whole projects to S3 as Parquet needs the Enterprise plan for accounts created after August 3, 2026. LangSmith also fine-tunes models itself. Its smithtune command-line tool, in beta since September 2026, trains a LoRA model on Fireworks or Baseten with your own provider key and compares the base and tuned models in a LangSmith experiment.
Clean the data before you train
Remove duplicates. Production traffic repeats itself, and a template that runs thousands of times a day can swamp everything else in the dataset. Drop exact duplicates by hashing the normalized input, then near-duplicates, such as the same request with a different order ID. In a study of language model training data, Lee et al. found that after deduplication, models repeated memorized text ten times less often and reached the same or better accuracy in fewer training steps.
Scrub personal data. A language model can repeat text from its training data word for word (Carlini et al., 2021), so names, emails and account numbers in your traces should never reach the training set. Both tracers can mask data before it leaves your app, so new traces arrive clean. Langfuse's Python SDK accepts a mask_otel_spans function, and LangSmith's client accepts an anonymizer or can hide inputs and outputs entirely. For traces you already have, run a scrubber over inputs and outputs before the data leaves your systems. The open-source Presidio library finds personal data such as names and email addresses and replaces it with placeholders:
from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine
analyzer, anonymizer = AnalyzerEngine(), AnonymizerEngine()
def scrub(text: str) -> str:
found = analyzer.analyze(text=text, language="en")
return anonymizer.anonymize(text=text, analyzer_results=found).text
print(scrub("Refund order 4417 for Jane Smith, [email protected]"))
# Refund order <DATE_TIME> for <PERSON>, <EMAIL_ADDRESS>Placeholders like these let the model learn the shape of a request without the values. Check a sample by hand, because detectors make mistakes both ways. Above, Presidio took the order number for a date, and it can just as easily miss an internal account number.
Check the labels. If your targets come from outcomes rather than human edits, read 50 at random against a written definition of a correct output. If more than a few are wrong, fix the rule that produced them before you train.
SFT, preference data or RL: what your traces support
The training method follows from what your traces contain.
| Method | What each example needs | Where it comes from in traces | Use it when |
|---|---|---|---|
| Supervised fine-tuning (SFT) | An input and one good output | Accepted or corrected outputs | You can say what the right output is |
| Preference training (DPO) | An input, a better output and a worse one | Edits (edited version vs original), regenerations, A/B picks | You can tell better from worse but rarely have a perfect answer |
| Reinforcement learning (RL) | An input and a scoring function | Inputs from traces; the score comes from code you write | Correctness can be checked by code and SFT has stalled |
DPO trains on pairs of answers to the same prompt, one preferred and one not. GRPO, introduced in the DeepSeekMath paper, samples several answers to each prompt and scores them against each other, so it needs a scoring function rather than stored answers. Start with SFT. It needs the simplest data, and it is the baseline any other method has to beat. The guide to types of fine-tuning covers each method in more depth.
How many examples you need
There is no universal number. OpenAI's fine-tuning guide sets a minimum of 10 examples and recommends starting with 50 well-crafted ones. The research favors quality over volume. LIMA tuned a 65B model's chat style with 1,000 curated examples, and AlpaGasus beat a model trained on 52,000 instructions using 9,000 filtered ones. Across the 31 tasks in LoRA Land, the size of the training set did not predict how much fine-tuning helped.
Your own data answers the question better than any rule of thumb. Train on a quarter of your examples, then half, then all of them, and score each model on the same test set. If the score is still climbing at full size, more data will help. If it flattened early, fix the quality of the data or the definition of the task instead of adding more.
Keep the test set out of the training data
A model that has seen the test questions will look better than it is. With traces this happens easily, because the same user, session or template shows up many times.
- Split by session or user, not by row. Steps from one conversation should never land on both sides.
- Split by time when you can. Train on older traces and test on the newest week. That matches how the model will be used, on traffic it has never seen.
- Remove near-duplicates across the split, not only within the training set.
- Freeze the test set. Tune settings on a separate validation slice. Each look at the test set while you iterate leaks a little of it into your choices.
Exact-match checks are not enough on their own. Yang et al. showed that paraphrased test questions slip past n-gram decontamination, and that a 13B model trained on them can match GPT-4 on the benchmark it has effectively seen.
import json
rows = [json.loads(line) for line in open("clean.jsonl")]
cutoff = "2026-11-01" # the newest traffic becomes the test set
train = [r for r in rows if r["timestamp"] < cutoff]
test = [r for r in rows if r["timestamp"] >= cutoff]
# Drop any training row from a session that also appears in the test set.
test_sessions = {r["session_id"] for r in test}
train = [r for r in train if r["session_id"] not in test_sessions]
print(len(train), "train,", len(test), "test")Measure the result against what you run today
Score three models on the same frozen test set with the same checks. The base model before training shows what training added. The model your agent calls today is the bar to clear. Your trained model has to beat the first and at least match the second.
Use the check that fits the step. Exact match or F1 suits labels and extracted fields, schema validation suits JSON, and an LLM judge with a short yes-or-no rubric suits free text. Break the score down by segment as well, because an average can hide a rare case that got worse. Then put the trained model on live traffic in shadow mode, answering copies of real requests while users still see the current model, before you switch.
The same work in Overmind
Overmind runs this work in one place, and your coding agent can drive each step.
- Import. Connect Langfuse, LangSmith, Braintrust or Galileo on the Integrations page with the provider's API key and choose how far back to import. The traces land in Observability, and Overmind can keep polling for new ones. It imports the traces, not the scores or feedback attached to them, so filter on those in your tracer or recreate the checks as Overmind evaluators. Python agents can also send traces with the Overmind SDK, and any other stack over OpenTelemetry.
- Select and split. In Observability, select traces and click Add to dataset. Each trace becomes one row. When you split rows into training and test sets, Overmind removes exact duplicates and keeps rows with the same input, trace or conversation on one side. If training rows still overlap your test set, training warns you.
- Prepare. In the Data Workshop, your coding agent writes each cleaning step as a small Python script, which Overmind runs in an isolated sandbox and versions. Overmind does not detect personal data for you, so scrub it before import or in one of these steps.
- Evaluate. Write evaluators as code checks or model judges. The ones you enable score imported traces as they arrive.
- Train and compare. Overmind trains with supervised fine-tuning, either LoRA or, on models up to 14B, full fine-tuning. It can score the trained model against its untouched base on your test set, and against the model you use today if OpenRouter serves it.
- Serve. Call the model with the OpenAI SDK at
https://api.overmindlab.ai/api/v1, or download the weights and run them yourself.
Have traces in Langfuse or LangSmith? Your coding agent can set this up for you. Open your agent's repo in Claude Code, Cursor, Codex or OpenCode and paste Set me up on Overmind: https://console.overmindlab.ai/setup.md into the chat. Or start for free and connect your tracer from the console.
FAQ
Can I fine-tune a model directly on my Langfuse traces?
Not inside Langfuse, which records and evaluates traces but doesn't train models. Export the traces, filter and clean them, then train them with a fine-tuning tool. Overmind can import them straight from Langfuse.
Does LangSmith fine-tune models?
Yes. LangSmith Fine-Tuning, in public beta since September 2026, uses the smithtune command-line tool to run supervised LoRA fine-tuning on Fireworks or Baseten with your own provider key. It then compares the base and tuned models in a LangSmith experiment, and can deploy the tuned model.
How many traces do I need to fine-tune a model?
Fewer than you might expect, if they're clean. OpenAI's guide recommends starting with 50 well-crafted examples. In research, LIMA tuned a model's chat style with 1,000 curated examples, and AlpaGasus beat a 52,000-example dataset with 9,000 filtered ones. Train on growing slices of your data to see where your own curve flattens.
Should I train on traces where the agent failed?
Not as they are. Supervised fine-tuning copies its examples, so failed runs teach failure. Put them in your test set, or train on a corrected version.
Can I train on traces where GPT or Claude wrote the output?
Check the vendor's terms first. OpenAI's usage policies from November 20, 2026 and Anthropic's from November 12, 2026 restrict using their outputs to train other models. Outputs your team wrote or corrected are yours. Ask your legal team if you're unsure.
Overmind is the model training platform for AI teams. Start for free.
