Learnings from building, deploying and improving agents in production.
If you're running an LLM-powered agent in production, the seven patterns below show up again and again.
Instrument every agent trace
Every agent run generates a trace: the full sequence of inputs, tool calls, retrieved context, intermediate reasoning, and final output. Without that record, you're reconstructing agent behaviour from scattered log files, which is miserable work.
OpenTelemetry has become the default standard for this. Its GenAI semantic conventions set one shared format for what each step of a run records, including model name, token usage, and tool calls. A trace from a LangChain agent then looks the same as one from a raw API call. Most observability platforms now speak this format natively, including Langfuse, Arize Phoenix, Datadog LLM Observability, and LangSmith.
Tactical advice: pick one OpenTelemetry-compatible tracing tool. Route every agent run into it. Keep the traces for at least 30 days. Tag them with user ID, use case, and agent version, so you can filter them later without going back to change your tracing code.
Build evals from real traffic
Before deploying an agent, run it against some evals. An eval is a scored check of whether the agent did the job. However, when real traffic arrives, so will edge cases.
Production traces can be a real eval source. Sample traces weekly and turn the failures into new test cases. Older eval sets can become a regression check, to confirm old fixes still hold, rather than your source of truth. Langfuse, Braintrust, Arize, and LangSmith all support this workflow directly, turning sampled production data into structured eval sets.
Tactical advice: by week two in production, your primary eval set should be built from real traces, not synthetic ones written before launch.
Turn traces into a labelled dataset
A trace only records what happened. Before you can train or evaluate on it, you need to know whether the run was any good. That judgment is the label, and labels are what a dataset is actually made of. More unlabelled traces will not make it better, so label what you have before you collect more.
Start with your agent's codebase. It already spells out what the agent is meant to do, so turn that into a checklist of pass or fail criteria. Score traces against the checklist and write a note on each one that fails. Hamel Husain calls this error analysis. Group those notes into a failure taxonomy, a fixed list of failure types you can count and track over time.
Once every trace carries a label, and a failure type where it failed, you have a dataset. That is what you fine-tune on, and what you score every later change against.
Tactical advice: draft 5 to 20 pass/fail criteria straight from the code, then refine them as you label. Spend a few hours a week labelling yourself for the first month. Don't outsource the first pass. Your judgment about what "good" looks like only shows up once you've sat with the traces yourself.
Prompt engineering has a ceiling
Prompt iteration is cheap and fast, so use it for as long as it keeps improving your eval score. Eventually each new instruction fixes yesterday's bug and quietly introduces a new one. The system prompt bloats, latency creeps up, and the model starts ignoring parts of it altogether.
At that point you're choosing between more tweaking or a different lever entirely. Most teams keep tweaking a month longer than they should, because prompt edits feel free. They aren't. Engineering time is the most expensive line item in the whole pipeline.
Tactical advice: track your eval score before and after every prompt change. First run the eval twice on an unchanged prompt to see how much the score wobbles on its own. Ignore any change smaller than that.
Fine-tuning doesn't need a research lab
Fine-tuning an open-weights model used to mean a research team and a large GPU budget. LoRA freezes the base model and trains a small set of low-rank adapter matrices instead. Those adapters are often under 1% of total parameters. That is why a 70B model can be fine-tuned on a single high-memory GPU, using QLoRA's 4-bit quantisation. Quantisation stores the weights at lower precision, so the model takes up less memory.
Pick an open-weights base, run a LoRA pass over labelled trajectories, meaning recorded agent runs you have already scored. Then serve the result on Fireworks, Together, or Modal. Unsloth, OpenPipe, and Predibase handle most of the pipeline for you.
Tactical advice: if you have 1,000+ labelled trajectories and a narrow task, run a LoRA fine-tune. Compare it against your prompt-engineered baseline on the same eval set.
Freeze the eval set before you train
The eval set you score a fine-tune against has to be the same one you scored the baseline against, and it has to be locked before training starts. Change the criteria and the training run in the same week and you cannot tell which one moved the number.
Test the model inside the agent harness, not on its own. A model that scores better in isolation can still make the agent worse, because the agent depends on tool call formats, output structure, and instruction following that a benchmark score does not capture.
Tactical advice: every model change runs against the same frozen eval set, every time. Models regress for all sorts of reasons, and a fixed scoreboard is the only way you find out.
The self-improvement loop
The first full pass through capture, label, train, deploy, and measure takes weeks. By the fifth pass, most of the friction is gone and the loop runs in hours.
That compounding is the whole point. A team that runs this loop five times in a quarter ends up with an agent that matches their business needs at a fraction of frontier API cost. A team that runs it once, ships, and calls it done is roughly where it started six months ago. This is essentially Overmind's product thesis. It connects your repository and traces, then proposes changes through its optimiser. It tests those changes against your labelled data and only surfaces the ones that actually move your score. The loop from evidence to shipped improvement stays short, instead of becoming its own six-week project.
Tactical advice: pick a cadence and put it on the calendar. Weekly is aggressive but doable. Monthly is the floor. Anything slower and you're not really running a loop, you're running occasional cleanup.
Infrastructure is more work than the agent
Any one of these seven steps is a real job on its own. Strung together, they're several jobs. Most teams find the surrounding infrastructure costs more engineering time than the agent.
Overmind closes that gap. It reads your agent's codebase to understand what it does. Then it handles trace capture, labelling, fine-tuning, deployment, and measurement as one connected system, instead of five separate vendor integrations.
If you're past the prototype stage and want the loop running without building it from scratch, it's worth a look.
Overmind is the model training platform for AI teams. It turns your production traces into specialised models you own. Get started.
