# Overmind: full site content

> Turn your agents’ production data into specialized models you own. Prepare data, train, evaluate, and deploy from one platform.

Every page listed in https://www.overmindlab.ai/llms.txt, in the same order, as Markdown. Each section is headed by the page's canonical URL. Append .md to any page URL to fetch one page on its own.

# https://www.overmindlab.ai/product/model-training

# Own your model

> Canonical: https://www.overmindlab.ai/product/model-training

*Model training*

Fine-tune a model on your own data, measure it with your own evaluators, and serve it through an OpenAI-compatible endpoint. Keep the training data, benchmark and deployment connected from the first experiment to production.

## From dataset to a model you can deploy

Choose the experiment, inspect the evidence, and decide when the model is ready for traffic.

### Configure from the data you have

Choose training, validation and evaluation data, then inspect model recommendations, compatibility, cost and duration estimates. Launch multiple experiments with different base models or hyperparameters. Use LoRA adapters or full fine-tuning where the selected model supports them.

`Model recommendations` `Cost estimates` `LoRA & full training`

### Follow every stage of the run

Track preparation, training and deployment in one place. Inspect train and validation loss, learning rate, token accuracy and gradient norm alongside steps, tokens and credits. Each experiment keeps its configuration and data version linked to the resulting model.

`Live metrics` `Experiment history` `Pinned data`

### Choose what you benchmark against

Select the codebase incumbent or a ready trained model as the benchmark, independently of live serving. Choose which before-and-after evaluations to run. Comparisons use pinned eval data, prompt snapshots and graders; inspect per-metric deltas and sample results before making a deployment live.

`Benchmark selection` `Frozen graders` `Per-metric deltas`

### Serve it. Switch it. Keep the weights.

Successful training deploys a model to hosted inference. Call it from an OpenAI-compatible client, or route a capability alias to a ready deployment. Copy a coding-agent prompt to update your repository, test the endpoint, and download the checkpoint when you want the weights.

`Hosted inference` `Capability aliases` `Downloadable weights`

## What you get

### Training

- **Dataset-aware setup** — Use row shape, token lengths and tool-calling requirements to choose a compatible model and configuration.
- **Multiple experiments** — Compare base models and hyperparameters while keeping training and evaluation inputs explicit.
- **Validation controls** — Choose a validation dataset or split and inspect loss, accuracy and other run metrics.

### Benchmarking

- **Independent benchmark** — Choose the comparison target without changing the model serving live traffic.
- **Before-and-after evaluation** — Select the benchmark and training-model evaluations to run, with the same pinned data and graders.
- **Classification metrics** — For labeled classification tasks, inspect precision, recall, F1 and confusion matrices alongside overall scores.

### Serving and ownership

- **OpenAI-compatible inference** — Use standard chat completions and model endpoints for your hosted fine-tunes.
- **Controlled rollout** — Switch a capability alias to a ready model or use a coding-agent prompt to make a repository change.
- **Downloadable checkpoints** — Keep access to the trained weights, with the model linked to its source job and dataset.

## Your data. Your model.

Train, evaluate and serve specialized models from the same platform.

- [Train your model](https://console.overmindlab.ai)

- Book a call: https://www.overmindlab.ai/contact

## How training works, in the documentation

- [How do I fine-tune a model on my own traces?](https://docs.overmindlab.ai/models/training)
- [How do I turn production traces into a dataset?](https://docs.overmindlab.ai/core/datasets)
- [How do I serve a trained model on an OpenAI-compatible endpoint?](https://docs.overmindlab.ai/models/inference)

---

# https://www.overmindlab.ai/product/agent-testing

# Prove every change before you ship it

> Canonical: https://www.overmindlab.ai/product/agent-testing

*Agent testing*

Generate evaluations from your agent’s context, compare models and prompts on the same data, and test code changes in your own repository. Inspect the scores, evidence and winning diffs before deciding what to ship.

## Measure the result. Inspect the change.

Generated evaluators and reproducible experiments connect production failures to changes in your code.

### Evaluators grounded in the capability

Overmind generates checks from the capability’s purpose, prompt, tools and available trace evidence. Inspect the generated library, edit your own evaluators, and organize them into eval sets. Generative testing and live trace scoring use the roles appropriate to each evaluator.

`Generated evaluators` `Eval sets` `Trace scoring`

### Compare on the same evidence

Evaluate recorded outputs or generate fresh responses from model and prompt variants. Each run pins its dataset version and evaluator snapshots. Compare against a baseline, inspect per-metric scores and judge reasoning, and add human annotations to samples that need review.

`Pinned data` `Baseline comparison` `Human annotations`

### Improve prompts, tools and code

Run /overmind optimize from your agent’s repository. Your coding agent writes candidate diffs and replays them locally; Overmind scores the results against your chosen eval set. Inspect iteration scores and candidate patches, then apply the winning change through your normal code review.

`Local execution` `Candidate diffs` `Measured improvements`

### Test the model before you change it

Use /overmind backtest to replay the same dataset through candidate models while keeping the agent’s prompts and logic fixed. Compare quality alongside latency and cost, and inspect the samples behind the averages before choosing the next model or training experiment.

`Model comparison` `Shared eval set` `Cost & latency`

## What you get

### Evaluation

- **Reusable evaluator library** — Combine LLM judges and deterministic checks in capability-scoped or project-wide eval sets.
- **Reproducible comparisons** — Pin data, prompts and evaluators so results remain tied to what the experiment actually tested.
- **Inspectable results** — Read score rationales and sample outputs, with incomplete generations distinguished from scored results.

### Optimization

- **Repository-level changes** — Test edits to prompts, tool definitions and application logic in the environment where your agent runs.
- **Model backtesting** — Compare models over a shared dataset and evaluation set with your existing harness.
- **Bounded experiments** — Choose iteration and candidate budgets, follow progress, and inspect the selected candidate and its diff.

## Stop guessing, start measuring

From a production failure to a tested change in your repository.

- [Generate your evals](https://console.overmindlab.ai)

- Book a call: https://www.overmindlab.ai/contact

## How evaluation works, in the documentation

- [How do I run an evaluation against a baseline model?](https://docs.overmindlab.ai/agent-testing/eval)
- [How does the optimizer score a change to my agent?](https://docs.overmindlab.ai/agent-testing/optimisers)
- [How do I turn production traces into a dataset?](https://docs.overmindlab.ai/core/datasets)

---

# https://www.overmindlab.ai/product/data-workshop

# Turn real behavior into data

> Canonical: https://www.overmindlab.ai/product/data-workshop

*Data workshop*

Turn production traces and uploaded files into versioned datasets for evaluation and training. A dedicated dataset agent prepares the rows, checks their quality, and records each transformation as an executable Python cell.

## From raw rows to a reproducible dataset

Keep the original source, inspect every transformation, and pin the version each run uses.

### Start with traces or your own files

Select the production activity you want to learn from, upload a file, or bring rows through the API. Choose Train or Eval as the dataset intent and associate a capability where needed. The source lands as supplied, with trace provenance retained for data drawn from production.

`Traces & files` `Train or Eval` `Source provenance`

### An ML engineer for each dataset

Ask the dataset’s own agent to prepare, inspect or reshape your data. It tests transformations before adding executable Python cells, checks the result against the intended use, and surfaces quality findings. Changes that require semantic judgment are presented as concrete proposals for review.

`Dataset chat` `Python cells` `Reviewed proposals`

### See exactly what changed

Each executed cell produces a version with its row and value changes visible. Inspect the script, compare the output with the previous cell, and choose the active version. Evaluations, optimization, training and exports freeze the version they use so later edits cannot change an earlier result.

`Executable history` `Row changes` `Frozen versions`

### Prepare once, use it across the platform

Send an eval dataset into testing or optimization, or use a training dataset to configure a fine-tune. Row shape, token statistics and tool usage inform compatibility and model recommendations. Quality findings stay visible; unreadable or technically incompatible data must be fixed before it can be used.

`Readiness checks` `Token profiles` `Training & evaluation`

## What you get

### Preparation

- **Source preserved** — Keep the original input beside the transformation chain and its derived versions.
- **Agent-assisted transformations** — Clean, restructure and audit rows through the dataset chat, with executable cells you can inspect.
- **Quality checks** — Inspect duplicates, truncated turns, leakage, task alignment and coverage, with findings attached to the data.

### Reproducibility

- **Versioned cells** — See row additions, removals and value changes at each step in the notebook.
- **Pinned downstream data** — A run or export freezes the exact version it reads and links back to it.
- **Console, CLI and MCP** — Inspect datasets through MCP and use the CLI for local uploads and exports.

## Your data, ready for the next run

Prepare a dataset once. Keep the evidence behind every evaluation and model.

- [Build your dataset](https://console.overmindlab.ai)

- Book a call: https://www.overmindlab.ai/contact

## How datasets work, in the documentation

- [How do I turn production traces into a dataset?](https://docs.overmindlab.ai/core/datasets)
- [How do I run an evaluation against a baseline model?](https://docs.overmindlab.ai/agent-testing/eval)
- [How do I send my first trace to Overmind?](https://docs.overmindlab.ai/quickstart)

---

# https://www.overmindlab.ai/product/observability

# Understand your agents

> Canonical: https://www.overmindlab.ai/product/observability

*Observability*

Connect what your agent is built to do with what happens in production. Map capabilities from your code, follow each execution, and investigate failures with the prompts, tools, scores and source behind them.

## From source code to production evidence

One project connects your capability map, instrumentation and execution history.

### A map grounded in your code

Scan your repository with your coding agent and sync its capabilities to Overmind. Prompts, tools, execution paths and input/output contracts stay linked to their source files, giving datasets, evaluations and training the same definition of what the agent should do.

`Capabilities` `Source references` `Execution paths`

### Bring the telemetry you already have

Auto-instrument supported LLM libraries with the Python or TypeScript SDK, send OpenTelemetry spans, or import runs from Langfuse, LangSmith, Braintrust and Galileo. Add capability context to connect the activity back to the right part of your agent.

`OpenTelemetry` `Python & TypeScript` `Connectors`

### Inspect the task, then the evidence

Browse task executions, root traces and sessions with latency, token usage and cost. Inspect scores and verdicts from your active evaluators, then open the trace and span detail to see the inputs, outputs and tool activity that explain the result.

`Task executions` `Span detail` `Live scoring`

### Investigate from your coding agent

Give your coding agent the same production context through MCP. Query failures and traces, inspect capability health, and retrieve instrumentation plans. It can use that evidence to make a focused change in your repository and verify the telemetry it produces.

`Capability health` `Instrumentation plans` `MCP`

## What you get

### Context and collection

- **Local repository discovery** — Map capabilities, prompts and tools from the code your team owns. Sync the result to your project.
- **OpenTelemetry ingestion** — Use the SDKs or existing OTLP instrumentation to send spans without adopting a proprietary trace format.
- **Provider integrations** — Import existing telemetry and configure source projects, capability mappings and sync schedules.

### Investigation

- **Executions, traces and sessions** — Move from a task result to its trace, span tree and conversation context.
- **Scores with evidence** — Inspect evaluator verdicts alongside the recorded outputs and tool calls they assess.
- **Coding-agent access** — Query production evidence and instrumentation guidance through project-scoped MCP tools.

## See your agents clearly

Connect your code and telemetry to the same improvement workflow.

- [Connect your repo](https://console.overmindlab.ai)

- Book a call: https://www.overmindlab.ai/contact

## How tracing works, in the documentation

- [How does Overmind ingest OpenTelemetry traces?](https://docs.overmindlab.ai/core/observability)
- [How do I send my first trace to Overmind?](https://docs.overmindlab.ai/quickstart)
- [How do I turn production traces into a dataset?](https://docs.overmindlab.ai/core/datasets)

---

# https://www.overmindlab.ai/

# The training platform for specialized agents

Turn your agents’ production data into specialized models you own. Prepare data, train, evaluate, and deploy from one platform.

- [Console](https://console.overmindlab.ai)
- [Open source](https://github.com/overmind-core/overmind)
- Book a demo: https://www.overmindlab.ai/contact

> Canonical: https://www.overmindlab.ai/

## An AI lab for your engineering team.

You don’t need a frontier model for every task. Train smaller models for the work your agents do, from labeling emails to calling APIs, and measure them against the model you use today.

## Find the cause of agent failures.

Connect your repository and production traces to see how your agent makes decisions. Inspect the failing step, the tool call, and the context behind it.

[Explore observability](https://www.overmindlab.ai/product/observability)

## Turn real work into training data.

Build datasets from your agents’ production traces. Clean, filter, and prepare examples for training and evaluation, capturing the knowledge specific to your business.

[Explore data workshop](https://www.overmindlab.ai/product/data-workshop)

## Prove improvements before you ship.

Test prompts and models against real tasks from your production data. Compare results with your current model, inspect failures, and catch regressions before deployment.

[Explore agent testing](https://www.overmindlab.ai/product/agent-testing)

## Train a model for your tasks.

Fine-tune an open model on your own data. Overmind handles training and deployment. You own the weights, with the choice to host on Overmind or run it yourself.

[Explore model training](https://www.overmindlab.ai/product/model-training)

## Own the model your product runs on

Create a free account and get started

- [Start for free](https://console.overmindlab.ai)
- Talk to the team: https://www.overmindlab.ai/contact

## Common questions

### What is Overmind?

Overmind is the training platform for specialized agents. It turns your agents' production data into specialized open-weight models you own. You prepare data, train, evaluate, and deploy from one platform.

### How does Overmind turn production traces into a model I own?

Connect your agent with the Python SDK or send traces over OpenTelemetry. Your coding agent maps your agent's prompts, tools, and control flow, and Overmind generates evals from that map. The Data Workshop turns production traces into training data and checks it against your agent's contract.

You can then fine-tune an open-weight model, compare it with your current model on the same eval set, and serve it through an OpenAI-compatible API. You can also download the weights and run them yourself.

### Do I need to rewrite my agent to start?

No. Set up the Python SDK once and it traces OpenAI, Anthropic, Gemini, Agno, and LangChain calls over OpenTelemetry. Mark each agent run with one call, and add decorators where you want more detail. There are no wrapper classes and no proprietary trace format.

### Can I keep LangSmith (or another tracer) and still use Overmind?

Yes. Keep your existing tracer and use Overmind for datasets, training, benchmarking, and serving. Overmind also has its own tracing and evals.

You can import traces from Langfuse, LangSmith, Braintrust, and Galileo, or send them from any OpenTelemetry exporter over OTLP/HTTP.

### Do I need my own ML infrastructure to train and serve?

No. Overmind recommends training settings from your dataset and runs LoRA or, where the model supports it, full fine-tuning on managed infrastructure. You can follow training metrics live and compare the result with your current model on the same eval set.

Serve the result on an Overmind endpoint through an OpenAI-compatible API, or download the weights and serve them yourself. Overmind also gives your coding agent a prompt that switches your app to the new model.

---

# https://www.overmindlab.ai/product/mcp

# Coding agents

> Canonical: https://www.overmindlab.ai/product/mcp

*Your frontier lab in a box*

Give your coding agent access to your production data, evaluations and model training. Go from a question to a better agent, in one conversation.

Guided example with sample data. Background jobs are condensed; results are illustrative excerpts.

### Analyze: Analyze failures

Inspect traces and failed checks.

**You:** Our dispute agent is contesting cases it should investigate. Look at recent failures against our merchant playbook. What is it missing?

**Your coding agent:** I’ll inspect the failed policy checks and the evidence available when each decision was made.

#### inspect_capability_health

Checked Dispute Resolver · last 14 days

Input:

```json
{
  "capability": "9b1e4f7a-2c3d-4e5f-8a90-1b2c3d4e5f60",
  "days": 14
}
```

Example result (excerpt):

```json
{
  "summary": "Evidence handling is a recurring failure in dispute decisions.",
  "days": 14,
  "capability": "9b1e4f7a-2c3d-4e5f-8a90-1b2c3d4e5f60"
}
```

#### query_failures

Found 12 recent policy failures

Input:

```json
{
  "capability": "9b1e4f7a-2c3d-4e5f-8a90-1b2c3d4e5f60",
  "since_days": 14,
  "limit": 20
}
```

Example result (excerpt):

```json
{
  "n": 12,
  "failures": [
    {
      "trace_id": "a903148c1d2946e197a72184bd096739",
      "summary": "Contested a non-receipt claim despite mismatched delivery evidence.",
      "failed_scores": [
        {
          "evaluator": "Policy adherence",
          "reasoning": "Delivery evidence must match the order address. A delivery status alone is insufficient to contest."
        }
      ]
    }
  ]
}
```

**Your coding agent:** I found 12 recent policy failures. I’m opening a non-receipt case to check whether the evidence was missing—or the model ignored it.

#### query_traces

Opened the decision and its supporting tool calls

Input:

```json
{
  "capability": "9b1e4f7a-2c3d-4e5f-8a90-1b2c3d4e5f60",
  "trace_id": "a903148c1d2946e197a72184bd096739",
  "all_spans": true,
  "limit": 20
}
```

Example result (excerpt):

```json
{
  "summary": "4 spans returned.",
  "n": 4,
  "all_spans": true
}
```

#### resources/read

Read the evidence used in the failed decision

Input:

```json
{
  "uri": "overmind://traces/a903148c1d2946e197a72184bd096739"
}
```

Example result (excerpt):

```json
{
  "kind": "trace",
  "trace_id": "a903148c1d2946e197a72184bd096739",
  "span_count": 4,
  "spans": [
    {
      "name": "resolve_dispute",
      "attributes": {
        "input.value": "A $184 order is marked delivered. The carrier scan is at a different postcode, and there is no signature. The customer denies receiving it.",
        "output.value": {
          "decision": "contest",
          "rationale": "The carrier confirms delivery. Submit the tracking record as evidence."
        }
      }
    },
    {
      "name": "lookup_delivery",
      "attributes": {
        "output.value": {
          "status": "delivered",
          "postcode_match": false,
          "signature": null
        }
      }
    }
  ]
}
```

**The evidence was available; the decision ignored it.** In GLD-042, the agent recommended contesting a non-receipt claim.

- **Carrier status:** Delivered.
- **Delivery location:** Postcode mismatch.
- **Signature:** Not available.

The merchant playbook says:

> Delivery evidence must match the order address. A delivery status alone is insufficient to contest.

The correct next step was to **request matching proof of delivery**.

### Prepare data: Prepare data

Build a reviewed training dataset.

**You:** Use the cases our reviewers have resolved to teach it those distinctions. Keep the Golden cases out of training.

**Your coding agent:** I’ll keep the reviewer-approved resolution as the target, with the original evidence as input. Cases without a reviewed answer will be excluded.

#### create_dataset_from_traces

Collected 4,812 resolved-case traces

Input:

```json
{
  "name": "Dispute Resolutions - Train",
  "capability": "9b1e4f7a-2c3d-4e5f-8a90-1b2c3d4e5f60",
  "filters": {
    "capability": "9b1e4f7a-2c3d-4e5f-8a90-1b2c3d4e5f60"
  },
  "intent": "train",
  "limit": 4812
}
```

Example result (excerpt):

```json
{
  "summary": "Dataset created from 4,812 traces.",
  "traces": 4812,
  "dataset": {
    "id": "c7d1e9a2-5b3f-4e8c-a1d4-6f2b9e0c3a57",
    "name": "Dispute Resolutions - Train",
    "state": "landing"
  },
  "job": {
    "kind": "dataset_run",
    "id": "c7d1e9a2-5b3f-4e8c-a1d4-6f2b9e0c3a57",
    "status": "landing"
  }
}
```

#### get_job

Source data ready

Input:

```json
{
  "kind": "dataset_run",
  "id": "c7d1e9a2-5b3f-4e8c-a1d4-6f2b9e0c3a57"
}
```

Example result (excerpt):

```json
{
  "kind": "dataset_run",
  "id": "c7d1e9a2-5b3f-4e8c-a1d4-6f2b9e0c3a57",
  "label": "Source data ready",
  "status": "completed"
}
```

**Your coding agent:** There are 4,812 source conversations. I’m preserving the reviewed decisions, grouping by case ID and excluding the Golden set before preparation.

#### message_dataset_agent

Prepared evidence → reviewed decision examples

Input:

```json
{
  "dataset": "c7d1e9a2-5b3f-4e8c-a1d4-6f2b9e0c3a57",
  "message": "Drop empty records and deduplicate by case ID. Normalize dates and amounts without dropping original values. Redact personal data. Keep only cases with reviewer-approved resolutions in captured metadata, using that existing resolution as the target. Preserve the original evidence and policy. Exclude IDs and content that overlap Dispute Resolutions - Golden. Do not invent, relabel or replace answers."
}
```

Example result (excerpt):

```json
{
  "summary": "Preparation started with captured reviewer resolutions; no semantic relabelling requested.",
  "job": {
    "kind": "dataset_run",
    "id": "c7d1e9a2-5b3f-4e8c-a1d4-6f2b9e0c3a57",
    "status": "running"
  }
}
```

#### get_job

4,400 examples prepared; Golden cases excluded

Input:

```json
{
  "kind": "dataset_run",
  "id": "c7d1e9a2-5b3f-4e8c-a1d4-6f2b9e0c3a57"
}
```

Example result (excerpt):

```json
{
  "kind": "dataset_run",
  "id": "c7d1e9a2-5b3f-4e8c-a1d4-6f2b9e0c3a57",
  "label": "4,400 examples prepared; Golden cases excluded",
  "status": "completed"
}
```

**Dispute Resolutions — Train:** 4,400 reviewed examples. 250 Golden cases held out; no overlap.

| Split | Cases | Purpose |
| --- | --- | --- |
| Training | 3,520 | Learn the reviewed decisions |
| Validation | 880 | Monitor loss during training |
| Golden | 250 | Compare models after training |

- **TRN-318:** Carrier scan at neighbour’s address; $76 order; no customer confirmation.
  Reviewed target: Request proof of delivery to the correct address before deciding. Source: reviewer-approved resolution, case TRN-318.

- **TRN-562:** $29 renewal on 9 July; cancellation confirmed 4 July; no prior credit.
  Reviewed target: Accept the dispute and recommend one $29 credit. Cancellation preceded the renewal. Source: reviewer-approved resolution, case TRN-562.

- **TRN-904:** Merchant and customer records disagree on the cancellation time; timezone absent.
  Reviewed target: Escalate for source verification. The available records cannot establish event order. Source: reviewer-approved resolution, case TRN-904.

### Train: Train a model

Fine-tune and monitor the run.

**You:** Can a smaller model learn this playbook? Check the data and estimate a training run first.

**Your coding agent:** I’ll check model compatibility and estimate a three-epoch LoRA run. Validation will monitor learning; the Golden evaluation will measure decision quality afterwards.

#### check_finetune_readiness

Training data compatible · zero Golden overlap

Input:

```json
{
  "dataset": "c7d1e9a2-5b3f-4e8c-a1d4-6f2b9e0c3a57",
  "capability": "9b1e4f7a-2c3d-4e5f-8a90-1b2c3d4e5f60"
}
```

Example result (excerpt):

```json
{
  "summary": "Ready for fine-tuning.",
  "ready": true,
  "missing": [],
  "warnings": [],
  "overlap_count": 0,
  "n_candidates": 8,
  "task_type": "structured_output",
  "has_tool_calling": false
}
```

#### estimate_finetune

Qwen3.5 9B · approximately 21 minutes

Input:

```json
{
  "dataset": "c7d1e9a2-5b3f-4e8c-a1d4-6f2b9e0c3a57",
  "base_model": "Qwen/Qwen3.5-9B",
  "n_epochs": 3,
  "use_lora": true
}
```

Example result (excerpt):

```json
{
  "summary": "Estimated training time: 21 minutes.",
  "time_estimate": {
    "seconds": 1260,
    "human": "21 minutes"
  },
  "trained_tokens": 1207560
}
```

**Training recommendation:** Qwen3.5 9B, LoRA, 3 epochs, estimated 21 minutes. Golden cases stay held out.

**You:** Go ahead with that run. Keep the evaluation data held out.

#### start_finetune

Started dispute-resolver-sft-1

Input:

```json
{
  "name": "dispute-resolver-sft-1",
  "dataset": "c7d1e9a2-5b3f-4e8c-a1d4-6f2b9e0c3a57",
  "capability": "9b1e4f7a-2c3d-4e5f-8a90-1b2c3d4e5f60",
  "base_model": "Qwen/Qwen3.5-9B",
  "baseline_model": "gpt-4.1",
  "eval_dataset": "a2f6c8e1-9d4b-4c7a-b3e5-8f1d2c6a9b40",
  "eval_set": "Dispute resolution quality",
  "validation_enabled": true,
  "validation_split_ratio": 0.2,
  "eval_incumbent_before": true,
  "eval_model_before": false,
  "eval_model_after": true,
  "hyperparameters": {
    "n_epochs": 3,
    "use_lora": true,
    "learning_rate": 0.00005,
    "context_length": 4096
  }
}
```

Example result (excerpt):

```json
{
  "summary": "Fine-tuning job started with incumbent-before and trained-after evaluations.",
  "job": {
    "kind": "finetune_job",
    "id": "8c21f7a0-3e5d-4b9f-a2c6-7d1e0f4b8a93",
    "name": "dispute-resolver-sft-1",
    "status": "pending"
  }
}
```

#### get_job

Following preparation and training progress

Input:

```json
{
  "kind": "finetune_job",
  "id": "8c21f7a0-3e5d-4b9f-a2c6-7d1e0f4b8a93"
}
```

Example result (excerpt):

```json
{
  "kind": "finetune_job",
  "id": "8c21f7a0-3e5d-4b9f-a2c6-7d1e0f4b8a93",
  "label": "Following preparation and training progress",
  "status": "running",
  "progress": {
    "phase": "preparing",
    "trained_steps": 0,
    "total_steps": 1200,
    "current_epoch": 0,
    "elapsed_seconds": 0
  }
}
```

**Your coding agent:** The job is queued. I’ll follow preparation, training metrics and validation checkpoints as Overmind reports progress.

**Training run:** dispute-resolver-sft-1. 1,200 optimizer steps, 600 logged metric points, 1,207,560 tokens, 20m 47s. Train loss 0.0031. Sample launch-video metrics; time condensed.

- Epoch 1, 400 steps: validation loss 0.081, token accuracy 93.1%.

- Epoch 2, 800 steps: validation loss 0.034, token accuracy 96.8%.

- Epoch 3, 1200 steps: validation loss 0.0198, token accuracy 97.9%.

#### resources/read

Read metrics history, validation checkpoints and run events

Input:

```json
{
  "uri": "overmind://finetunes/8c21f7a0-3e5d-4b9f-a2c6-7d1e0f4b8a93"
}
```

Example result (excerpt):

```json
{
  "kind": "finetune",
  "id": "8c21f7a0-3e5d-4b9f-a2c6-7d1e0f4b8a93",
  "status": "succeeded",
  "base_model": "Qwen/Qwen3.5-9B",
  "progress": {
    "phase": "completed",
    "trained_steps": 1200,
    "total_steps": 1200,
    "train_loss": 0.0031,
    "token_accuracy": 0.998,
    "eval_token_accuracy": 0.979,
    "tokens_processed": 1207560,
    "elapsed_seconds": 1247,
    "judge_evals": [
      {
        "kind": "baseline",
        "eval_run_id": "0c9b8a7d-6e5f-4a3b-8c2d-1e0f9a8b7c6d",
        "status": "succeeded",
        "sample_count": 250
      },
      {
        "kind": "final",
        "eval_run_id": "7a6b5c4d-3e2f-4a1b-9c8d-7e6f5a4b3c2d",
        "status": "running"
      }
    ],
    "metrics_history": [
      {
        "step": 2,
        "epoch": 0.01,
        "train_loss": 0.9204,
        "token_accuracy": 0.658,
        "lr": 0.00005,
        "grad_norm": 2.9829
      },
      {
        "step": 400,
        "epoch": 1,
        "train_loss": 0.1816,
        "token_accuracy": 0.9427,
        "lr": 0.0000375,
        "grad_norm": 1.6024
      },
      {
        "step": 800,
        "epoch": 2,
        "train_loss": 0.0377,
        "token_accuracy": 0.9913,
        "lr": 0.0000125,
        "grad_norm": 0.5936
      },
      {
        "step": 1200,
        "epoch": 3,
        "train_loss": 0.0031,
        "token_accuracy": 0.998,
        "lr": 0,
        "grad_norm": 0.3026
      }
    ],
    "eval_history": [
      {
        "step": 400,
        "epoch": 1,
        "eval_loss": 0.081
      },
      {
        "step": 800,
        "epoch": 2,
        "eval_loss": 0.034
      },
      {
        "step": 1200,
        "epoch": 3,
        "eval_loss": 0.0198
      }
    ]
  },
  "deployed_model": "d781f29c-3135-41a9-b2f7-76c1df55a004",
  "events": [
    {
      "type": "info",
      "message": "Training complete. Adapter saved and deployment ready."
    }
  ]
}
```

**Your coding agent:** Training completed in 20m 47s. Train loss is 0.0031; validation loss is 0.0198. That measures fit to the examples. The held-out evaluation will tell us whether dispute decisions are better.

### Evaluate: Run evaluations

Compare scores and inspect answers.

**You:** How did it compare with GPT-4.1 on the Golden set? Show me the score changes and any mistakes it still makes.

**Your coding agent:** The training plan includes evaluations before and after training. GPT-4.1 has finished; I’ll follow the trained-model evaluation, compare the same three rubrics, then inspect the decisions.

#### check_evaluation_readiness

Golden dataset and all three evaluators ready

Input:

```json
{
  "dataset": "a2f6c8e1-9d4b-4c7a-b3e5-8f1d2c6a9b40",
  "eval_set": "Dispute resolution quality",
  "mode": "generate"
}
```

Example result (excerpt):

```json
{
  "summary": "Evaluation dataset and evaluator bindings are ready.",
  "ready": true
}
```

#### get_job

Incumbent evaluation complete · 250 cases

Input:

```json
{
  "kind": "eval_run",
  "id": "0c9b8a7d-6e5f-4a3b-8c2d-1e0f9a8b7c6d"
}
```

Example result (excerpt):

```json
{
  "kind": "eval_run",
  "id": "0c9b8a7d-6e5f-4a3b-8c2d-1e0f9a8b7c6d",
  "label": "Incumbent evaluation complete · 250 cases",
  "status": "completed"
}
```

#### get_job

Following the trained-model evaluation · same 250 cases

Input:

```json
{
  "kind": "eval_run",
  "id": "7a6b5c4d-3e2f-4a1b-9c8d-7e6f5a4b3c2d"
}
```

Example result (excerpt):

```json
{
  "kind": "eval_run",
  "id": "7a6b5c4d-3e2f-4a1b-9c8d-7e6f5a4b3c2d",
  "label": "Following the trained-model evaluation · same 250 cases",
  "status": "running"
}
```

**Your coding agent:** The incumbent has finished. I’m waiting for the trained-model run to complete, then I’ll read the comparison and individual answers.

#### get_job

Trained-model evaluation completed

Input:

```json
{
  "kind": "eval_run",
  "id": "7a6b5c4d-3e2f-4a1b-9c8d-7e6f5a4b3c2d"
}
```

Example result (excerpt):

```json
{
  "kind": "eval_run",
  "id": "7a6b5c4d-3e2f-4a1b-9c8d-7e6f5a4b3c2d",
  "label": "Trained-model evaluation completed",
  "status": "completed"
}
```

#### compare_evaluations

Mean rubric score 72.0 → 86.2 · +14.2 points

Input:

```json
{
  "run": "7a6b5c4d-3e2f-4a1b-9c8d-7e6f5a4b3c2d",
  "baseline": "0c9b8a7d-6e5f-4a3b-8c2d-1e0f9a8b7c6d"
}
```

Example result (excerpt):

```json
{
  "summary": "Evaluation comparison ready.",
  "current_run_id": "7a6b5c4d-3e2f-4a1b-9c8d-7e6f5a4b3c2d",
  "baseline_run_id": "0c9b8a7d-6e5f-4a3b-8c2d-1e0f9a8b7c6d",
  "current_name": "Dispute Resolver — trained",
  "baseline_name": "Dispute Resolver — incumbent",
  "rows": [
    {
      "name": "Resolution correctness",
      "current": {
        "mean": 0.89,
        "pass_rate": null,
        "n": 250,
        "variant_count": 1,
        "primary": 0.89
      },
      "baseline": {
        "mean": 0.74,
        "pass_rate": null,
        "n": 250,
        "variant_count": 1,
        "primary": 0.74
      },
      "delta": 0.15,
      "status": "improved"
    },
    {
      "name": "Policy adherence",
      "current": {
        "mean": 0.85,
        "pass_rate": null,
        "n": 250,
        "variant_count": 1,
        "primary": 0.85
      },
      "baseline": {
        "mean": 0.69,
        "pass_rate": null,
        "n": 250,
        "variant_count": 1,
        "primary": 0.69
      },
      "delta": 0.16,
      "status": "improved"
    },
    {
      "name": "Tone",
      "current": {
        "mean": 0.846,
        "pass_rate": null,
        "n": 250,
        "variant_count": 1,
        "primary": 0.846
      },
      "baseline": {
        "mean": 0.73,
        "pass_rate": null,
        "n": 250,
        "variant_count": 1,
        "primary": 0.73
      },
      "delta": 0.116,
      "status": "improved"
    }
  ],
  "overall": {
    "current": {
      "mean": 0.862,
      "pass_rate": null,
      "n": 750,
      "variant_count": 3,
      "primary": 0.862
    },
    "baseline": {
      "mean": 0.7200000000000001,
      "pass_rate": null,
      "n": 750,
      "variant_count": 3,
      "primary": 0.7200000000000001
    },
    "delta": 0.142,
    "status": "improved"
  },
  "trust": {
    "current": {
      "trusted": true,
      "degraded": 0,
      "evaluator_errors": 0,
      "errored": 0,
      "not_applicable": 0
    },
    "baseline": {
      "trusted": true,
      "degraded": 0,
      "evaluator_errors": 0,
      "errored": 0,
      "not_applicable": 0
    }
  },
  "not_applicable_by_evaluator": {}
}
```

#### resources/read

Read incumbent outputs and judge reasoning

Input:

```json
{
  "uri": "overmind://eval-runs/0c9b8a7d-6e5f-4a3b-8c2d-1e0f9a8b7c6d"
}
```

Example result (excerpt):

```json
{
  "kind": "eval_run",
  "id": "0c9b8a7d-6e5f-4a3b-8c2d-1e0f9a8b7c6d",
  "status": "completed",
  "sample_count": 250,
  "samples_truncated": true,
  "samples": [
    {
      "row_index": 0,
      "variant": "GPT-4.1",
      "io": {
        "input": "A $184 order is marked delivered. The carrier scan is at a different postcode, and there is no signature. The customer denies receiving it.",
        "output": "The carrier confirms delivery. Submit the tracking record as evidence."
      },
      "scores": [
        {
          "name": "Resolution correctness",
          "value": 0.25,
          "passed": false,
          "reasoning": "Delivery evidence must match the order address. A delivery status alone is insufficient to contest."
        }
      ]
    },
    {
      "row_index": 1,
      "variant": "GPT-4.1",
      "io": {
        "input": "A $49 renewal was charged on 18 August. Support confirmed cancellation on 12 August. The merchant has not issued a credit.",
        "output": "The customer accepted recurring billing when they subscribed. Contest with the subscription agreement."
      },
      "scores": [
        {
          "name": "Resolution correctness",
          "value": 0.2,
          "passed": false,
          "reasoning": "Accept liability for a renewal after a confirmed cancellation; check for prior credits before recommending a refund."
        }
      ]
    },
    {
      "row_index": 2,
      "variant": "GPT-4.1",
      "io": {
        "input": "A $92 order was delivered to the order address. A signed receipt and the customer’s message confirming delivery are attached.",
        "output": "Contest with the signed delivery receipt and the customer’s confirmation."
      },
      "scores": [
        {
          "name": "Resolution correctness",
          "value": 0.95,
          "passed": true,
          "reasoning": "Contest a non-receipt claim when matching delivery evidence and customer confirmation are available."
        }
      ]
    },
    {
      "row_index": 3,
      "variant": "GPT-4.1",
      "io": {
        "input": "The cancellation email and billing event have the same timestamp but different timezones. Their ordering cannot be established from the captured records.",
        "output": "The charge appears to predate cancellation. Contest it."
      },
      "scores": [
        {
          "name": "Resolution correctness",
          "value": 0.2,
          "passed": false,
          "reasoning": "Escalate unresolved conflicts in source evidence. Do not infer the event ordering."
        }
      ]
    }
  ]
}
```

#### resources/read

Read improved cases and the remaining evidence-ordering failure

Input:

```json
{
  "uri": "overmind://eval-runs/7a6b5c4d-3e2f-4a1b-9c8d-7e6f5a4b3c2d"
}
```

Example result (excerpt):

```json
{
  "kind": "eval_run",
  "id": "7a6b5c4d-3e2f-4a1b-9c8d-7e6f5a4b3c2d",
  "status": "completed",
  "sample_count": 250,
  "samples_truncated": true,
  "samples": [
    {
      "row_index": 0,
      "variant": "Dispute Resolver SFT",
      "io": {
        "input": "A $184 order is marked delivered. The carrier scan is at a different postcode, and there is no signature. The customer denies receiving it.",
        "output": "The scan does not match the order address. Request proof of delivery before deciding whether to contest."
      },
      "scores": [
        {
          "name": "Resolution correctness",
          "value": 1,
          "passed": true,
          "reasoning": "The trained response uses the postcode mismatch and missing signature. The original response treats a status label as sufficient evidence."
        }
      ]
    },
    {
      "row_index": 1,
      "variant": "Dispute Resolver SFT",
      "io": {
        "input": "A $49 renewal was charged on 18 August. Support confirmed cancellation on 12 August. The merchant has not issued a credit.",
        "output": "Cancellation was confirmed six days before renewal. Accept liability and recommend a $49 refund; no earlier credit exists."
      },
      "scores": [
        {
          "name": "Resolution correctness",
          "value": 0.95,
          "passed": true,
          "reasoning": "The cancellation timestamp takes precedence over the original billing agreement. The trained response also checks for a duplicate credit."
        }
      ]
    },
    {
      "row_index": 2,
      "variant": "Dispute Resolver SFT",
      "io": {
        "input": "A $92 order was delivered to the order address. A signed receipt and the customer’s message confirming delivery are attached.",
        "output": "The evidence is complete and consistent. Prepare a response citing the signed receipt and the delivery confirmation."
      },
      "scores": [
        {
          "name": "Resolution correctness",
          "value": 0.95,
          "passed": true,
          "reasoning": "Both responses select the correct resolution and cite the available evidence. This is a control case, not a new improvement."
        }
      ]
    },
    {
      "row_index": 3,
      "variant": "Dispute Resolver SFT",
      "io": {
        "input": "The cancellation email and billing event have the same timestamp but different timezones. Their ordering cannot be established from the captured records.",
        "output": "The cancellation appears to have arrived first. Accept the dispute."
      },
      "scores": [
        {
          "name": "Resolution correctness",
          "value": 0.35,
          "passed": false,
          "reasoning": "Both models infer an order the evidence does not establish. Keep this case in the next review batch; the trained model is not correct on every case."
        }
      ]
    }
  ]
}
```

**Paired evaluation outputs:** same held-out cases and policy checks.

Mean rubric score: 72.0 → 86.2 / 100 (+14.2 points). These are illustrative scores, not pass rates.

| Metric | GPT-4.1 | Trained | Change |
| --- | --- | --- | --- |
| Resolution correctness | 74.0 | 89.0 | +15.0 |
| Policy adherence | 69.0 | 85.0 | +16.0 |
| Tone | 73.0 | 84.6 | +11.6 |
| Mean score | 72.0 | 86.2 | +14.2 |

- Delivery exception: current model — Contest; trained — Request evidence; reference — Request matching proof of delivery; do not contest yet.
  Judge reasoning: The trained response uses the postcode mismatch and missing signature. The original response treats a status label as sufficient evidence.

- Cancelled subscription: current model — Contest; trained — Accept liability; reference — Accept liability and recommend a single $49 refund.
  Judge reasoning: The cancellation timestamp takes precedence over the original billing agreement. The trained response also checks for a duplicate credit.

- Evidence complete: current model — Contest; trained — Contest; reference — Contest with both supporting documents.
  Judge reasoning: Both responses select the correct resolution and cite the available evidence. This is a control case, not a new improvement.

- Still needs review: current model — Contest; trained — Accept liability; reference — Escalate for review of the original timestamps.
  Judge reasoning: Both models infer an order the evidence does not establish. Keep this case in the next review batch; the trained model is not correct on every case.

### Run inference: Run inference

Test the model and update the code.

**You:** Try a new delivery dispute. If it follows the playbook, activate it for Dispute Resolver and show me the code change. Keep human approval for financial actions.

**Your coding agent:** I’ll test a new case against the ready deployment first. The model will return a recommendation; the existing approval step remains in your agent.

#### get_job

Deployment ready for inference

Input:

```json
{
  "kind": "deployment",
  "id": "d781f29c-3135-41a9-b2f7-76c1df55a004"
}
```

Example result (excerpt):

```json
{
  "kind": "deployment",
  "id": "d781f29c-3135-41a9-b2f7-76c1df55a004",
  "label": "Deployment ready for inference",
  "status": "ready"
}
```

#### run_inference

New case → request matching delivery evidence

Input:

```json
{
  "deployment": "d781f29c-3135-41a9-b2f7-76c1df55a004",
  "messages": [
    {
      "role": "user",
      "content": "Triage case DSP-1284 using the merchant playbook. A $136 order is marked delivered, but the delivery photo shows a different door number. No signature is available. The customer denies receipt. Return a JSON recommendation; do not execute a financial action."
    }
  ],
  "temperature": 0,
  "max_tokens": 512
}
```

Example result (excerpt):

```json
{
  "summary": "Inference completed.",
  "model_id": "overmind/8c21f7a0-3e5d-4b9f-a2c6-7d1e0f4b8a93",
  "content": "{\"case_id\":\"DSP-1284\",\"decision\":\"request_evidence\",\"reason\":\"The delivery photo does not match the order address, so delivery is not established.\",\"evidence_used\":[\"delivery_photo_address_mismatch\",\"signature_missing\"],\"next_action\":\"Request proof of delivery to the order address before deciding whether to contest.\",\"requires_human_approval\":true}",
  "usage": {
    "prompt_tokens": 218,
    "completion_tokens": 96,
    "total_tokens": 314
  },
  "latency_ms": 1840,
  "is_cold": false,
  "finish_reason": "stop",
  "truncated": false
}
```

**New inference request:** Triage case DSP-1284 using the merchant playbook. A $136 order is marked delivered, but the delivery photo shows a different door number. No signature is available. The customer denies receipt. Return a JSON recommendation; do not execute a financial action.

**Trained model response:**

```json
{
  "case_id": "DSP-1284",
  "decision": "request_evidence",
  "reason": "The delivery photo does not match the order address, so delivery is not established.",
  "evidence_used": [
    "delivery_photo_address_mismatch",
    "signature_missing"
  ],
  "next_action": "Request proof of delivery to the order address before deciding whether to contest.",
  "requires_human_approval": true
}
```

**Your coding agent:** It requests the missing proof of delivery instead of contesting automatically. I’ll activate this model and prepare the alias change.

#### set_active_model

Activated the trained Dispute Resolver

Input:

```json
{
  "capability": "9b1e4f7a-2c3d-4e5f-8a90-1b2c3d4e5f60",
  "deployment": "d781f29c-3135-41a9-b2f7-76c1df55a004"
}
```

Example result (excerpt):

```json
{
  "summary": "Active model updated for Dispute Resolver.",
  "cleared": false,
  "active_model": {
    "id": "d781f29c-3135-41a9-b2f7-76c1df55a004",
    "model_id": "overmind/8c21f7a0-3e5d-4b9f-a2c6-7d1e0f4b8a93",
    "status": "ready"
  }
}
```

#### get_model_swap_prompt

Prepared the permanent capability alias

Input:

```json
{
  "finetune": "8c21f7a0-3e5d-4b9f-a2c6-7d1e0f4b8a93",
  "pin": false
}
```

Example result (excerpt):

```json
{
  "summary": "Model-swap instructions prepared.",
  "capability_id": "9b1e4f7a-2c3d-4e5f-8a90-1b2c3d4e5f60",
  "old_model": "gpt-4.1",
  "new_model": "overmind/9b1e4f7a-2c3d-4e5f-8a90-1b2c3d4e5f60",
  "pin": false
}
```

**Proposed integration:** agents/dispute_resolver.py

```python
from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.overmindlab.ai/api/v1",
    api_key=os.environ["OVERMIND_API_KEY"],
)

response = client.chat.completions.create(
    model="overmind/9b1e4f7a-2c3d-4e5f-8a90-1b2c3d4e5f60",
    messages=messages,
)
```

**Your coding agent:** The new case follows the playbook. The trained model is active, and the code change is ready for review. Future model swaps can use this same capability alias.

## Connect your coding agent.

Choose your coding agent and copy the prompt to connect your project to Overmind Cloud.

[Open Console](https://console.overmindlab.ai) · [Read the quickstart](https://docs.overmindlab.ai/quickstart)

### Onboarding prompt for Cursor

```text
Connect Cursor to Overmind Cloud using the hosted MCP server at https://api.overmindlab.ai/api/mcp/.

Use the project already open in this workspace. If no project is open, ask me to open the project I want to connect.

Read https://docs.overmindlab.ai/quickstart. Guide me through signing in at https://console.overmindlab.ai and obtaining the cloud onboarding credentials. Keep the temporary bootstrap key in the current shell session only, out of tracked files and chat output.

Install the published overmind package from PyPI using this project's package manager and dependency manifest. For a non-Python project, use an isolated CLI environment. Set OVERMIND_API_URL=https://api.overmindlab.ai and run overmind init --ide cursor --env production through that environment.

Read references/onboard.md and references/onboarding-progress.md in the installed Overmind skill. Show the onboarding roadmap and what will be synced, then run overmind sync to connect the Console project and configure MCP with its project-scoped credential.

Ask me to reload Cursor once. After reload, read overmind://project/current to verify the connection and continue the installed onboarding skill without repeating bootstrap setup. Summarize the setup and recommend the next step.
```

MCP configuration: `.cursor/mcp.json`

### Onboarding prompt for Claude Code

```text
Connect Claude Code to Overmind Cloud using the hosted MCP server at https://api.overmindlab.ai/api/mcp/.

Use the project already open in this workspace. If no project is open, ask me to open the project I want to connect.

Read https://docs.overmindlab.ai/quickstart. Guide me through signing in at https://console.overmindlab.ai and obtaining the cloud onboarding credentials. Keep the temporary bootstrap key in the current shell session only, out of tracked files and chat output.

Install the published overmind package from PyPI using this project's package manager and dependency manifest. For a non-Python project, use an isolated CLI environment. Set OVERMIND_API_URL=https://api.overmindlab.ai and run overmind init --ide claude --env production through that environment.

Read references/onboard.md and references/onboarding-progress.md in the installed Overmind skill. Show the onboarding roadmap and what will be synced, then run overmind sync to connect the Console project and configure MCP with its project-scoped credential.

Ask me to reload Claude Code once. After reload, read overmind://project/current to verify the connection and continue the installed onboarding skill without repeating bootstrap setup. Summarize the setup and recommend the next step.
```

MCP configuration: `.mcp.json`

### Onboarding prompt for Codex

```text
Connect Codex to Overmind Cloud using the hosted MCP server at https://api.overmindlab.ai/api/mcp/.

Use the project already open in this workspace. If no project is open, ask me to open the project I want to connect.

Read https://docs.overmindlab.ai/quickstart. Guide me through signing in at https://console.overmindlab.ai and obtaining the cloud onboarding credentials. Keep the temporary bootstrap key in the current shell session only, out of tracked files and chat output.

Install the published overmind package from PyPI using this project's package manager and dependency manifest. For a non-Python project, use an isolated CLI environment. Set OVERMIND_API_URL=https://api.overmindlab.ai and run overmind init --ide codex --env production through that environment.

Read references/onboard.md and references/onboarding-progress.md in the installed Overmind skill. Show the onboarding roadmap and what will be synced, then run overmind sync to connect the Console project and configure MCP with its project-scoped credential.

Ask me to reload Codex once. After reload, read overmind://project/current to verify the connection and continue the installed onboarding skill without repeating bootstrap setup. Summarize the setup and recommend the next step.
```

MCP configuration: `.codex/config.toml`

### Onboarding prompt for OpenCode

```text
Connect OpenCode to Overmind Cloud using the hosted MCP server at https://api.overmindlab.ai/api/mcp/.

Use the project already open in this workspace. If no project is open, ask me to open the project I want to connect.

Read https://docs.overmindlab.ai/quickstart. Guide me through signing in at https://console.overmindlab.ai and obtaining the cloud onboarding credentials. Keep the temporary bootstrap key in the current shell session only, out of tracked files and chat output.

Install the published overmind package from PyPI using this project's package manager and dependency manifest. For a non-Python project, use an isolated CLI environment. Set OVERMIND_API_URL=https://api.overmindlab.ai and run overmind init --ide opencode --env production through that environment.

Read references/onboard.md and references/onboarding-progress.md in the installed Overmind skill. Show the onboarding roadmap and what will be synced, then run overmind sync to connect the Console project and configure MCP with its project-scoped credential.

Ask me to reload OpenCode once. After reload, read overmind://project/current to verify the connection and continue the installed onboarding skill without repeating bootstrap setup. Summarize the setup and recommend the next step.
```

MCP configuration: `opencode.json`

---

# https://www.overmindlab.ai/models

# Model Library

> Canonical: https://www.overmindlab.ai/models

Every model here trains on your agent's own data, can be benchmarked against your production model, and serves through one OpenAI-compatible API. The weights are yours.

## Qwen 3

- [Qwen 3 4B](https://www.overmindlab.ai/models/qwen-3-4b): 4B, 128K context, —; train from $0.70, serve from $4.00 per 1M output tokens
- [Qwen 3 1.7B](https://www.overmindlab.ai/models/qwen-3-1-7b): 1.7B, 128K context, —; train from $0.55, serve from $4.00 per 1M output tokens
- [Qwen 3 0.6B](https://www.overmindlab.ai/models/qwen-3-0-6b): 0.6B, 128K context, —; train from $0.50, serve from $2.50 per 1M output tokens
- [Qwen 3 8B](https://www.overmindlab.ai/models/qwen-3-8b): 8B, 128K context, —; train from $1.00, serve from $6.00 per 1M output tokens
- [Qwen 3 Coder 30B MoE](https://www.overmindlab.ai/models/qwen-3-coder-30b-a3b-instruct): 30B MoE (3B active), 256K context, —; train from $2.00, serve from $8.00 per 1M output tokens
- [Qwen 3 14B](https://www.overmindlab.ai/models/qwen-3-14b): 14B, 128K context, —; train from $1.25, serve from $10.00 per 1M output tokens
- [Qwen 3 32B](https://www.overmindlab.ai/models/qwen-3-32b): 32B, 128K context, —; train from $2.25, serve from $20.00 per 1M output tokens

## Qwen 3.5

- [Qwen 3.5 4B](https://www.overmindlab.ai/models/qwen-3-5-4b): 4B, 256K context, —; train from $0.70, serve from $4.00 per 1M output tokens
- [Qwen 3.5 2B](https://www.overmindlab.ai/models/qwen-3-5-2b): 2B, 256K context, —; train from $0.55, serve from $4.00 per 1M output tokens
- [Qwen 3.5 0.8B](https://www.overmindlab.ai/models/qwen-3-5-0-8b): 0.8B, 256K context, —; train from $0.50, serve from $2.50 per 1M output tokens
- [Qwen 3.5 9B](https://www.overmindlab.ai/models/qwen-3-5-9b): 9B, 256K context, —; train from $1.00, serve from $6.00 per 1M output tokens
- [Qwen 3.5 27B](https://www.overmindlab.ai/models/qwen-3-5-27b): 27B, 256K context, —; train from $2.00, serve from $15.00 per 1M output tokens
- [Qwen 3.5 35B MoE](https://www.overmindlab.ai/models/qwen-3-5-35b-a3b): 35B MoE (3B active), 256K context, —; train from $2.50, serve from $8.00 per 1M output tokens

## Llama 3

- [Llama 3.2 3B Instruct](https://www.overmindlab.ai/models/llama-3-2-3b-instruct): 3B, 128K context, —; train from $0.55, serve from $4.00 per 1M output tokens
- [Llama 3.2 1B Instruct](https://www.overmindlab.ai/models/llama-3-2-1b-instruct): 1B, 128K context, —; train from $0.50, serve from $2.50 per 1M output tokens
- [Llama 3.1 8B Instruct](https://www.overmindlab.ai/models/meta-llama-3-1-8b-instruct-reference): 8B, 128K context, —; train from $1.00, serve from $6.00 per 1M output tokens
- [Llama 3.1 8B Instruct](https://www.overmindlab.ai/models/llama-3-1-8b-instruct): 8B, 128K context, —; train from $1.00, serve from $6.00 per 1M output tokens
- [Llama 3.3 70B Instruct](https://www.overmindlab.ai/models/llama-3-3-70b-instruct): 70B, 128K context, —; train from $4.50, serve from $35.00 per 1M output tokens

## Antares

- [Antares 1B (Security)](https://www.overmindlab.ai/models/antares-1b): 1.8B, 128K context, —; train from $0.55, serve from $4.00 per 1M output tokens
- [Antares 350M (Security)](https://www.overmindlab.ai/models/antares-350m): 350M, 32K context, —; train from $0.50, serve from $2.50 per 1M output tokens

## LFM2.5

- [LFM2.5 1.2B Instruct](https://www.overmindlab.ai/models/lfm-2-5-1-2b-instruct): 1.2B, 32K context, —; train from $0.50, serve from $2.50 per 1M output tokens
- [LFM2.5 350M](https://www.overmindlab.ai/models/lfm-2-5-350m): 350M, 32K context, —; train from $0.50, serve from $2.50 per 1M output tokens
- [LFM2.5 230M](https://www.overmindlab.ai/models/lfm-2-5-230m): 230M, 32K context, —; train from $0.50, serve from $2.50 per 1M output tokens

## Qwen 2.5

- [Qwen 2.5 0.5B Instruct](https://www.overmindlab.ai/models/qwen-2-5-0-5b-instruct): 0.5B, 32K context, —; train from $0.50, serve from $2.50 per 1M output tokens
- [Qwen 2.5 7B Instruct](https://www.overmindlab.ai/models/qwen-2-5-7b-instruct): 7B, 128K context, —; train from $1.00, serve from $6.00 per 1M output tokens
- [Qwen 2.5 14B Instruct](https://www.overmindlab.ai/models/qwen-2-5-14b-instruct): 14B, 128K context, —; train from $1.25, serve from $10.00 per 1M output tokens
- [Qwen 2.5 72B Instruct](https://www.overmindlab.ai/models/qwen-2-5-72b-instruct): 72B, 128K context, —; train from $5.00, serve from $40.00 per 1M output tokens
- [Qwen 2.5 32B Instruct](https://www.overmindlab.ai/models/qwen-2-5-32b-instruct): 32B, 128K context, —; train from $2.25, serve from $20.00 per 1M output tokens
- [Qwen 2.5 Coder 32B Instruct](https://www.overmindlab.ai/models/qwen-2-5-coder-32b-instruct): 32B, 128K context, —; train from $2.25, serve from $20.00 per 1M output tokens

## Gemma 4

- [Gemma 4 E4B](https://www.overmindlab.ai/models/gemma-4-e4b-it): E4B (~8B), 128K context, —; train from $0.75, serve from $4.00 per 1M output tokens
- [Gemma 4 E2B](https://www.overmindlab.ai/models/gemma-4-e2b-it): E2B (~5.1B), 128K context, —; train from $0.55, serve from $4.00 per 1M output tokens
- [Gemma 4 12B](https://www.overmindlab.ai/models/gemma-4-12b-it): 12B, 256K context, —; train from $1.00, serve from $6.00 per 1M output tokens
- [Gemma 4 31B](https://www.overmindlab.ai/models/gemma-4-31b-it): 31B, 256K context, —; train from $2.50, serve from $10.00 per 1M output tokens
- [Gemma 4 26B-A4B](https://www.overmindlab.ai/models/gemma-4-26b-a4b-it): 26B (4B active), 256K context, —; train from $2.00, serve from $8.00 per 1M output tokens

## Qwen 3.8

- [Qwen 3.8 27B](https://www.overmindlab.ai/models/qwen-3-8-27b): 27B, 256K context, —; train from $2.00, serve from $15.00 per 1M output tokens

## Qwen 3.6

- [Qwen 3.6 27B](https://www.overmindlab.ai/models/qwen-3-6-27b): 27B, 256K context, —; train from $2.00, serve from $15.00 per 1M output tokens

## GPT-OSS

- [GPT-OSS 20B](https://www.overmindlab.ai/models/gpt-oss-20b): 20B MoE, 128K context, —; train from $1.75, serve from $12.00 per 1M output tokens

## Nemotron 3.5

- [Nemotron 3.5 Lightning 30B-A3B](https://www.overmindlab.ai/models/nvidia-nemotron-3-5-lightning-30b-a3b): 30B-A3B, 256K context, —; train from $2.00, serve from $6.00 per 1M output tokens

## Muse Glimmer

- [Muse Glimmer 30B](https://www.overmindlab.ai/models/muse-glimmer-30b): 30B, 128K context, —; train from $2.50, serve from $10.00 per 1M output tokens

Inference API: https://docs.overmindlab.ai/models/inference.md

---

# https://www.overmindlab.ai/pricing

# Pricing

> Canonical: https://www.overmindlab.ai/pricing

Flexibility to meet your needs for tracing, testing, training, and deployment at any scale.

## Free — $0 / month

For prototypes, demos, and getting agents instrumented.

Includes:

- Tracing: Fair-use OTLP ingest · browse spans & sessions
- Agent discovery: Map tools, prompts, and control flow
- Agent Optimizer: Up to 10 optimize runs / month
- Data Workshop: Upload, quality checks, Workshop chat
- Evaluations: Unlimited eval runs
- Training: Up to 2 training jobs / month
- Inference: Up to 2 deploy jobs / month · OpenAI-compatible API
- Console, CLI, and MCP server
- 5 projects
- Community Discord support

*$50 one-time credit to try the platform.*

[Start for free](https://console.overmindlab.ai)

## Pro — $120 / month (most popular)

For teams shipping agents that need to improve in production.

Same features, higher limits:

- Tracing: Fair-use OTLP ingest · browse spans & sessions
- Agent discovery: Map tools, prompts, and control flow
- Agent Optimizer: Unlimited optimize runs
- Data Workshop: Upload, quality checks, Workshop chat
- Evaluations: Unlimited eval runs
- Training: Unlimited training jobs
- Inference: Unlimited deploy jobs · OpenAI-compatible API
- Console, CLI, and MCP server
- Unlimited projects
- Priority support

[Get Pro](https://console.overmindlab.ai/login?next=%2Fsettings%3Fbilling%3Dcheckout)

## Enterprise — Custom

For organizations with security, scale, and compliance needs.

Everything in Pro, plus:

- Dedicated technical support
- SSO & custom auth
- Security & compliance: Credentials encrypted at rest. Compliance scope agreed per contract
- Custom onboarding & playbooks
- Unlimited seats and usage contracts

Speak to us: https://www.overmindlab.ai/contact

## Common questions

### Can I train and deploy a model on Free?

Yes. Free includes up to 2 training jobs and 2 deployments a month. You can prepare your dataset in the Data Workshop and serve your trained model through an OpenAI-compatible API.

### How does the $50 starter credit work?

New accounts receive a $50 one-time credit. Training, inference, Agent Optimizer runs, and the Data Workshop chat agent use credits.

When your balance runs out, add credits in Settings to start new training, eval, deployment, or optimization runs. You can still send traces and view existing results. Your plan's monthly limits still apply.

### Can I take my trained model elsewhere?

Yes. You can download the trained weights and checkpoints from the Console or with the Overmind CLI, then serve them on your own infrastructure.

## What each plan lets you do, in the documentation

- [How do I send my first trace to Overmind?](https://docs.overmindlab.ai/quickstart)
- [How do I fine-tune a model on my own traces?](https://docs.overmindlab.ai/models/training)
- [How do I serve a trained model on an OpenAI-compatible endpoint?](https://docs.overmindlab.ai/models/inference)
- [How do I run an evaluation against a baseline model?](https://docs.overmindlab.ai/agent-testing/eval)

---

# https://www.overmindlab.ai/compare

# Compare Overmind

> Canonical: https://www.overmindlab.ai/compare

Overmind is a harness for agent optimization and model training, automating the loop from production traces to specialized models. The tools outlined below cover part of the loop. These comparison pages set out what each tool does, where it could be the better choice, and where Overmind fits in.

## Which tools does Overmind compare with?

- [Overmind vs LangSmith](https://www.overmindlab.ai/compare/overmind-vs-langsmith): The most adopted agent observability platform, now with LoRA fine-tuning via smithtune on Fireworks or Baseten. Overmind adds trace import from any source, audited datasets, a benchmark against production, hosted serving and downloadable weights.
- [Overmind vs Langfuse](https://www.overmindlab.ai/compare/overmind-vs-langfuse): The open-source observability standard. Overmind imports its traces today and runs the training loop it deliberately leaves out.
- [Overmind vs OpenPipe](https://www.overmindlab.ai/compare/overmind-vs-openpipe): Proved that production requests should train cheaper models you own. Sunset July 2026; Overmind runs that idea as a full loop.
- [Overmind vs OpenAI fine-tuning](https://www.overmindlab.ai/compare/openai-fine-tuning-alternatives): OpenAI closes self-serve fine-tuning to new jobs on 6 January 2027. The dates, four alternatives, and the migration steps.

---

# https://www.overmindlab.ai/compare/overmind-vs-langsmith

# Overmind vs LangSmith

> Canonical: https://www.overmindlab.ai/compare/overmind-vs-langsmith

By Tyler Edwards, CEO, Overmind

*Last reviewed: 2026-10-13*

## In short

LangSmith is the most widely adopted agent engineering platform: tracing, evals, a prompt playground, annotation queues and agent deployment. Overmind takes the same traces, or traces from other tools, through audited datasets, a fine-tuned model, an optional benchmark against your production model, and hosted serving.

LangSmith is the most widely adopted agent engineering platform, with tracing, evals, a prompt playground, annotation queues, and agent deployment.

**Bottom line:** LangSmith now fine-tunes LangSmith trajectories on Fireworks or Baseten. Overmind does the whole loop in one place: traces from any source, audited datasets, LoRA or full fine-tuning, a benchmark against your production model, hosted serving, and weights you own.

[Start for free](https://console.overmindlab.ai)

## Start with the traces already in LangSmith

You can assess the dataset and evaluation workflow while continuing to use LangSmith. Begin with a representative slice of one production task. Model training is optional.

1. **Connect LangSmith.** Open Observability, then Integrations, choose LangSmith, and add your API key and the API URL for your region. The connector backfills retained history, then syncs incrementally. Sync now requests a manual pull. [Connector setup](https://docs.overmindlab.ai/core/observability#getting-traces-in)
2. **Build an evaluation dataset.** Review the imported traces, select examples for one task, and choose Add to dataset. Pick Evaluation, review the field mapping and source provenance, then inspect quality issues in Data Workshop. [Trace-to-dataset guide](https://docs.overmindlab.ai/core/datasets)
3. **Compare against a baseline.** Define an eval set for the task and create an evaluation run from your dataset or captured traces. Pin a baseline and the quality criteria. Read individual failures as well as the aggregate score before choosing what to change next. [Evaluation setup](https://docs.overmindlab.ai/agent-testing/eval#eval-runs)

The connector reads from LangSmith, so available history depends on what LangSmith retains and what your API key can see. Imported traces become datasets, but scoring on arrival only applies to spans sent to Overmind's own OTLP endpoint. Evaluating outputs does not replay your production tools, and the optimizer runs from the CLI in your own repository and needs an executable agent.

## Is Overmind a LangSmith alternative?

Since September 2026 LangSmith can also fine-tune, through smithtune on Fireworks or Baseten. Overmind runs the loop on traces from any source and ends at a model it hosts, benchmarked against production, with weights you download.

- Traces: Overmind covered (OTLP + LangSmith import); LangSmith covered (OTel-native)
- Datasets: Overmind covered (full); LangSmith covered (annotation queues)
- Evals: Overmind covered (full); LangSmith covered (on + offline)
- Optimize: Overmind covered (full); LangSmith partial (manual playground)
- Train: Overmind covered (full); LangSmith partial (LoRA SFT via Fireworks or Baseten (beta))
- Serve: Overmind covered (full); LangSmith partial (proxy only)

## Which one fits your team?

Two of the six verdicts here send you to LangSmith, and one of those is a closed-network requirement Overmind cannot meet: its self-hosted platform still calls OpenRouter, Modal, AWS S3 and Hugging Face.

### Instrument now. Decide later.

Without traffic there is nothing to curate, train, or benchmark, in either product. Wire up whichever tracer your team will actually ship this week and let it fill. Overmind takes standard OTLP and never charges for ingest, so a free project can collect in the background while you make your mind up.

*Training wants a few hundred clean, trace-sourced rows. Volume is the gate here, not the vendor.*

### LangSmith is the fit.

If the job is seeing and scoring what the agent does, LangSmith does it with more depth than Overmind. Annotation queues, pairwise review, end-to-end OpenTelemetry, and the largest ecosystem in the category. Overmind only becomes relevant the day those traces should train something.

*7,000+ customers, five of the Fortune 10. Overmind does not try to out-observe it.*

### Keep the frontier model. Take the loop.

A small specialist wins on a narrow job, not on open-ended reasoning across domains, so do not plan on swapping the model. Take the rest: evals generated from your agent's own design, and an optimizer that rewrites prompts, tool definitions, and logic as scored git diffs. Training is there for the day a narrow sub-task splits out.

*The optimizer scores every candidate diff against your eval set and hands back the winner as a git diff you apply. No training involved.*

### LangSmith Enterprise, for a closed network.

A control plane with no outside dependencies settles it. LangSmith self-hosts on its Enterprise plan. Overmind's platform is open source under AGPL-3.0 and self-hosts with Docker Compose, but it will not start without OpenRouter, Modal, AWS S3 and Hugging Face credentials, and training runs on Modal. If those services are acceptable, self-host Overmind. If what actually has to stay inside is the model rather than the console, start over and pick the middle answer.

*A self-hosted Overmind still trains on Modal. Downloaded weights run anywhere.*

### Overmind, with the weights downloaded.

The console can stay hosted; the model does not have to be. Every trained model ships downloadable weights, adapter and config and tokenizer, to run wherever you like. Train and benchmark on Overmind, serve inside your own network, and keep sending OTLP spans back from there.

*Open-weight Qwen, Llama, and Antares bases, LoRA or full fine-tune. Nothing locks the artefact in.*

### Overmind is the fit.

A repeatable job with real traffic behind it is the case that trains well. Traces become an audited dataset, the dataset trains a model, and the model can be benchmarked against the one your agent runs today on held-out data, with any train and eval overlap reported. Going live is one Make live click that points your capability's model alias at the new model. Nothing here says drop LangSmith: both products read standard OTLP, so dual-sending spans keeps your review workflows where they are.

*A 0.5B specialist beat GPT-4o mini 50% to 29% on the same product eval, and the weights are yours to download.*

## What are the key differences between Overmind and LangSmith?

| Dimension | Overmind | LangSmith |
| --- | --- | --- |
| Fine-tuning | LoRA or full, on Qwen, Llama, and Antares | LoRA SFT via smithtune (beta) on Fireworks or Baseten, with your provider key |
| Model serving | Hosts your trained models, and proxies frontier ones | Proxies only. Gateway (beta) hosts nothing |
| Prompt improvement | Optimizer returns scored diffs you apply | Playground, versioned prompts, manual iteration |
| Proof before switching | Base vs trained on held-out data. Production model opt-in | Experiments compare app versions on datasets |
| Tracing | OTLP ingest, scored on arrival. Read-only LangSmith import | Py/TS/Go/Java SDKs, end-to-end OTel |
| Human review | Annotations that calibrate LLM judges | Annotation queues, pairwise A/B, bulk actions |
| License and self-hosting | AGPL-3.0 platform, MIT SDK. Docker Compose or console.overmindlab.ai | Commercial. Whole platform self-hosts on the Enterprise plan |
| Pricing | Free / Pro + credits. Ingest never credit-gated | Free (1 seat, 5k traces/mo); $39/seat/mo + usage |

## What does LangSmith do that Overmind does not?

LangSmith goes deeper on seeing and reviewing what an agent did. Annotation queues, pairwise A/B review and bulk actions, tracing SDKs for Python, TypeScript, Go and Java with end-to-end OpenTelemetry, and self-hosting of the whole platform on the Enterprise plan. Overmind's platform self-hosts too, under AGPL-3.0 with Docker Compose, but it still calls OpenRouter, Modal, AWS S3 and Hugging Face, so it cannot run in a closed network. If the job is observing and scoring the agent you already have, LangSmith is the better buy.

## What else do teams ask about LangSmith and Overmind?

### Is Overmind a replacement for LangSmith?

Only if you want it to be. Overmind has its own tracing and evals, so it stands alone. Teams that like LangSmith keep it for observability and use Overmind for datasets, training, benchmarking, and serving.

### Does LangSmith do fine-tuning?

Yes, since 24 September 2026. LangSmith Fine-Tuning, with the smithtune CLI (public beta), turns LangSmith trajectories into a curated dataset, runs supervised fine-tuning with LoRA on Fireworks or Baseten using your own provider account, compares the result with the base model in a replay evaluation, and can deploy it. Overmind covers the same steps and also takes traces from other tools (OTLP, Langfuse, Braintrust, Galileo and file uploads), audits datasets in Data Workshop, can benchmark against your production model, hosts the model on its own OpenAI-compatible endpoint, and lets you download the weights.

### Do I need LangChain to use either?

No. LangSmith traces any framework via OpenTelemetry. Overmind has no framework at all: its Python SDK instruments OpenAI, Anthropic, and Gemini clients directly, and any OTel exporter can send to it.

### What do they cost?

LangSmith: free for one seat and 5,000 traces a month, then $39 per seat per month plus usage units; self-hosting is Enterprise-only. Overmind: Free and Pro subscriptions plus credits, and trace ingest never draws credits. As of August 2026. Overmind's AGPL-3.0 platform can also be self-hosted with no license fee; you pay for the services it calls.

## Sources and setup guides

- [LangSmith observability docs](https://docs.langchain.com/langsmith/observability)
- [LangSmith Engine](https://docs.langchain.com/langsmith/engine)
- [LangSmith deployment options](https://docs.langchain.com/langsmith/platform-setup)
- [LangSmith Fine-Tuning (smithtune)](https://www.langchain.com/blog/langsmith-fine-tuning)
- [Overmind connectors](https://docs.overmindlab.ai/core/observability#getting-traces-in)
- [Overmind optimizer](https://docs.overmindlab.ai/agent-testing/optimisers)
- [Overmind training](https://docs.overmindlab.ai/models/training)
- [Overmind self-hosting](https://docs.overmindlab.ai/platform/self-hosting)

## Which other tools does Overmind compare with?

- [Overmind vs Langfuse](https://www.overmindlab.ai/compare/overmind-vs-langfuse)
- [Overmind vs OpenPipe](https://www.overmindlab.ai/compare/overmind-vs-openpipe)
- [Overmind vs OpenAI fine-tuning](https://www.overmindlab.ai/compare/openai-fine-tuning-alternatives)
- [All comparisons](https://www.overmindlab.ai/compare)
- [Start free](https://console.overmindlab.ai)

---

# https://www.overmindlab.ai/compare/overmind-vs-langfuse

# Overmind vs Langfuse

> Canonical: https://www.overmindlab.ai/compare/overmind-vs-langfuse

By Tyler Edwards, CEO, Overmind

*Last reviewed: 2026-10-05*

## In short

Langfuse is the open-source standard for agent observability, and it self-hosts free under an MIT license. Overmind imports Langfuse traces today and runs the loop Langfuse leaves out: audited datasets, fine-tuning, an optional benchmark against your production model, and serving.

Langfuse is the open-source standard for agent observability, with tracing, evals, prompt management, datasets, playground.

**Bottom line:** Overmind runs the loop Langfuse deliberately leaves out. Audited datasets, fine-tuning, an optional benchmark against production, serving. And it imports Langfuse traces today.

[Start for free](https://console.overmindlab.ai)

## Start with the traces already in Langfuse

You can assess the dataset and evaluation workflow while continuing to use Langfuse. Begin with a representative slice of one production task. Model training is optional.

1. **Connect Langfuse.** Open Observability, then Integrations, choose Langfuse, and add the public key, secret key and host for your project. The connector backfills retained history, then syncs incrementally. Sync now requests a manual pull. [Connector setup](https://docs.overmindlab.ai/core/observability#getting-traces-in)
2. **Build an evaluation dataset.** Review the imported traces, select examples for one task, and choose Add to dataset. Pick Evaluation, review the field mapping and source provenance, then inspect quality issues in Data Workshop. [Trace-to-dataset guide](https://docs.overmindlab.ai/core/datasets)
3. **Compare against a baseline.** Define an eval set for the task and create an evaluation run from your dataset or captured traces. Pin a baseline and the quality criteria. Read individual failures as well as the aggregate score before choosing what to change next. [Evaluation setup](https://docs.overmindlab.ai/agent-testing/eval#eval-runs)

The connector reads from Langfuse, so available history depends on what Langfuse retains and what your API key can see. Imported traces become datasets, but scoring on arrival only applies to spans sent to Overmind's own OTLP endpoint. Evaluating outputs does not replay your production tools, and the optimizer runs from the CLI in your own repository and needs an executable agent.

## Is Overmind a Langfuse alternative?

Langfuse describes fine-tuning as an export: trace data leaves for external tools. Overmind is one of the places it can go, connector included.

- Traces: Overmind covered (OTLP + Langfuse); Langfuse covered (OTel + 100 integrations)
- Datasets: Overmind covered (full); Langfuse covered (versioned)
- Evals: Overmind covered (full); Langfuse covered (judge + code)
- Optimize: Overmind covered (full); Langfuse partial (manual playground)
- Train: Overmind covered (full); Langfuse not offered (export only)
- Serve: Overmind covered (full); Langfuse not offered (BYO model key)

## Should you replace Langfuse or run both?

Three of the seven verdicts here end at Langfuse. A stack with no commercial services and regional data residency are two things Overmind has no answer to.

### Collect first. Both doors stay open.

Without traffic there is nothing to curate, train, or benchmark. Langfuse is the cheapest way to start filling: MIT, self-hosted, no usage limits, nothing to sign. Come back when you have a few hundred real runs and a job worth specializing, and the connector imports the lot.

*Langfuse self-hosting costs nothing at any scale. Overmind has no reason to argue with that.*

### Langfuse is the fit.

If the job is seeing and scoring what the agent does, Langfuse does it openly and at no cost. MIT-licensed end to end, 100+ integrations, native OpenTelemetry, and self-hosting with no usage limits. Overmind ships its own tracing, but does not try to win on that ground.

*50 million SDK installs a month, 33k GitHub stars, ClickHouse behind it since January 2026.*

### Keep the frontier model. Take the loop.

A small specialist wins on a narrow job, not on open-ended reasoning across domains, so do not plan on swapping the model. Take the part Langfuse deliberately leaves out: an optimizer that rewrites prompts, tool definitions, and logic as scored git diffs. Point it at traces the connector already imported.

*The optimizer scores every candidate diff against your eval set and hands back the winner as a git diff you apply. No training involved.*

### Langfuse, and only Langfuse.

A blanket no-commercial-SaaS rule rules Overmind out. Its platform is open source under AGPL-3.0 and self-hosts, but it will not start without OpenRouter, Modal, AWS S3 and Hugging Face accounts, and training runs on Modal's machines. Langfuse is MIT end to end and self-hosts on Helm or Terraform. That is the honest answer.

*The nearest Overmind gets is a self-hosted platform and weights you download and serve yourself. Training still runs on Modal.*

### Langfuse holds the residency line.

Langfuse runs isolated EU, US, Japan, and HIPAA regions. Overmind's hosted service publishes no regional isolation, and a self-hosted Overmind still sends training data to Modal, so if your data has to stay in a named region this is not a close call: keep Langfuse for the trace store. A compliance requirement that is not about geography is worth putting to the team directly, because this page will not answer it.

*Self-hosting keeps traces on your infrastructure and trained weights run anywhere. Training is the half Overmind has no regional answer to.*

### Keep Langfuse. Add Overmind.

An open, self-hosted trace store and a training loop are not the same purchase. Langfuse stays where it is; the connector reads from it with encrypted credentials, a resumable backfill, then incremental sync. Imported traces become datasets, evals, and a trained model on the Overmind side. One scope note: scoring on arrival only applies to spans sent to Overmind's own OTLP endpoint, so teams that want it dual-send.

*Langfuse, LangSmith, Braintrust and Galileo connectors are live. Each backfills retained history, then syncs incrementally.*

### Overmind is the fit.

Choosing fresh, with a repeatable job and real traffic behind it, means you can skip the export step entirely: OTLP ingest, audited datasets, generated evals, the optimizer, training, and an optional benchmark against the model you run today. Ingest never draws credits, so the observability half costs nothing here either.

*A 0.5B specialist beat GPT-4o mini 50% to 29% on the same product eval, and the weights are yours to download.*

## How does Overmind compare with Langfuse feature by feature?

| Dimension | Overmind | Langfuse |
| --- | --- | --- |
| License | AGPL-3.0 platform, MIT SDK. Trained weights are yours | MIT, entire product, no usage limits |
| Self-hosting | Docker Compose or console.overmindlab.ai. Needs OpenRouter, Modal, S3, HF | Whole platform, free forever. Helm and Terraform |
| Fine-tuning | LoRA or full, hyperparameters recommended from your data | None. Data exports for external tools |
| Model serving | OpenAI-compatible endpoint, weights downloadable | None. Playground and judges use your API keys |
| Prompt improvement | Optimizer returns scored diffs you apply | Versioned prompts, instant rollbacks, manual editing |
| Proof before switching | Base vs trained, overlap reported. Production model opt-in | Experiments on datasets, app-level |
| Data residency | No published regional isolation. Self-host keeps traces in-house | Isolated EU, US, Japan, and HIPAA regions |
| Pricing | Free / Pro + credits. Ingest never credit-gated | Free 50k units/mo; $29, $199, $2,499 tiers; self-host $0 |

## What does Langfuse do that Overmind does not?

Langfuse is MIT-licensed end to end and self-hosts free at any scale, with Helm and Terraform, versioned prompts with instant rollbacks, 100+ integrations, and isolated EU, US, Japan and HIPAA regions. Overmind's platform is open source under AGPL-3.0 and self-hosts with Docker Compose, but it needs OpenRouter, Modal, AWS S3 and Hugging Face to run, and its hosted service publishes no regional isolation. A no-commercial-SaaS rule or a named-region requirement settles the question in Langfuse's favor. Self-hosting and downloadable weights solve the trace store and serving, not training.

## What else do teams ask about Langfuse and Overmind?

### Is Overmind a replacement for Langfuse?

It does not have to be. The common pattern is Langfuse for observability, Overmind for the loop from its traces to datasets, evals, and a trained model, via the live import connector. Overmind ships its own tracing too, so starting fresh with one tool also works.

### Does Langfuse do fine-tuning or serving?

No. As of August 2026 its docs describe exporting trace data for fine-tuning in external tools, and every model-touching feature uses your own provider API key. It observes and evaluates models; it does not train or host them.

### Langfuse self-hosts for free. What does Overmind cost?

Free and Pro subscriptions plus credits for compute-heavy work: dataset analysis, evals, optimization, training, serving. Trace ingest never draws credits. Overmind's AGPL-3.0 platform self-hosts too, but it needs OpenRouter, Modal, AWS S3 and Hugging Face accounts to run. If the requirement is a free self-hosted trace store, Langfuse wins that outright; Overmind's pitch is the loop, not the store.

### Can I migrate gradually?

Yes. Backfill via the connector while your instrumentation stays put, and curate datasets from imported traces. Add Overmind's OTel endpoint alongside Langfuse's later if you want live scoring. No rewrite at any step.

## Sources and setup guides

- [Langfuse docs](https://langfuse.com/docs)
- [Langfuse self-hosting and licensed features](https://langfuse.com/self-hosting)
- [Langfuse evaluations](https://langfuse.com/docs/evaluation/overview)
- [Overmind connectors](https://docs.overmindlab.ai/core/observability#getting-traces-in)
- [Overmind optimizer](https://docs.overmindlab.ai/agent-testing/optimisers)
- [Overmind training](https://docs.overmindlab.ai/models/training)
- [Overmind self-hosting](https://docs.overmindlab.ai/platform/self-hosting)

## Which other tools does Overmind compare with?

- [Overmind vs LangSmith](https://www.overmindlab.ai/compare/overmind-vs-langsmith)
- [Overmind vs OpenPipe](https://www.overmindlab.ai/compare/overmind-vs-openpipe)
- [Overmind vs OpenAI fine-tuning](https://www.overmindlab.ai/compare/openai-fine-tuning-alternatives)
- [All comparisons](https://www.overmindlab.ai/compare)
- [Start free](https://console.overmindlab.ai)

---

# https://www.overmindlab.ai/compare/overmind-vs-openpipe

# Overmind vs OpenPipe

> Canonical: https://www.overmindlab.ai/compare/overmind-vs-openpipe

By Tyler Edwards, CEO, Overmind

*Last reviewed: 2026-10-05*

## In short

OpenPipe stopped accepting training and inference on 30 July 2026, and its reinforcement-learning tooling lives on as W&B Training and the open-source ART trainer. Overmind runs the same idea as a live platform, with deep tracing, audited datasets, an optimizer, and an optional benchmark against production before any switch.

OpenPipe stopped accepting training and inference on 30 July 2026. Its RL tooling lives on as W&B Training, and there is an open-source trainer, ART.

**Bottom line:** Overmind runs the same idea as a live platform, with the loop OpenPipe never had. Deep tracing, audited datasets, an optimizer, and an optional benchmark against production before any switch.

[Start for free](https://console.overmindlab.ai)

## Is Overmind an OpenPipe alternative?

OpenPipe's coverage, shown as it stood at sunset. Since 30 July 2026 the platform accepts no new training or inference; RL continues inside W&B Training.

- Traces: Overmind covered (full); OpenPipe partial (request logs)
- Datasets: Overmind covered (full); OpenPipe covered (JSONL, relabelling)
- Evals: Overmind covered (full); OpenPipe partial (basic, TS-only)
- Optimize: Overmind covered (full); OpenPipe not offered (not offered)
- Train: Overmind partial (supervised only); OpenPipe not offered (closed to new runs)
- Serve: Overmind covered (full); OpenPipe not offered (closed to new runs)

## Where does the OpenPipe sunset leave you?

Overmind does not do reinforcement learning. Three of the answers here send you to ART and W&B Training instead of Overmind, and two more run them alongside.

### ART and W&B Training are the fit.

For GRPO over multi-step trajectories, that is the honest answer. ART is Apache-licensed, actively maintained, and now powers W&B Training, so OpenPipe's reinforcement-learning work continues under new ownership. Overmind trains LoRA and full fine-tunes and has nothing to offer here.

*ART sits at 10k+ GitHub stars. Overmind's training methods are supervised only, by design.*

### ART for the RL. Overmind for the rest.

They do not overlap much. Keep ART for reinforcement learning on your own GPUs, and let Overmind do the parts it has: OpenTelemetry capture, audited datasets with provenance, generated evals, and supervised runs you can benchmark against your production model. Datasets curated in Overmind export as JSONL, so ART can read them.

*This is the one arrangement where the sunset costs you nothing. Both halves survived it.*

### Instrument first. There is nothing to train on yet.

OpenPipe's premise was that production requests are training data, and that premise needs production requests. Ship it, capture over OpenTelemetry so the instrumentation outlives any vendor including Overmind, and come back with a few hundred real runs. Ingest never draws credits, so collecting costs nothing.

*The OTel choice is the point. OpenPipe's proprietary SDK wrapper is exactly what made its shutdown expensive.*

### Keep the frontier model. Take the loop.

A small specialist wins on a narrow job, not on open-ended reasoning across domains, so the OpenPipe trade does not apply to you as stated. What does apply: evals generated from your agent's own design, and an optimizer that rewrites prompts, tool definitions, and logic as scored git diffs. Neither OpenPipe nor its successor offers that.

*The optimizer scores every candidate diff against your eval set and hands back the winner as a git diff you apply. No training involved.*

### Overmind, starting with your data.

OpenPipe datasets are OpenAI-format JSONL, which the Data Workshop imports directly, audited and versioned on arrival. The SDK wrapper becomes three environment variables of OpenTelemetry config. Weights you exported stay yours, and weights trained here download the same way, so the exit door works in both directions.

*The one thing you gain on the way across: the option to benchmark against your production model before traffic moves. OpenPipe left that to you.*

### Overmind is the fit.

The OpenPipe premise kept whole, and widened. Capture is OpenTelemetry, so the instrumentation outlives the vendor. Datasets pass a two-pass audit, with provenance, before a single token trains. Training can benchmark against the model your agent runs today, with overlap reported, and going live is one Make live click on the capability's model alias.

*A 0.5B specialist beat GPT-4o mini 50% to 29% on the same product eval, and the weights are yours to download.*

## How does Overmind compare with OpenPipe feature by feature?

| Dimension | Overmind | OpenPipe |
| --- | --- | --- |
| Status | Live, in active development | Sunset 30 July 2026; site is an archive |
| Data capture | OpenTelemetry-native, any OTel exporter | Proprietary SDK wrapper or proxy. Never OTel |
| Observability | Spans, sessions, live scores, capability map from your repo | Request logs with metadata tags |
| Datasets | Two-pass audit, provenance, versions. JSONL imports | Filtered logs or JSONL upload, stronger-model relabelling |
| Training | LoRA or full fine-tuning, parallel experiments | SFT and DPO historically; RL via ART / W&B Training |
| Code optimization | Scored diffs you apply locally | Not offered |
| Proof before switching | Base vs trained, overlap reported. Production model opt-in | Head-to-head evals; production baseline was your job |
| Weights | Downloadable, always | Exportable for open models; successor serves LoRA via W&B |
| License and hosting | AGPL-3.0 platform, MIT SDK. Self-host or console.overmindlab.ai | Hosted platform, now closed. ART trainer is Apache-2.0 |

## What does OpenPipe do that Overmind does not?

OpenPipe itself takes no new training or inference after 30 July 2026, so the live comparison is with its successor. Reinforcement learning over multi-step trajectories, GRPO and RULER, continues in the open-source ART trainer and W&B Training; Overmind trains with LoRA and full fine-tuning only and has nothing to offer there. OpenPipe's dataset flow also relabelled rows with a stronger model, where Overmind's Data Workshop audits and versions them instead.

## What else do teams ask after the OpenPipe sunset?

### Is OpenPipe still available?

Not as a standalone platform. CoreWeave acquired it in September 2025, and openpipe.ai stopped accepting new training and inference on 30 July 2026. The site describes itself as an archive; the technology continues inside W&B Training. ART, the open-source RL trainer, remains active.

### Can I import my OpenPipe datasets into Overmind?

Yes. They are OpenAI-format JSONL, which the Data Workshop imports directly. Imported rows get the same treatment as native ones: quality audit, provenance, versioned steps.

### Does Overmind do RL like ART?

No. Overmind does supervised fine-tuning only, LoRA or full, with an optional benchmark against your production model. For GRPO-style RL over multi-step trajectories, ART and W&B Training are the right tools, and they can sit alongside Overmind's observability and datasets.

### How does pricing compare?

OpenPipe's per-token rates stopped applying when the platform closed to new work in July 2026; its successor prices through Weights & Biases. Overmind: Free and Pro subscriptions plus credits, with trace ingest never credit-gated. As of August 2026.

## Sources and setup guides

- [CoreWeave to acquire OpenPipe (September 2025)](https://coreweave.com/news/coreweave-to-acquire-openpipe-leader-in-reinforcement-learning)
- [OpenPipe training and inference migrating to W&B (sunset 30 July 2026)](https://openpipe.ai/blog/openpipe-training-inference-migrating-to-wandb)
- [ART, the open-source RL trainer](https://github.com/OpenPipe/ART)
- [Overmind datasets](https://docs.overmindlab.ai/core/datasets)
- [Overmind training](https://docs.overmindlab.ai/models/training)
- [Overmind self-hosting](https://docs.overmindlab.ai/platform/self-hosting)

## Which other tools does Overmind compare with?

- [Overmind vs LangSmith](https://www.overmindlab.ai/compare/overmind-vs-langsmith)
- [Overmind vs Langfuse](https://www.overmindlab.ai/compare/overmind-vs-langfuse)
- [Overmind vs OpenAI fine-tuning](https://www.overmindlab.ai/compare/openai-fine-tuning-alternatives)
- [All comparisons](https://www.overmindlab.ai/compare)
- [Start free](https://console.overmindlab.ai)

---

# https://www.overmindlab.ai/compare/openai-fine-tuning-alternatives

# Alternatives to OpenAI fine-tuning, and how to migrate before 6 January 2027

> Canonical: https://www.overmindlab.ai/compare/openai-fine-tuning-alternatives

By Tyler Edwards, CEO, Overmind

*Last reviewed: 2026-10-05*

## In short

OpenAI is winding down self-serve fine-tuning. From 6 January 2027 no customer can create new jobs, and six fine-tuned models shut down on 23 October 2026. You have three options. Stay on an OpenAI base model with prompting, retrain with another provider such as Together AI or Fireworks AI, or use Overmind to train an open-weight model you own.

Overmind traces your OpenAI SDK calls in Python, turns recent traffic and your original training file into train and eval datasets, fine-tunes an open-weight model, and serves it on an OpenAI-compatible endpoint. The code change is the base URL and the model name.

**Bottom line:** Start tracing while the old fine-tune still answers. Its recorded replies are the baseline the new model has to match, and for the six models on the 23 October list that window closes first.

[Start for free](https://console.overmindlab.ai)

## When does OpenAI fine-tuning shut down?

Six facts from OpenAI's own pages, in date order. Inference on existing fine-tunes continues until OpenAI deprecates each base model.

| Date | What changes | Source |
| --- | --- | --- |
| 7 May 2026 | Organisations that had never run fine-tuning can no longer create jobs. | [OpenAI deprecations](https://developers.openai.com/api/docs/deprecations), checked 5 October 2026 |
| 2 July 2026 | Organisations with no fine-tuned inference in the past 60 days can no longer create jobs. | [OpenAI deprecations](https://developers.openai.com/api/docs/deprecations), checked 5 October 2026 |
| 23 October 2026 | `ft-gpt-3.5-turbo`, `ft-gpt-4`, `ft-gpt-4.1-nano-2025-04-14`, `ft-o4-mini-2025-04-16`, `ft-babbage-002` and `ft-davinci-002` shut down. | [OpenAI deprecations](https://developers.openai.com/api/docs/deprecations), checked 5 October 2026 |
| 31 October and 30 November 2026 | Existing evals become read-only, then the Evals dashboard and API shut down. OpenAI's migration note points to Promptfoo. | [OpenAI deprecations](https://developers.openai.com/api/docs/deprecations), checked 5 October 2026 |
| 6 January 2027 | No customer can create new fine-tuning jobs. Existing fine-tunes keep serving until their base model is deprecated. | [OpenAI deprecations](https://developers.openai.com/api/docs/deprecations), checked 5 October 2026 |
| No date | There is no documented way to export fine-tuned weights. | [OpenAI fine-tuning API reference](https://developers.openai.com/api/reference/resources/fine_tuning), checked 5 October 2026 |

## What are the alternatives to OpenAI fine-tuning?

Overmind, Together AI, Fireworks AI and distil labs all train open-weight models you can download, and all serve them behind an OpenAI-compatible API. They differ in what you upload, how you check the new model against the old one, and how you pay. Rival cells link to that vendor's own docs, checked 5 October 2026.

| Option | What you upload | Weights download | Serving and compatibility | Evals against your current model | Open source or self-host | Price basis |
| --- | --- | --- | --- | --- | --- | --- |
| **Overmind** | Traces from the Python SDK, which instruments the OpenAI SDK. Or files (CSV, TSV, JSON, JSONL, NDJSON, Parquet), or Langfuse, LangSmith, Braintrust and Galileo connectors ([datasets](https://docs.overmindlab.ai/core/datasets)) | Yes. Checkpoint zip from the Console or `overmind model download-checkpoint` ([training](https://docs.overmindlab.ai/models/training)) | OpenAI-compatible chat completions endpoint. Scales to zero; cold starts take 2 to 7 minutes ([inference](https://docs.overmindlab.ai/models/inference)) | Opt-in. Score the trained model against the replies your current model returned, recorded in your traces, with five evaluator kinds ([eval runs](https://docs.overmindlab.ai/agent-testing/eval)) | Yes. AGPL-3.0 platform, MIT SDK, self-host with Docker Compose ([self-hosting](https://docs.overmindlab.ai/platform/self-hosting)) | Free and Pro plans plus credits. Fine-tuned inference is billed on GPU time ([pricing](/pricing)) |
| **Together AI** | A prepared dataset, JSONL or Parquet in six schemas. No trace import documented ([data preparation](https://docs.together.ai/docs/fine-tuning/data-preparation.md)) | Yes. CLI or SDK download as .tar.zst; LoRA as merged, adapter or raw ([deployment](https://docs.together.ai/docs/fine-tuning/deployment.md)) | Dedicated endpoints only, billed per minute per replica even when idle ([deployment](https://docs.together.ai/docs/fine-tuning/deployment.md)). Chat completions documented as OpenAI-compatible ([OpenAI compatibility](https://docs.together.ai/docs/inference/openai-compatibility.md)) | A separate Evaluations product. Its compare type can test a fine-tuned endpoint against another model; you build the run ([run an evaluation](https://docs.together.ai/docs/run-an-evaluation.md)) | No platform source found in its GitHub org. SDKs are Apache-2.0 ([GitHub](https://github.com/togethercomputer)) | Per 1M training tokens, $0.34 to $40 by model, with a per-job minimum. Hosting per GPU minute ([pricing](https://www.together.ai/pricing)) |
| **Fireworks AI** | JSONL in OpenAI chat format. Existing OpenAI SFT files work unchanged ([managed training](https://docs.fireworks.ai/fine-tuning/managed-finetuning-intro.md)) | Yes. `firectl model download`, then merge the adapter ([deploying trained models](https://docs.fireworks.ai/fine-tuning/deploying-loras.md)) | Dedicated deployments only, billed per GPU second ([serving fees](https://docs.fireworks.ai/faq-new/billing-pricing/are-there-extra-fees-for-serving-fine-tuned-models.md)). Called with the OpenAI SDK ([OpenAI compatibility](https://docs.fireworks.ai/tools-sdks/openai-compatibility.md)) | Evaluators score outputs on your data. No one-step comparison with your current model documented ([evaluating trained models](https://docs.fireworks.ai/fine-tuning/evaluating-fine-tuned-models.md)) | No platform source found in its GitHub org. Eval Protocol is MIT ([GitHub](https://github.com/fw-ai)) | Per 1M training tokens, LoRA SFT $0.50 to $10 by model size. Hosting per GPU second ([pricing](https://fireworks.ai/pricing)) |
| **distil labs** | Calls recorded through its OpenAI-compatible endpoint (the model must be one OpenRouter serves), exported logs as JSONL, or 20 or more examples. A teacher model then generates the training data ([FAQ](https://distillabs.ai/faq)) | Yes, as a LoRA adapter for vLLM ([local deployment](https://distillabs.ai/docs/deployment/local-deployment/)). Its FAQ and pricing page differ on commercial self-hosting terms ([FAQ](https://distillabs.ai/faq)) | Dedicated H100 endpoint behind an OpenAI-compatible API ([inference](https://distillabs.ai/docs/deployment/inference/)) | Built in. Scored against your original production model when you start from traces ([model iterations](https://distillabs.ai/docs/build/improving/)) | No open-source statement found. The Enterprise tier runs in your cloud or VPC ([pricing](https://distillabs.ai/pricing)) | Free tier with 2 training runs. Production billed per GPU hour of uptime ([pricing](https://distillabs.ai/pricing)) |
| **Stay on an OpenAI base model, with prompting** | Nothing to train. Examples move into the prompt | None. Fine-tuned weights have no documented export ([API reference](https://developers.openai.com/api/reference/resources/fine_tuning)) | OpenAI's API, as today, with a new model name | Evals read-only from 31 October 2026, shut on 30 November 2026 ([deprecations](https://developers.openai.com/api/docs/deprecations)) | No self-host option documented ([model optimization guide](https://developers.openai.com/api/docs/guides/model-optimization)) | Per token ([pricing](https://developers.openai.com/api/docs/pricing)) |

If all you need is the same file retrained, Fireworks is the most direct like-for-like move. Overmind fits when you want the new model trained on recent production traffic and checked against what your fine-tune actually returned. Browse the trainable bases in the [model library](/models). OpenPipe users went through the same move a vendor earlier; see [Overmind vs OpenPipe](/compare/overmind-vs-openpipe).

## How do you migrate a fine-tuned OpenAI model?

Start while the old fine-tune still answers, because its replies become the baseline. Each step links to the docs page that covers it.

1. **Export your training file.** Start from the JSONL file you trained the OpenAI model on. Overmind's training contract is a messages column in OpenAI chat format. Uploads accept CSV, TSV, JSON, JSONL, NDJSON and Parquet, and land exactly as they arrived. [Upload a dataset](https://docs.overmindlab.ai/core/datasets)
2. **Trace recent traffic.** Add overmind.init(providers="auto") to the Python service that calls OpenAI. It traces OpenAI SDK calls, so every request your fine-tune handles is recorded with its reply. Any OpenTelemetry exporter can send traces too. [Tracing setup](https://docs.overmindlab.ai/core/observability)
3. **Build the eval dataset.** Select recent traces, choose Add to dataset, and pick Train + eval. The evaluation share defaults to 30%. The Data Workshop's agent prepares and audits the rows, and proposes changes you approve or deny. [Datasets guide](https://docs.overmindlab.ai/core/datasets)
4. **Train an open-weight model.** Run supervised fine-tuning, LoRA or full, on one of 38 open-weight bases from the Qwen, Llama, Gemma 4, LFM2.5, Antares, GPT-OSS, Nemotron and Muse Glimmer families. [Training guide](https://docs.overmindlab.ai/models/training)
5. **Compare before you switch.** Score the trained model on the eval set. Its expected outputs are the replies your current model returned, so the check is against what runs today. The default benchmark compares the untouched base with the trained model; adding an incumbent comparison is opt-in. [The benchmark](https://docs.overmindlab.ai/models/training)
6. **Switch, and keep the fallback.** Point the OpenAI SDK at Overmind's endpoint and change the model id. Make live switches a capability alias with no deploy. Keep the OpenAI fine-tune callable as a fallback until OpenAI deprecates its base model. [Calling your model](https://docs.overmindlab.ai/models/inference)

The comparison uses replies recorded in your traces, so trace before the old model goes away. Serving scales to zero, so the first request after an idle spell can take 2 to 7 minutes. Code on OpenAI's Responses API moves to chat completions first.

## What changes in your code?

If your code calls chat completions on the OpenAI SDK, two arguments change. Point the client at Overmind with an Overmind API key, and swap the model id.

**Before**, calling your OpenAI fine-tune

```python
client = OpenAI()
reply = client.chat.completions.create(model="ft:gpt-4.1-mini-2025-04-14:your-org::abc123", messages=messages)
```

**After**, calling the model you trained on Overmind

```python
client = OpenAI(base_url="https://api.overmindlab.ai/api/v1", api_key=os.environ["OVERMIND_API_KEY"])
reply = client.chat.completions.create(model="overmind/<capability-uuid>", messages=messages)
```

The alias `overmind/<capability-uuid>` follows whichever model you make live, so later swaps need no code change. To pin one version, use its `ft-<job>-<base>` id instead. Code built on OpenAI's Responses API moves to chat completions first. Details in the [inference docs](https://docs.overmindlab.ai/models/inference).

## When should you stay on OpenAI?

- Your fine-tune is not on the 23 October list, it works, and you will not need to retrain it before 6 January 2027. It keeps serving until OpenAI deprecates its base model.
- Your job needs a proprietary GPT model, DPO, vision fine-tuning or reinforcement fine-tuning. Overmind trains open-weight models with supervised fine-tuning only.
- A current OpenAI base model with examples in the prompt matches your fine-tune on your own eval set. Then there is nothing to train.

Staying still has a deadline. After 6 January 2027 any retrain has to happen somewhere else, so plan the move before then.

## How do you move off OpenAI Evals?

Existing OpenAI evals become read-only on 31 October 2026, and the Evals dashboard and API shut down on 30 November 2026. OpenAI's migration note points to Promptfoo.

In Overmind, an eval run scores one or more variants against a baseline on a frozen dataset version. Build that dataset from traces of the same traffic, so the expected outputs are what your model really returned. There are five evaluator kinds (LLM judge, deterministic, trajectory, statistical and agentic). See [eval runs](https://docs.overmindlab.ai/agent-testing/eval).

## Is Overmind an alternative to OpenAI fine-tuning?

OpenAI fine-tuning as of 5 October 2026. Existing fine-tunes keep serving until their base model is deprecated; no customer can create new jobs from 6 January 2027.

- Traces: Overmind covered (OpenAI SDK traced); OpenAI fine-tuning partial (stored outputs, 30 days)
- Datasets: Overmind covered (full); OpenAI fine-tuning partial (JSONL upload)
- Evals: Overmind covered (full); OpenAI fine-tuning partial (shuts 30 Nov 2026)
- Train: Overmind partial (supervised only); OpenAI fine-tuning partial (no new jobs from 6 Jan 2027)
- Serve: Overmind covered (full); OpenAI fine-tuning partial (until base deprecated)

## Where should your fine-tune go?

Four of the five answers here send you somewhere other than Overmind. Staying on OpenAI is a real option until 6 January 2027.

### Stay on OpenAI for now.

Existing fine-tunes keep serving until OpenAI deprecates their base model, so nothing forces a move today. Plan the next step before 6 January 2027, because after that date any retrain has to happen elsewhere.

*OpenAI deprecations page, checked 5 October 2026.*

### A prompted OpenAI base model.

If the fine-tune mostly taught format or tone, test a current OpenAI base model with examples in the prompt against the replies your fine-tune returned. No training and no new vendor. Keep the eval set, because OpenAI Evals shut on 30 November 2026.

*Inference on existing fine-tunes continues until the base model is deprecated.*

### Fireworks is the fastest like-for-like move.

Fireworks documents that existing OpenAI SFT files work unchanged. It trains LoRA adapters on open models, serves them on dedicated deployments you call with the OpenAI SDK, and lets you download the weights.

*Fireworks managed training and deployment docs, checked 5 October 2026.*

### distil labs fits that shape.

distil labs starts from 20 or more examples or recorded traffic, uses a teacher model to generate the training data, and scores the trained model against your original production model. Overmind trains on the data you have and adds synthetic rows only when you ask.

*distil labs FAQ and docs, checked 5 October 2026.*

### Overmind is the fit.

Trace your OpenAI SDK calls, build train and eval datasets from recent traffic and your original training file, and fine-tune an open-weight model. Score it against the replies your current model returned, then serve it on an OpenAI-compatible endpoint. The weights are yours to download.

*38 open-weight bases from 230M to 72B, LoRA or full fine-tuning.*

## How does Overmind compare with OpenAI fine-tuning?

| Dimension | Overmind | OpenAI fine-tuning |
| --- | --- | --- |
| New training jobs | Available, hosted or self-hosted | None for any customer from 6 January 2027; already closed to some organisations |
| Models | 38 open-weight bases from 230M to 72B, including Qwen, Llama, Gemma 4 and GPT-OSS | Proprietary only. SFT and DPO on gpt-4.1, gpt-4.1-mini and gpt-4.1-nano |
| Methods | Supervised fine-tuning, LoRA or full | SFT and DPO; vision on gpt-4o; reinforcement fine-tuning on o4-mini |
| Training data | Traces, file uploads, or Langfuse, LangSmith, Braintrust and Galileo connectors | Prepared JSONL, minimum 10 examples. No trace import documented |
| Weights | Downloadable checkpoint zip | No documented export |
| Serving | OpenAI-compatible chat completions; scales to zero | OpenAI's API, until the base model is deprecated |
| Evals | Eval runs with five evaluator kinds, on datasets from your traces | Read-only from 31 October 2026, shut down 30 November 2026 |
| Open source | AGPL-3.0 platform, MIT SDK, Docker Compose self-host | No self-host option documented |
| Pricing | Free and Pro plans plus credits; fine-tuned inference on GPU time | Per token. Training gpt-4.1 $25 per 1M; tuned gpt-4.1 inference $3 in, $12 out |

## What does OpenAI fine-tuning do that Overmind does not?

OpenAI fine-tunes proprietary GPT models, including gpt-4.1, and offers DPO, vision fine-tuning on gpt-4o and reinforcement fine-tuning on o4-mini. Overmind trains open-weight models with supervised fine-tuning only. Existing OpenAI fine-tunes also keep serving on OpenAI's API until their base model is deprecated, with no change to your code. If yours works and you will not retrain it before 6 January 2027, staying is a reasonable choice.

## What else do teams ask about moving off OpenAI fine-tuning?

### Is OpenAI shutting down fine-tuning?

OpenAI is winding down self-serve fine-tuning. Since 7 May 2026 organisations that had never fine-tuned cannot start, and since 2 July neither can those without fine-tuned inference in the past 60 days. From 6 January 2027 no customer can create new jobs. Existing fine-tuned models keep serving until their base model is deprecated (OpenAI deprecations page, checked 5 October 2026).

### Which fine-tuned OpenAI models shut down first?

On 23 October 2026 OpenAI removes ft-gpt-3.5-turbo, ft-gpt-4, ft-gpt-4.1-nano-2025-04-14, ft-o4-mini-2025-04-16, ft-babbage-002 and ft-davinci-002, along with several base models. Applications calling them need a tested replacement before that date; OpenAI lists recommended substitutes on its deprecations page.

### Can I download the weights of my OpenAI fine-tuned model?

OpenAI documents no way to export fine-tuned weights. Moving means training a new model on the same job, starting from your original training file and your recent production traffic, then checking it against what the old model returned.

### Do I have to change my code to move off OpenAI?

If your code calls chat completions, usually only the base URL, API key and model name. Overmind serves trained models on an OpenAI-compatible chat completions endpoint, so the OpenAI SDK keeps working for those calls. Code built on OpenAI's Responses API has to move to chat completions first.

### What happens to my OpenAI evals?

Existing evals become read-only on 31 October 2026 and the Evals dashboard and API shut down on 30 November 2026. Overmind's eval runs score variants against a baseline with five evaluator kinds, on datasets built from your production traces.

## Sources and setup guides

- [OpenAI deprecations](https://developers.openai.com/api/docs/deprecations)
- [OpenAI supervised fine-tuning guide](https://developers.openai.com/api/docs/guides/supervised-fine-tuning)
- [OpenAI model optimization guide](https://developers.openai.com/api/docs/guides/model-optimization)
- [OpenAI fine-tuning API reference](https://developers.openai.com/api/reference/resources/fine_tuning)
- [OpenAI Evals guide](https://developers.openai.com/api/docs/guides/evals)
- [OpenAI pricing](https://developers.openai.com/api/docs/pricing)
- [Together AI fine-tuning](https://docs.together.ai/docs/fine-tuning/overview.md)
- [Together AI pricing](https://www.together.ai/pricing)
- [Fireworks AI managed training](https://docs.fireworks.ai/fine-tuning/managed-finetuning-intro.md)
- [Fireworks AI pricing](https://fireworks.ai/pricing)
- [distil labs FAQ](https://distillabs.ai/faq)
- [distil labs pricing](https://distillabs.ai/pricing)
- [Overmind datasets](https://docs.overmindlab.ai/core/datasets)
- [Overmind training](https://docs.overmindlab.ai/models/training)
- [Overmind inference](https://docs.overmindlab.ai/models/inference)

## Which other tools does Overmind compare with?

- [Overmind vs LangSmith](https://www.overmindlab.ai/compare/overmind-vs-langsmith)
- [Overmind vs Langfuse](https://www.overmindlab.ai/compare/overmind-vs-langfuse)
- [Overmind vs OpenPipe](https://www.overmindlab.ai/compare/overmind-vs-openpipe)
- [All comparisons](https://www.overmindlab.ai/compare)
- [Start free](https://console.overmindlab.ai)

---

# https://www.overmindlab.ai/glossary

# Glossary

> Canonical: https://www.overmindlab.ai/glossary

The terms AI teams meet when they trace, evaluate and train agents, defined plainly. Listed alphabetically.

## Terms

- [LLM observability](https://www.overmindlab.ai/glossary/llm-observability): LLM observability records every prompt, completion, tool call and retry your AI application makes as traces, so you can explain any single run. It shows what happened; judging whether the output was good is a separate job called evaluation.

---

# https://www.overmindlab.ai/glossary/llm-observability

# What is LLM observability?

> Canonical: https://www.overmindlab.ai/glossary/llm-observability

- Author: Tyler Edwards, CEO, Overmind
- Published: 2026-09-11
- Last reviewed: 2026-09-11
- Also called: LLM monitoring, agent observability

**In short:** LLM observability records every prompt, completion, tool call and retry your AI application makes as traces, so you can explain any single run. It shows what happened; judging whether the output was good is a separate job called evaluation.

**LLM observability is the practice of capturing every prompt, completion, tool call and retry an AI application makes as structured traces, then using those traces to explain why the system behaved the way it did. It is usually built on OpenTelemetry spans, and it covers three things ordinary application monitoring does not: non-deterministic output, token cost per request, and multi-step agent trajectories. It tells you what happened. It does not tell you whether what happened was any good.**

## Who is writing this, and why?

The last sentence of the definition is the whole argument of this page, so it is worth being clear about where it comes from. Overmind is a model training platform for AI teams: it turns production traces into specialized models a team owns, trained, benchmarked and served. Observability is the first layer of that loop rather than the endpoint. Overmind ingests OpenTelemetry from anywhere and has a direct interest in you having good traces. What follows is about making traces useful, whichever tool you collect them in.

## What does LLM observability capture?

A trace is the set of spans sharing one trace ID. The root span is the end-to-end run: one user request, one agent invocation, one scheduled job. Its children are the individual steps.

For an LLM application, each span carries more than a normal HTTP span does:

| What the span records | Why it exists | Typical attribute |
| --- | --- | --- |
| The rendered prompt | The template is not the prompt. Variables, retrieved context and history are what the model actually saw | `gen_ai.input.messages` |
| The completion | The only place the model's actual words survive | `gen_ai.output.messages` |
| Model and version | Providers change weights behind a stable model name | `gen_ai.request.model`, `gen_ai.response.model` |
| Token counts, in and out | The unit your bill is denominated in | `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens` |
| Cached tokens | Prompt caching changes cost by an order of magnitude | `gen_ai.usage.cache_read_input_tokens` |
| Cost | Either stamped by the client or derived from tokens and model pricing | `genai.cost` |
| Tool name and arguments | Which tool the model chose, and what it passed | `gen_ai.operation.name = execute_tool`, `tool.name` |
| Latency and status | Where the time went and whether the step errored | OTel span timing and status |
| Session identity | Which multi-turn conversation this run belongs to | `conversation.id` |
| Agent identity | Which of your agents this is, across deploys | resource attribute, e.g. `overmind.agent.name` |

Put those together and you get the thing every vendor demos: a waterfall of the run, expandable to the exact text of every model call. That view is useful and it is solved. Most tools in the category do it well.

## How is LLM observability different from traditional observability?

Classic observability was built for deterministic systems. The same input produces the same output, so a failure is an exception, a bad status code, or a latency spike. LLM applications break every one of those assumptions.

|  | Traditional observability | LLM observability |
| --- | --- | --- |
| **Failure looks like** | An error, a timeout, a 500 | A confident, well-formatted, wrong answer that returns 200 |
| **"Correct" is** | Defined by a schema or a test | A judgment call, often without a reference answer |
| **The unit of work** | A request or a transaction | A trajectory: many model calls, tool calls and retries towards one goal |
| **Cost driver** | CPU, memory, egress | Tokens, and they scale with usage rather than with headcount |
| **Repeatability** | Same input, same output | Same input, different output, sometimes materially so |
| **The dangerous state** | Down | Up, fast, cheap, and quietly wrong |
| **Instrumentation** | Mature, mostly automatic | OTel GenAI conventions are still stabilising, and dialects disagree |

The three pillars of classic observability, metrics, logs and traces, still apply. They are just not sufficient on their own, because none of the three has an opinion about output quality. A dashboard that is green across metrics, clean across logs and complete across traces is entirely compatible with an agent that has been doing the wrong thing for a week.

## What is the difference between LLM monitoring, observability, evaluation and optimization?

These four words get used interchangeably by vendors, including by us in careless moments, and the distinction is the most useful thing on this page.

|  | Question it answers | Runs | Output | Needs a ground truth? |
| --- | --- | --- | --- | --- |
| **Monitoring** | Is it up, fast and inside budget? | Continuously, on aggregates | Alerts and dashboards | No |
| **Observability** | Why did this specific run do that? | On demand, per trace | Evidence: spans, prompts, tool calls | No |
| **Evaluation** | Was the output any good? | On a dataset, or on live traces | A score, a pass or a fail, and a reason | Yes, eventually |
| **Optimization** | How do I make it better? | On a change, against a baseline | A prompt diff, a code diff, or new model weights | Yes |

Monitoring tells you the agent is alive. Observability tells you what it did. Evaluation tells you whether that was acceptable. Optimization is the only one of the four that changes anything.

Most teams buy the first two, build a bit of the third by hand, and never reach the fourth. That is not a tooling failure so much as a sequencing habit: the first two are the ones that page you at 3am, so they get bought first.

## What is agent observability, and how is it different?

"Agent observability" is the term the conversation is moving to, and it is not just a rebrand. The difference is what counts as the unit of work.

|  | LLM observability | Agent observability |
| --- | --- | --- |
| **Unit** | One model call | One trajectory, from goal to outcome |
| **Question** | What did the model say? | Why did the agent take this path? |
| **Structure** | A list of calls | A tree of decisions, tool calls, retries and delegations |
| **Typical failure** | A hallucinated fact | A correct tool called on a wrong premise, three steps back |
| **Extra state** | None | Sub-agent handoffs, memory reads and writes, sandbox lifetime, partial commits |

Tracing a single agent run is largely solved. What is not solved is the run with forty-seven steps, partial state committed halfway through, and a sub-agent that returned a plausible summary of work it never did. Practitioners describe this consistently: tracing is fine until roughly twenty agents call each other, at which point, as one engineer put it in an [r/AI_Agents thread on enterprise agent platforms](https://reddit.com/r/AI_Agents/comments/1vscy5m/what_platforms_actually_help_enterprises_deploy/p5keasb/) in August 2026, "the observability breaks down, you get lost in the noise."

If you are instrumenting today, instrument for the trajectory. Give spans a type (entry point, workflow, tool call, function, model call, retrieval), stamp a stable agent identity on every span, and group multi-turn exchanges under one conversation ID. Auto-instrumentation alone gives you a flat list of model calls, and a flat list cannot show you a decision path.

## Why do agents fail silently even with full observability?

Because the failure modes that hurt do not produce a signal. Three of them recur constantly in production teams' own accounts.

**The silent success.** The agent returns a perfectly formatted completion summary describing work it did not do. Every span is green. Every status is 200. One engineer [on r/AI_Agents](https://reddit.com/r/AI_Agents/comments/1uwyf1h/oodleai_10_per_million_agent_traces/p5bp9iu/) described running a pipeline for a week before noticing the agent had been returning "task complete" with nothing underneath it. Nothing in the trace is wrong; the trace faithfully records a lie.

**The run that just stops.** A machine dies mid-run, a credential expires, a network call is blocked. The trace simply ends. There is no error span, because nothing errored, and there is no completion, because nothing completed. It looks identical to a run still in flight. One production team [put it precisely](https://reddit.com/r/AI_Agents/comments/1vscy5m/what_platforms_actually_help_enterprises_deploy/): ask any vendor what they show you when a run dies silently instead of erroring.

**The correct step on a wrong premise.** The agent calls the right tool with well-formed arguments, and the arguments encode a misunderstanding formed four steps earlier. Schema validation passes. Tool selection metrics pass. The output is wrong for reasons no per-span check can see, because the error is in the relationship between spans.

The pattern behind all three: **a trace is evidence, not a verdict.** Observability records. It cannot grade, because grading requires a definition of good, and that definition lives in your product, not in your telemetry.

This is also why "just add an alert" does not close the gap. You can alert on error rate, latency and spend, and all three will stay flat through every failure above.

## What should you do with the traces once you have them?

The honest answer is that the traces are an asset and most teams treat them as an archive. Four things turn them into something that changes behavior, in rough order of effort.

**1. Score them as they arrive, not when you go looking.** Attach evaluators to root spans so every production run gets graded on arrival, at whatever sampling rate you can afford. This turns the trace list into a quality monitor: filter to low scores in the last hour and read exactly what failed. In Overmind this is [live scoring](/product/observability), and evaluators come in six kinds, from deterministic rules (schema validity, tool selection, latency, cost) through model judges to trajectory and dataset-level scorers. Deterministic checks are free and instant, so send every mechanical question there and reserve the judge for the genuinely subjective ones.

Two details make live scores trustworthy rather than decorative. Outcomes are typed: every result is `scored`, `abstained`, `not_applicable`, `skipped` or `error`, and only `scored` feeds averages, so an abstention never becomes a fake number. And runs that cannot be faithfully reconstructed are marked degraded and excluded rather than quietly polluting the aggregate.

**2. Turn failures into a dataset instead of a Slack thread.** Select the traces that went wrong and promote them into a dataset, with each row keeping a link back to its source trace. That provenance matters more than it sounds: six weeks later, when a metric moves, the difference between "we think this row came from a real incident" and "here is the exact run" is the difference between a decision and an argument. [How to turn traces into a training dataset](/research/traces-to-training-dataset) walks through the selection step.

**3. Pin the dataset as a regression set.** Once failures are a dataset at a recorded commit, a prompt change can be run against them before it ships rather than after a customer finds it. The failure you already had is the cheapest test case you will ever get. This is the job of [agent testing](/product/agent-testing).

**4. Then change something.** Either the prompt and tool logic, which an optimizer can rewrite and open as a reviewable diff, or the model itself, fine-tuned on the trajectories your agent actually produces and benchmarked against your current production model using your own evaluators. On our own benchmark a 0.5B specialist [trained this way](/product/model-training) scored 50% against GPT-4o mini's 29% on the same product evaluation.

None of steps 1 to 3 requires you to leave your current observability tool. Steps 1 and 2 are worth doing even if you never get to step 4.

## Do you need OpenTelemetry for LLM observability?

You do not strictly need it, and you should use it anyway.

OpenTelemetry is the only thing in this category that is vendor-neutral. Instrument with OTel and your traces are portable; instrument with a proprietary SDK and your history is hostage to a pricing page. The GenAI semantic conventions are still stabilising, which is a real cost, but the alternative is worse.

The practical wrinkle is dialects. There are several competing attribute vocabularies in circulation: the OTel GenAI semantic conventions, OpenLLMetry/Traceloop keys, OpenInference, and each vendor's own. They disagree about something as basic as what to call an input token count. Any backend worth using should read more than one. Overmind's ingest [reads three dialects](https://docs.overmindlab.ai/core/observability) in priority order, its own SDK keys, OTel GenAI, and OpenLLMetry/Traceloop, and normalizes Traceloop scope names on the way in. OpenInference attributes are not parsed today, which is the kind of thing worth checking before you commit rather than after.

Getting spans in should not require infrastructure. Overmind takes OTLP/HTTP protobuf posted straight to the API with an API key header, with no collector, sidecar or proxy in between, so a stock OTel exporter works with nothing but a base URL:

```bash
export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.overmindlab.ai"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_HEADERS="x-api-key=ovr_..."
```

Or, in Python, with auto-instrumentation for the OpenAI, Anthropic, Gemini and Agno clients:

```python
import overmind

overmind.init(
    service_name="research-agent",
    environment="production",
    agent_name="Research Agent",
)
```

Trace ingest is deliberately exempt from credit gating, because telemetry you stop sending when you run out of budget is telemetry you cannot trust.

## How do you choose an LLM observability tool?

The ranked lists of tools are easy to find and not very useful, because every product in the category passes the feature-checklist test. These are the questions that actually separate them, drawn from what production teams say they wish they had asked.

- **What do you show me when a run dies silently instead of erroring?** The single best question in the category. Most answers are unconvincing.
- **Is it OpenTelemetry, genuinely?** Can I point a stock OTel exporter at it and get sensible spans, or does it need your SDK? Which attribute dialects do you parse?
- **Can I see cost next to the behavior that caused it?** Per-run spend tied to the specific tool call that looped, not a monthly total.
- **Can a production failure become a regression case without copy-paste?** If turning a bad trace into a test requires a human to retype it, it will not happen twice.
- **Does anything score traces automatically, or do I only find failures by browsing?** Browsing does not scale past the first thousand runs.
- **Where does trace data live, what is the subprocessor list, and what happens to a trace containing user data when an erasure request arrives?** Enterprise buyers ask this first and it is usually answered with a link to a docs page rather than an answer.
- **What happens when my provider changes model weights behind a stable model name?** If the answer is "you will see it in the dashboard", ask which number moves.

Ask us the same seven. On the sixth, the honest answer is that credentials and connector secrets are encrypted at rest and the rest is a conversation, not a certification: we do not claim compliance we do not hold. Start with the [observability product page](/product/observability) or the [docs](https://docs.overmindlab.ai/core/observability), then [open a free project](https://console.overmindlab.ai) and send it a trace.

## Common questions

### What is the difference between LLM observability and monitoring?

Monitoring answers whether the system is up, fast and inside budget, using aggregates and alerts. Observability answers why one specific run behaved the way it did, using the trace of that run. Monitoring tells you something is wrong; observability helps you find out what. Neither tells you whether the output was correct, which is the job of evaluation.

### Is LLM observability just APM for AI?

No, though it borrows the machinery. APM assumes deterministic systems where failure means an error or a latency spike. LLM applications fail most often while returning a 200 with a well-formed, confident, wrong answer. The extra work is capturing prompts, completions, token cost and multi-step trajectories, and then grading output that has no reference answer.

### What are the three pillars of LLM observability?

The classic three, metrics, logs and traces, still apply, with traces doing most of the work because the unit of interest is a multi-step run rather than a single request. In practice teams add two more: token and cost accounting per run, and output quality scores. Without the last one you have complete records of behavior you cannot judge.

### What is the difference between LLM observability and agent observability?

LLM observability treats the model call as the unit of work. Agent observability treats the trajectory as the unit: the whole path from goal to outcome, including tool calls, retries, sub-agent handoffs and memory reads. Tracing an individual call is largely solved. Reconstructing why an agent chose a path forty steps ago is not.

### Do I need OpenTelemetry for LLM observability?

Not strictly, but it is the most widely adopted vendor-neutral standard and it keeps your trace history portable. The caveat is that several attribute dialects are in circulation and they disagree on basic keys like token counts, so check which ones your backend parses before you commit. A backend that accepts OTLP directly, with no collector to run, removes most of the setup cost.

### How much trace data should I keep, and does sampling break it?

Keep whole traces rather than sampled spans: half a trajectory tells you almost nothing about a run that went wrong. If cost forces sampling, sample at the trace level and keep 100% of runs that error, cost more than a threshold, or score badly. Uniform random sampling is the most likely to drop the rare, expensive failures you needed to see, so bias the sample towards them.

### Can LLM observability tell me whether my agent is any good?

No, and this is the most common misunderstanding about the category. A trace records what happened with perfect fidelity, including when what happened was a confident lie. Judging quality requires a separate layer: deterministic checks for anything mechanical, model judges calibrated against human labels for anything subjective, and a definition of good that lives in your product rather than your telemetry.

### What do I do with the traces once I have them?

Score them as they arrive so failures surface without browsing, promote the bad ones into a dataset that keeps provenance back to the source trace, pin that dataset as a regression set so the next prompt change is tested before it ships, and then change something: the prompt and tool logic, or the model itself, trained on the trajectories your agent actually produces and benchmarked against what you run today.

## Related reading

- [So, you have observability. Now what?](https://www.overmindlab.ai/research/so-you-have-observability-now-what)
- [How do you turn traces into a training dataset?](https://www.overmindlab.ai/research/traces-to-training-dataset)
- [The anatomy of an AI agent](https://www.overmindlab.ai/research/anatomy-of-an-ai-agent)
- [Observability reference (docs)](https://docs.overmindlab.ai/core/observability)
- [Eval reference (docs)](https://docs.overmindlab.ai/agent-testing/eval)

- [All glossary terms](https://www.overmindlab.ai/glossary)

---

# https://www.overmindlab.ai/research/my-summer-at-overmind

# My summer at Overmind

> Canonical: https://www.overmindlab.ai/research/my-summer-at-overmind

- Author: Shashank
- Category: News
- Tags: Reflections, Agents
- Published: 2026-09-28
- Updated: 2026-09-28

> A summer internship reflection on autonomy, hard problems, and building across the Overmind product.

![My summer at Overmind](https://cdn.sanity.io/images/gv4t9qa3/production/45edf2bbcdc3a20c842dc68f5d4fed228e7d8cd8-2048x2048.png)

Who knew spilling Thai food down my future boss’s pants would give me a summer internship? Well, that’s what happened to me. I was at a dinner in San Jose during my trip to NVIDIA GTC, where I met Sam, my future boss. Upon exchanging ideas about the ever-evolving landscape of agents and LLMs, we seemed to agree upon one thing: agents are nowhere near perfect and have so much room for improvement, whether in cost, reliability, speed, or efficiency.

My internship at Overmind has been an unconventional one. In a good way.

It genuinely feels like I just joined last week - time really flies when you are having fun - but it simultaneously feels like I joined a year ago. So much has happened since joining, which is truly the magic of working in a startup.

Most internships are structured in a fairly common way: interns get assigned to a mentor or supervisor, they get assigned a project, and they work that project to completion throughout their time at a company. Things were different here. From the week I joined, I was able to contribute to the core product. It felt extremely empowering, but I also remember asking the Founding Engineers an awful lot of questions. Sorry.

## What was the culture like?

From the get-go, I was given a lot of autonomy and was involved in core discussions around the product. This also meant that there were a lot of hard problems to solve. I genuinely liked the fact that Overmind encourages employees to come to the office. There’s just truly something about collectively staring at a whiteboard in the office and brainstorming together. We were able to tackle the issues and bounce ideas off each other, and it felt like we had each other’s backs.

## What did i work on?

I pretty much worked on all product surfaces. From evals and agentic behavioral analysis to model fine-tuning and harness optimization, the whole pipeline is connected. I was given the space and bandwidth to experiment and come up with my own solutions to problems. Sometimes, I’d be working on something from the ground up, and sometimes I would be collaborating with someone else. Sometimes I was doing hardcore engineering, and sometimes I was doing UI/UX reviews or running marketing case study experiments. Given that Overmind is relatively early-stage, it really felt like I was deeply embedded in the roots of product development from scratch, which felt eye-opening, since I was exposed to so many facets of building a real customer-facing product.

## What was so eye-opening about it?

As a Computer Science student, we are often only exposed to the technical side of things, like engineering-related concepts. However, there truly is so much more to building something that solves a gaping void in the market. The whole element of whether people would actually use it, how to market your product, how to get people hooked onto your product, and how to grow your brand and image in a vast and competitive agentic space - all these questions do not have easy answers, and they are probably not something you could conventionally comprehend in a lecture hall. In this regard, I am truly grateful Overmind has opened my eyes.

## Want to work at Overmind?

The most important requirement: you need to like cookies and Filipino food. Don’t ask me why.

Just kidding.

Overmind really looks for people with a drive to solve the hardest problems in the agentic space and truly understand the gaps that need to be filled from the messy noise out there. Everyone’s building in the AI space now, but agents are still riddled with endless problems. You need to be someone who is willing to dive deep into problems, and most importantly, adaptable enough to venture into uncharted territory.

## To end off

It was really fun exploring so many different areas of engineering, like data science, classical ML, software engineering, model fine-tuning, and agentic engineering. I managed to learn so much in a relatively short span. These were things you simply can’t learn in a lecture or classroom. As such, things will need to be self-taught, and a growth mindset in a startup environment like Overmind would prove vital.

That said, there still genuinely are so many problems in this space, from evals to pre-training pipelines to training and post-training backtesting. The whole model development cycle needs to be streamlined, and that’s truly what Overmind is all about.

## Related research

- [Small is the new frontier](https://www.overmindlab.ai/research/small-is-the-new-frontier): Why specialized models trained on your own production data are becoming the practical path to owned intelligence.
- [When bigger isn’t better](https://www.overmindlab.ai/research/when-bigger-isnt-better): On specialized tasks, Overmind-trained small language models outperform frontier models on accuracy, hallucination, and cost.
- [How to train your agent](https://www.overmindlab.ai/research/how-to-train-your-agent): Learnings from building, deploying and improving agents in production.

---

# https://www.overmindlab.ai/research/small-is-the-new-frontier

# Small is the new frontier

> Canonical: https://www.overmindlab.ai/research/small-is-the-new-frontier

- Author: Tyler Edwards
- Category: Research
- Tags: Models, Data, Agents
- Published: 2026-09-28
- Updated: 2026-09-28

> Why specialized models trained on your own production data are becoming the practical path to owned intelligence.

![Small is the new frontier](https://cdn.sanity.io/images/gv4t9qa3/production/5c76a3744fa44ae39de022e9b4d5ff17ad5725b6-2048x2048.png)

LLMs are designed to be Jills of all trades.

Ask ChatGPT or Claude how to make tortellini, come up with a plan for buying a house in the next three years, or explain what Socrates said about an unexamined life, and it can distinguish what you want, how to get it, and serve it up to you shockingly well.

But for most business-critical tasks, your model doesn’t need to be a Swiss Army knife.

It just needs to do the thing you want it to do - extracting invoices, underwriting a loan, running a GTM motion end to end - exactly the way you want it to, every single time. Especially if that workflow is responsible for millions in revenue.

To pull that off with an LLM, you need a bench of AI engineers and AIOps folks to write detailed instructions, wire up and troubleshoot integrations across observability tools and model providers, curate and clean training data, stand up eval and deployment pipelines, run live tracing on production, aggregate results, and layer in additional context as your workflows evolve.

These are new skills. Skills that most people, outside of academia, don’t have. Which is why we’ve seen so many labs pop up. PhDs who bestow the capability of training and fine-tuning models on you from on high, with their research, their FDEs, their pricing.

But do you really own your intelligence if you’re putting it through someone else’s lab?

## You don’t need a lab anymore

Despite what they’d have you think, what’s happening inside OpenAI and Anthropic isn’t magic. Neither is what happens inside the labs that fine-tune frontier models for you.

The science behind agent training and fine-tuning is well-understood. It can be commoditized, given to people to do themselves, and not in a piecemeal type of way.

Overmind is lowering the barrier to entry, making it faster, cheaper, and technically trivial to capture production data, evaluate your agents, fine-tune open-weight models on your own traces, and deploy them.

Point Overmind at your codebase and it scans every line to build context, learning what your company does, what your product does, and how your code works. With that context, it dynamically adjusts every part of the training and fine-tuning process to fit your product, not the other way around.

With a harness like that, you get:

## Better economics

The teams we talk to are trying to improve their margins while also trying to improve their product. On frontier APIs, those feel like competing priorities. On Overmind, they’re not.

A model fine-tuned on your production data can outperform frontier generalist models at a fraction of the inference cost. For example, one of our fintech customers reduced their invoice-extraction costs by 96% using an Overmind-trained model.

In our most recent benchmark report, Overmind-trained specialist models beat frontier models tested on contract clause detection accuracy at roughly 5% of the cost, fabricating quotes 20 to 30 times less often.

## True differentiation

So you built your product with Claude? So did a thousand other companies. What makes yours unique? Competitive?

Differentiation comes down to data. You’ve got data that no one else has. Creating your own bespoke model from it, tailoring it to your customers, in their specific vertical, will be your moat in ever more crowded spaces.

## Full ownership

Frontier models are frontier for a reason: they’re super powerful. But they’re also not yours, which means the version you’re building on right now isn’t going to be the same version you’re building on in six months, for better or for worse.

The model you train and fine-tune with Overmind is yours to keep and modify. Retrain it, retune it, roll it back to a version you liked better, teach it something new tomorrow. Host it with us or download the model and run it wherever you need to.

The longer you use it, the more valuable it gets.

## Fast time to value

Most teams can’t afford to spend months getting a proprietary model production-ready. In Overmind, training and fine-tuning take hours, and deployment takes a few clicks.

Whether you’re working in the Overmind console or working via a coding agent connected to Overmind’s MCP, you don’t have to manually port what you’ve built into your codebase.

Overmind bundles that work into a pull request against your repo. Your team reviews and merges, then boom, it’s live in prod.

## Fewer mistakes caught faster

Specialized agents have scoped context and scoped objective. They’re doing this, and only this. Not that. Less room for creative interpretation of your rules means fewer expensive mistakes.

Overmind takes that a step further with extremely nuanced evals. With access to your entire codebase, Overmind knows not only when an agent performed well or poorly, but why, and in relation to the rest of your product.

Overmind learns from your agents’ patterns. So when one starts deviating from the norm, Overmind flags it, and you have the power to decide: is that behavior actually a new capability worth training on? Or something you want the agent to stop doing? Your agent gets better either way.

## Data processing without a data team

Training platforms tout themselves as BYOD, bring your own data, but there’s a catch: that data has to be in a specific format. And data pre-processing is a huge lift.

That’s not necessarily laziness on their part. After all, what a fintech wants to do with its transaction logs has almost nothing in common with what a cyber team wants to do with their alerts.

Building a platform that can generalize across all those use cases is near-impossible, but Overmind gets close.

Hand over a messy pile of data in JSON, CSV, even a whole folder of images, and Overmind cleans, resamples, and shapes it into a dataset you can feel good about training your SLM on. No separate vendor for pre-processing or manual triage.

## Low maintenance

The whole reason AI engineers have jobs is that training models is not a one-and-done deal. You have to rewrite evals, retune prompts, and rebuild retrieval infrastructure as your product changes.

If you have hundreds of agents going at once, it’s easy for a couple of harnesses to fall out of sync. A few weeks later, you find out that they’ve regressed, and by then your internal or external customers have already been on the receiving end. Plus, you’ve lost weeks of production data you could’ve been learning from.

With Overmind, you’re not only getting a product we keep making better and better, you’re also getting a product that learns directly from you: your data, your codebase. Overmind’s Agent Optimizer re-scans your codebase over time, updating its context graph, working out where performance is breaking down, and adapting evals and other improvements automatically, so it stays aligned with whatever you’re building without you having to do any of the work.

## A fully connected ecosystem

There are incredible inference vendors, trace providers, observability platforms, and fine-tuning APIs out there. You may already use a few of them. But each one has its own learning curve, and stitching them together takes time away from the most important thing: creating bespoke models.

Overmind abstracts away the stitching, pulling traces from any observability tool, running training through any fine-tuning API, and pushing the tuned model to any inference provider - giving you the value of the ecosystem without the tax of becoming an expert.

We see our job as demystifying this work and putting it in the hands of any team. So while our hosted version will be the fastest, most optimized way to run Overmind, it’s not the only way. Our harness is open source.

## Thinking fast and small

We’re not the only ones thinking this way. As of today, there are over 3 million open source models on Hugging Face. Every day, another org announces they’ve built their own models - Thomson Reuters, Harvey, Optimizely, to name a few.

Frontier models will always have their place for tasks that require ample firepower. But the appeal of moving away from LLMs is clearly there.

What’s missing is a way to make owned models more accessible to teams that don’t have endless budget and bandwidth. The truth is, you don’t need to know how electricity works to be a great electrician, and you shouldn’t need a PhD to ship a specialized model.

Plug in your agent, get a trained model - at a fraction of the price.

Leave the lab behind.

Explore the [open-source Overmind repo](https://github.com/overmind-core/overmind).

## Related research

- [My summer at Overmind](https://www.overmindlab.ai/research/my-summer-at-overmind): A summer internship reflection on autonomy, hard problems, and building across the Overmind product.
- [When bigger isn’t better](https://www.overmindlab.ai/research/when-bigger-isnt-better): On specialized tasks, Overmind-trained small language models outperform frontier models on accuracy, hallucination, and cost.
- [How to train your agent](https://www.overmindlab.ai/research/how-to-train-your-agent): Learnings from building, deploying and improving agents in production.

---

# https://www.overmindlab.ai/research/when-bigger-isnt-better

# When bigger isn’t better

> Canonical: https://www.overmindlab.ai/research/when-bigger-isnt-better

- Category: Research
- Published: 2026-09-21
- Updated: 2026-09-28

> On specialized tasks, Overmind-trained small language models outperform frontier models on accuracy, hallucination, and cost.

![When bigger isn’t better](https://cdn.sanity.io/images/gv4t9qa3/production/ab60e5defa9282c8de7ceaf70c202d648459ca09-2048x2048.png)

The OpenAIs and Anthropics of the world are betting that the more powerful a model is, the more comprehensive, the better it’ll be at solving every kind of problem. Our data’s saying otherwise.

We ran three head-to-head benchmarks against frontier models—one in legal (contract clause detection), one in biomedical research (identifying scientific relationships in published papers), and one in aviation safety (turning NASA incident reports into analyst-like synopses).

We picked these fields because the outcomes of these workflows can’t just be “good enough,” they have to be right. And while not every AI-driven workflow has to be near-perfect, you certainly don’t want to gamble on business or customer-critical tasks.

> We are firm believers in task-specific models. Frankly, that means we are moving away from the very large models to smaller models that are good at one thing, and one thing only, and do it really, really well.
>
> — Head of Innovation, Fortune 50 bank

## Trained SLMs get the right answer more often

Frontier models can do a little bit of everything, which is great when you’re planning a trip to Mexico City or trying to get more zinc in your diet. It’s less great for when you need to extract the right answer from thousands of different documents.

Our specialist models (trained on a highly specific task and real data) don’t have that problem.

### 7X better at quoting a contract clause word for word

We kicked off our analysis by asking 4,000+ questions across 100+ contracts. For each one, the model had to say:

1. Whether a specific clause was in the document
2. Quote the exact wording of that clause

Something a real lawyer (ok, maybe a first-year associate) would do: skim a contract, find a clause, quote it verbatim so that they can brief their boss on risk, cite it in a redline, or flag it for a client.

The Overmind-trained model made roughly half as many mistakes as the frontier flagship model (91.6% versus 82.0%). It was also 7x more accurate at quoting clauses verbatim (59.6% versus 8.5%).

**Half as many wrong answers, 7x more exact quotes than the frontier flagship**

Contract clause detection, 4,000+ questions across 100+ contracts · *Higher is better*

|  | Overmind (our fine-tuned specialist) | Base model (what we trained) | Frontier flagship | Frontier mid-tier |
| --- | --- | --- | --- | --- |
| Accuracy (F1) (aka correct answer) | 91.6% | 80.4% | 82.0% | 80.5% |
| Exact quote match | 59.6% | 4.2% | 8.5% | 10.4% |

### 4X more accurate at reading a research paper

Same idea, different domain. This time, we pointed two models (Overmind-tuned Qwen 9B model and a frontier flagship model) at BioRED, a dataset of biomedical research papers human researchers have already annotated.

We hid those annotations from the models, of course, and then asked questions like, “Is there a relationship between chemical A and gene B in this paper?” across 10,000 different entity pairs. The models then had to:

1. Answer yes or no.
2. Return the answer in JSON format, so that, theoretically, that answer could be used in a downstream pipeline.

Again, this is common drudge work in biomedical research when there are no answers yet: pulling results from published papers so a researcher can feed them into a meta-analysis or design a completely new experiment.

The Overmind-trained model got the right answer 4x more often than the frontier flagship (54.9% versus 14.1%). It also returned answers in the required format every single time.

**Right answer 4x more often, and valid JSON every single time**

Biomedical relationship extraction, BioRED, 10,000 entity pairs · *Higher is better*

|  | Overmind SFT | Frontier flagship |
| --- | --- | --- |
| Accuracy (F1) (aka correct answer) | 54.9% | 14.1% |
| Valid JSON outputs | 100% | 99.3% |

0.7% doesn’t sound terrible, but that’s still 71 findings a researcher has to redo by hand (or that get dropped from an analysis entirely).

### 10% Closer to how a NASA analyst would’ve summarized a safety report

For our third experiment, we ventured into an even more mission-critical domain: aviation safety. NASA’s Aviation Safety Reporting System (ASRS) collects confidential incident reports from pilots, controllers, mechanics, and dispatchers.

We first trained our models on 1,000 such reports. Then, we gave our models and a frontier model 500 fresh reports with an identical prompt. For each report, the models were asked to:

1. Read the whole thing (reporter role, aircraft, flight phase, and the pilot’s account of what happened).
2. Write a single line to go in NASA’s searchable safety database, in the style an analyst would use.

Safety analysts write these synopses so that an investigator working a similar case can find it. Get the synopsis wrong, and the record might as well not be there. To see how well the models did, we compared them to the actual version an analyst wrote across two axes: meaning and wording.

Our 12B model matched the frontier model on meaning—0.657 versus 0.655, statistically a tie—and beat it on wording (0.295 versus 0.268), meaning its synopses used more of the same terms an analyst would’ve picked.

**12B model tied the frontier flagship on meaning and beat it on wording**

Aviation safety synopses, NASA ASRS, 500 reports · *Higher is better*

|  | Overmind SFT (12B) | Overmind SFT (smallest we trained) | Frontier flagship |
| --- | --- | --- | --- |
| Same meaning as the analyst (scored 0 to 1) | 0.657 | 0.641 | 0.655 |
| Same wording as the analyst (scored 0 to 1) | 0.295 | 0.275 | 0.268 |

Even the smallest model we trained nearly matched the frontier model on meaning and performed better on wording at, as we’ll see, a considerable difference in cost.

## Trained SLMs make up the wrong answer less often

Hallucinations are a liability in any workflow, but especially in ones where the output is used for something very official—a legal filing, a published abstract. In our analysis, frontier models made things up orders of magnitude more often than our Overmind-trained specialist models.

### 20-30X fewer phantom clauses in legal contracts

If you remember, we asked models to identify a specific clause in a contract and quote its exact wording. Sometimes the clause was in the document, sometimes it wasn’t. Frontier models had particular trouble with this, hallucinating in two ways:

- “False alarms,” i.e., claiming a clause was there when it wasn’t.
- Fabricating entirely new quotes that did not exist.

**Frontier flagship fabricated quotes 28x more often and raised false alarms 15x more often**

Phantom clauses in contract clause detection · *Lower is better*

|  | Overmind SFT | Base model | Frontier flagship | Frontier mid-tier |
| --- | --- | --- | --- | --- |
| False alarm rate | 0.65% | 2.46% | 10.01% | 11.72% |
| Fabrication rate | 0.12% | 3.36% | 3.36% | 2.81% |

Per question, the frontier flagship model we tested fabricated quotes from contracts 28x more often than Overmind, and raised false alarms 15x more often.

### 7X fewer invented findings in biomedical research

In our BioRED test, we counted “hallucinating” as claiming a relationship exists when it doesn’t. Not quite the same as accuracy, since the model can also get an answer wrong by saying, ‘No, there is no relationship’ when, in fact, there is one.

Turns out, frontier models said there was a relationship when there wasn’t a lot. More than seven times more often than the SLM we fine-tuned with Overmind.

**Frontier flagship claimed a relationship that wasn’t there more than 7x as often**

Invented findings in biomedical extraction, BioRED · *Lower is better*

|  | Overmind SFT | Frontier flagship |
| --- | --- | --- |
| Hallucination rate | 6.6% | 48.2% |

That means roughly half of every “yes” the frontier model gave was a relationship that didn’t appear anywhere in the BioRED dataset. Thankfully, this was just a test.

### 2.5X fewer invented details in aviation safety records

Safety synopses are dense with specifics, all kinds of numbers, identifiers, aircraft, and flight details. So in our ASRS test, we counted how often each model added a specific detail to its synopsis that did not appear anywhere in a report it was given.

The frontier model did this in more than 1 out of every 3 synopses.

**Smallest trained model invented a detail 60% less often than the frontier flagship**

Invented details in aviation safety synopses, NASA ASRS · *Lower is better*

|  | Overmind SFT (12B) | Overmind SFT (smallest we trained) | Frontier flagship |
| --- | --- | --- | --- |
| Hallucination rate | 19.4% | 13.4% | 33.4% |

The smallest model we trained hallucinated a detail—which could make or break a case—60% less often than the frontier model.

## Trained SLMs cost substantially less

Frontier APIs are cheap to rent at first, especially with all the free credits. But as usage grows (a given as you train, use, test, and retrain it over time), your bill grows with it.

Before long, a chunk of your AI budget is paying frontier prices for work a smaller, task-specific model could handle just as well—and in many cases, better.

### 20X cheaper legal contract review

We broke down how much it costs for each model to answer 1,000 questions.

While the frontier mid-tier model wasn’t so egregious (though double Overmind’s cost), the flagship model was an eye-watering 20.3 times more expensive.

And if you recall, we asked 4,000+ questions. At scale, these models really add up.

**Frontier flagship cost 20.3x more per 1,000 questions, the mid-tier double**

Cost to answer 1,000 contract questions · *Lower is better*

|  | Overmind SFT | Frontier mid-tier | Frontier flagship |
| --- | --- | --- | --- |
| Cost per 1,000 questions | $1.03 | $2.15 | $20.94 |
| Relative to Overmind |  | 2.1x more | 20.3x more |

### 35% Cheaper biomedical extraction

Overmind processed all questions (over 10,000) for $17.49. On the frontier flagship model, it costs $26.73. That’s a smaller gap than the legal test, but when you’re running a real research pipeline, cutting your bill by a third is significant.

**A third off the bill for the same 10,000+ questions**

Cost to process the BioRED test, 10,000+ questions · *Lower is better*

|  | Overmind SFT | Frontier flagship |
| --- | --- | --- |
| Total cost | $17.49 | $26.73 |

### 14 To 23% fewer tokens per aviation synopsis

On the aviation test, the frontier model averaged 42.3 words per synopsis. The models we trained averaged between 14 and 23% fewer (32.5 to 39.8 words).

**Trained models wrote 14 to 23% shorter synopses than the frontier flagship**

Aviation synopsis length, NASA ASRS, 500 reports · *Lower is better*

|  | Overmind SFT (smallest to 12B) | Frontier flagship |
| --- | --- | --- |
| Average words per synopsis | 32.5 to 39.8 | 42.3 |

Regardless of how much you pay per token, using fewer of them gets you money back. Plus, you’re using an open-weights model (aka yours to continue fine-tuning) and running it on hardware you already control.

## Do you really own your intelligence?

Not if you’re renting it from someone else. And if specialist models offer more control, better performance on specific tasks, and cost less, what’s stopping you?

Often it’s the belief that owning a model is a huge lift. With Overmind, it doesn’t have to be. We’re making it faster, cheaper—technically trivial—to capture production data, fine-tune open-weight models on your own traces, automatically evaluate and improve your agents, and deploy them.

## Related research

- [My summer at Overmind](https://www.overmindlab.ai/research/my-summer-at-overmind): A summer internship reflection on autonomy, hard problems, and building across the Overmind product.
- [Small is the new frontier](https://www.overmindlab.ai/research/small-is-the-new-frontier): Why specialized models trained on your own production data are becoming the practical path to owned intelligence.
- [How to train your agent](https://www.overmindlab.ai/research/how-to-train-your-agent): Learnings from building, deploying and improving agents in production.

---

# https://www.overmindlab.ai/research/how-to-train-your-agent

# How to train your agent

> Canonical: https://www.overmindlab.ai/research/how-to-train-your-agent

- Author: Pritam Soni
- Category: Tutorials
- Tags: Agents
- Published: 2026-07-30
- Updated: 2026-08-07

> Learnings from building, deploying and improving agents in production.

![How to train your agent](https://cdn.sanity.io/images/gv4t9qa3/production/604778fd9d419749ba37ec88d681e4a3c0d5bc5e-2048x2048.png)

If you're running an LLM-powered agent in production, the seven patterns below show up again and again.

## Instrument every agent trace

Every agent run generates a trace: the full sequence of inputs, tool calls, retrieved context, intermediate reasoning, and final output. Without that record, you're reconstructing agent behavior from scattered log files, which is miserable work.

OpenTelemetry has become the default standard for this. Its GenAI semantic conventions set one shared format for what each step of a run records, including model name, token usage, and tool calls. A trace from a LangChain agent then looks the same as one from a raw API call. Most observability platforms now speak this format natively, including Langfuse, Arize Phoenix, Datadog LLM Observability, and LangSmith.

**Tactical advice:** pick one OpenTelemetry-compatible tracing tool. Route every agent run into it. Keep the traces for at least 30 days. Tag them with user ID, use case, and agent version, so you can filter them later without going back to change your tracing code.

## Build evals from real traffic

Before deploying an agent, run it against some evals. An eval is a scored check of whether the agent did the job. However, when real traffic arrives, so will edge cases.

Production traces can be a real eval source. Sample traces weekly and turn the failures into new test cases. Older eval sets can become a regression check, to confirm old fixes still hold, rather than your source of truth. Langfuse, Braintrust, Arize, and LangSmith all support this workflow directly, turning sampled production data into structured eval sets.

**Tactical advice:** by week two in production, your primary eval set should be built from real traces, not synthetic ones written before launch.

## Turn traces into a labeled dataset

A trace only records what happened. Before you can train or evaluate on it, you need to know whether the run was any good. That judgment is the label, and labels are what a dataset is actually made of. More unlabelled traces will not make it better, so label what you have before you collect more.

Start with your agent's codebase. It already spells out what the agent is meant to do, so turn that into a checklist of pass or fail criteria. Score traces against the checklist and write a note on each one that fails. Hamel Husain calls this error analysis. Group those notes into a failure taxonomy, a fixed list of failure types you can count and track over time.

Once every trace carries a label, and a failure type where it failed, you have a dataset. That is what you fine-tune on, and what you score every later change against.

**Tactical advice:** draft 5 to 20 pass/fail criteria straight from the code, then refine them as you label. Spend a few hours a week labeling yourself for the first month. Don't outsource the first pass. Your judgment about what "good" looks like only shows up once you've sat with the traces yourself.

## **Prompt engineering has a ceiling**

Prompt iteration is cheap and fast, so use it for as long as it keeps improving your eval score. Eventually each new instruction fixes yesterday's bug and quietly introduces a new one. The system prompt bloats, latency creeps up, and the model starts ignoring parts of it altogether.

At that point you're choosing between more tweaking or a different lever entirely. Most teams keep tweaking a month longer than they should, because prompt edits feel free. They aren't. Engineering time is the most expensive line item in the whole pipeline.

**Tactical advice:** track your eval score before and after every prompt change. First run the eval twice on an unchanged prompt to see how much the score wobbles on its own. Ignore any change smaller than that.

## Fine-tuning doesn't need a research lab

Fine-tuning an open-weights model used to mean a research team and a large GPU budget. LoRA freezes the base model and trains a small set of low-rank adapter matrices instead. Those adapters are often under 1% of total parameters. That is why a 70B model can be fine-tuned on a single high-memory GPU, using QLoRA's 4-bit quantisation. Quantisation stores the weights at lower precision, so the model takes up less memory.

Pick an open-weights base, run a LoRA pass over labeled trajectories, meaning recorded agent runs you have already scored. Then serve the result on Fireworks, Together, or Modal. Unsloth, OpenPipe, and Predibase handle most of the pipeline for you.

**Tactical advice:** if you have 1,000+ labeled trajectories and a narrow task, run a LoRA fine-tune. Compare it against your prompt-engineered baseline on the same eval set.

## Freeze the eval set before you train

The eval set you score a fine-tune against has to be the same one you scored the baseline against, and it has to be locked before training starts. Change the criteria and the training run in the same week and you cannot tell which one moved the number.

Test the model inside the agent harness, not on its own. A model that scores better in isolation can still make the agent worse, because the agent depends on tool call formats, output structure, and instruction following that a benchmark score does not capture.

**Tactical advice: **every model change runs against the same frozen eval set, every time. Models regress for all sorts of reasons, and a fixed scoreboard is the only way you find out.

## The self-improvement loop

The first full pass through capture, label, train, deploy, and measure takes weeks. By the fifth pass, most of the friction is gone and the loop runs in hours.

That compounding is the whole point. A team that runs this loop five times in a quarter ends up with an agent that matches their business needs at a fraction of frontier API cost. A team that runs it once, ships, and calls it done is roughly where it started six months ago. This is essentially Overmind's product thesis. It connects your repository and traces, then proposes changes through its optimizer. It tests those changes against your labeled data and only surfaces the ones that actually move your score. The loop from evidence to shipped improvement stays short, instead of becoming its own six-week project.

**Tactical advice:** pick a cadence and put it on the calendar. Weekly is aggressive but doable. Monthly is the floor. Anything slower and you're not really running a loop, you're running occasional cleanup.

## Infrastructure is more work than the agent

Any one of these seven steps is a real job on its own. Strung together, they're several jobs. Most teams find the surrounding infrastructure costs more engineering time than the agent.

Overmind closes that gap. It reads your agent's codebase to understand what it does. Then it handles trace capture, labeling, fine-tuning, deployment, and measurement as one connected system, instead of five separate vendor integrations.

If you're past the prototype stage and want the loop running without building it from scratch, it's worth a look.  
  
_Overmind is the model training platform for AI teams. It turns your production traces into specialized models you own. [Get started](https://console.overmindlab.ai)._

## Related research

- [My summer at Overmind](https://www.overmindlab.ai/research/my-summer-at-overmind): A summer internship reflection on autonomy, hard problems, and building across the Overmind product.
- [Small is the new frontier](https://www.overmindlab.ai/research/small-is-the-new-frontier): Why specialized models trained on your own production data are becoming the practical path to owned intelligence.
- [When bigger isn’t better](https://www.overmindlab.ai/research/when-bigger-isnt-better): On specialized tasks, Overmind-trained small language models outperform frontier models on accuracy, hallucination, and cost.

---

# https://www.overmindlab.ai/research/who-trains-their-own-models

# Unwrapping the wrappers

> Canonical: https://www.overmindlab.ai/research/who-trains-their-own-models

- Author: Sam Brunt
- Category: Research
- Tags: Models
- Published: 2026-07-30
- Updated: 2026-08-07

> Application layer companies are training their own models. Why?

![Unwrapping the wrappers](https://cdn.sanity.io/images/gv4t9qa3/production/52436243946c705c3e53d056b89da88ec7764636-2048x2048.png)

Application layer companies, often labeled as 'ChatGPT wrappers', are starting to train their own models. Specialist models, post-trained or fine-tuned on a dataset their competitors cannot touch: their own production traffic.

Four recent examples:

- **Cursor** trained Composer, the model inside its coding agent, with online RL on production agent traces. Delivering frontier-level coding at a fraction of the latency and cost.
- **Intercom** trained Fin Apex, a customer-service model post-trained on billions of support interactions. It bought them control of the resolution-rate metric at scale.
- **Replit** trained a 7B code-repair model, a fine-tune of DeepSeek-Coder-Instruct-v1.5. It bought them a fast, narrow specialist trained on IDE data nobody else has.
- **Perplexity** trained Sonar, its search-answer model, a fine-tune of Llama 3.3 70B served on Cerebras hardware. It bought them sub-second, citation-heavy answers at search scale.

## Cursor: reinforcement learning on live agent traces

Cursor didn't fine-tune once and ship. The Composer family of models run on real-time RL: model checkpoints go to production, real user interactions become the reward signal, and an improved checkpoint ships several times a day. The model is a mixture-of-experts architecture trained inside a harness built to mirror Cursor's actual tool space, including file edits, semantic search, and terminal commands. The reward signal reflects how developers really use the product rather than a synthetic benchmark, according to [Cursor's technical post on Composer](https://cursor.com/blog/composer).

Cursor reports Composer generates tokens several times faster than comparable frontier models while landing close to the top tier on internal coding benchmarks, a tradeoff explored in more depth by the [Composer 2 technical report](https://arxiv.org/html/2603.24477v1). Nobody else has Cursor's trace data, which means nobody else can train Cursor's model, even with an identical base checkpoint and an identical training recipe.

What it bought them: coding performance competitive with larger models, at preferential latency and cost, plus a moat provided by their own proprietary data.

## Intercom: post-training on the metric the product is sold on

Fin launched in 2023 on top of GPT-4. Fin 2 moved to Claude in late 2024. Then Intercom's roughly 60-person AI research group, led by Fergal Reid, built Fin Apex 1.0, a model post-trained on the company's own support interaction data, as described in the [Fin Apex launch post](https://www.intercom.com/blog/announcing-fin-apex-the-age-of-vertical-models-is-here/).

One large enterprise customer saw its resolution rate jump from 68% to 75% overnight after the switch to Apex, a 22% cut in unresolved conversations according to Intercom's own numbers. Independent coverage from [VentureBeat](https://venturebeat.com/technology/intercoms-new-post-trained-fin-apex-1-0-beats-gpt-5-4-and-claude-sonnet-4-6) puts Apex's benchmark resolution rate at 73.1%, against 71.1% for both GPT-5.4 and Claude Opus 4.5. Intercom's own model documentation breaks the pipeline into separate retrieval, reranking, and generation models rather than one monolithic system, detailed on the [Fin CX models page](https://fin.ai/cx-models).

Worth noting: Intercom hasn't abandoned rented models. Apex is layered on top of retrieval and routing components, and the company still uses third-party models elsewhere. This architecture is common as companies balance performance, latency and cost.

## Replit: a narrow 7b specialist trained on IDE telemetry

Replit's bet was the opposite of "bigger is better." The team fine-tuned a 7B open-weights model, DeepSeek-Coder-Instruct-v1.5, chosen after testing several code-focused base models and documented in its [code-repair write-up](https://replit.com/blog/code-repair). 

The model targets a specific use case, roughly the 20 diagnostic categories responsible for the majority of everyday coding errors, and the fine-tuned 7B checkpoint matched or beat GPT-4 and Claude 3 on real-world fixes in that use case. Small [open source models](https://huggingface.co/deepseek-ai/deepseek-coder-7b-instruct-v1.5) can beat much larger general-purpose models at many tasks.

## Perplexity: search-optimized inference at 1,200 tokens per second

Perplexity fine-tuned Meta's Llama 3.3 70B into Sonar, tuned specifically for factual grounding, citation quality, and readability in its default search mode, as the company explains in its [Sonar launch post](https://www.perplexity.ai/hub/blog/meet-new-sonar). Sonar runs on Cerebras wafer-scale inference hardware, which Perplexity and [Cerebras jointly announced](https://www.cerebras.ai/press-release/cerebras-powers-perplexity-sonar-with-industrys-fastest-ai-inference) pushes throughput to roughly 1,200 tokens per second, an order of magnitude faster than comparable models on conventional GPU inference.

In Perplexity's own A/B testing, Sonar matched or beat larger frontier models on user satisfaction inside search, while running far cheaper and faster than calling those models directly.

## Why build a specialist model instead of prompting a frontier one?

Calling a frontier model for every request is sometimes the equivalent of running a data center to toast bread. It works, but it's the wrong tool for a narrow, high-volume, well-defined job. Across these four companies, three conditions kept showing up:

- **A narrow task, not a general one.** Small specialized models are excellent at one narrow task. Don't ask them to be broadly intelligent.
- **Proprietary production data no competitor can access.** Cursor has Cursor's traces. Intercom has Intercom's support conversations. Your production traffic is the one dataset nobody can buy.
- **Enough volume that per-token latency and cost actually move the P&L.** Fine-tuning a model can be expensive. It only pays off once inference volume is high enough that shaving cost per call compounds.

None of these companies want to out-build OpenAI or Anthropic on general intelligence. Each trained a model that wins one specific task, using the one dataset it uniquely owned.

![Flywheel diagram titled "Traces into training". Production runs feed a labeled dataset, the labeled dataset feeds a fine-tuned model, and a dotted redeploy loop sends the fine-tuned model back to production runs.](https://cdn.sanity.io/images/gv4t9qa3/production/e39a777c3f8b8c4038ebc85b5a5db9ad5f78b6c4-2014x1012.png)

Capture production traces, turn them into a labeled dataset, train a model on it, and ship that model back into production. That's where the application layer is going.

_Overmind is the model training platform for AI teams. It turns your production traces into specialized models you own. [Get started](https://console.overmindlab.ai)._

## Related research

- [My summer at Overmind](https://www.overmindlab.ai/research/my-summer-at-overmind): A summer internship reflection on autonomy, hard problems, and building across the Overmind product.
- [Small is the new frontier](https://www.overmindlab.ai/research/small-is-the-new-frontier): Why specialized models trained on your own production data are becoming the practical path to owned intelligence.
- [When bigger isn’t better](https://www.overmindlab.ai/research/when-bigger-isnt-better): On specialized tasks, Overmind-trained small language models outperform frontier models on accuracy, hallucination, and cost.

---

# https://www.overmindlab.ai/research/traces-to-training-dataset

# How do you turn traces into a training dataset?

> Canonical: https://www.overmindlab.ai/research/traces-to-training-dataset

- Author: Rohit Gupta
- Category: Tutorials
- Tags: Data
- Published: 2026-07-30
- Updated: 2026-10-13

> Every agentic system can create its own training data.

![How do you turn traces into a training dataset?](https://cdn.sanity.io/images/gv4t9qa3/production/31bb0d30e15e77a87cc89b5980bc2bbae98c2a4b-2048x2048.png)

Turning agent traces into a training dataset takes four stages. Capture every run as a trace. Sample the runs worth a human’s attention. Label each sampled run against a written spec of what good looks like. Build the survivors into the format your training method needs, then rebuild weekly as production keeps producing new runs. Overmind does all four from the traces your agent already emits, then trains an open-weight model on the result.

Every agentic AI system you run is quietly writing its own training set, and most teams never use it.

When an agent runs, it emits a stream of inputs and outputs: the prompt, the tool calls, the reasoning and the final answer. That stream is a trace, and once you standardise how you capture it (the [OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/) are the default schema here) you have raw telemetry. A training dataset is narrower: a curated set of runs, scored against a spec, designed for fine-tuning jobs.

Building a dataset like this is a four-stage pipeline.

![Pipeline diagram: 1 Capture every run as a trace, 2 Sample and pick what to label, 3 Label and score against your spec, 4 Build and format for training, then train the model. A dotted feedback loop feeds each week's new runs back into capture.](https://cdn.sanity.io/images/gv4t9qa3/production/15b45f9124582d666979d5d1e2379160aa4e22fd-2416x1082.png)

*Four stages, then a weekly rebuild.*

Here's a worked example, using a refund agent, to show what each stage actually involves.

## Step 1: how do you capture an agent trace?

You capture an agent trace by instrumenting every model call and tool call, so each run lands as one OpenTelemetry trace with its goal, its calls and its outcome attached.

If your agent is instrumented at all, you already have this part. Every run generates a trace: the goal it was given, every model call, every tool invocation, every observation from the environment, and the eventual outcome. That trace is the raw material everything downstream depends on.

The industry is converging on a shared vocabulary for this. OpenTelemetry’s GenAI working group has been building [standard span attributes for LLM calls](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/) since 2024, covering model name, token counts, latency, and tool execution as first-class fields rather than something each team invents from scratch. That matters more than it sounds like it should, because a trace schema you have to redesign every time you switch observability vendors is a trace schema nobody trusts enough to build a dataset on top of.

[Overmind](https://www.overmindlab.ai/) captures this stage for you. Overmind is the model training platform for AI teams, and the whole product starts here, turning your production traces into specialized models you own, trained, benchmarked and served. One `overmind.init(providers="auto")`call in a Python agent captures every LLM invocation across OpenAI, Anthropic, Google Gemini, Agno and LangChain or LangGraph, logging inputs, outputs, latency, token counts and errors without extra plumbing on your side. Anything else that speaks OpenTelemetry can post its traces straight to the [Observability](https://www.overmindlab.ai/product/observability) endpoint, and traces already sitting in Langfuse, LangSmith, Braintrust or Galileo import through a connector. The point isn’t the SDK itself. It’s that capture has to be automatic and total, or the sampling and labeling stages downstream never get the raw material they need.

## Step 2: how do you choose which runs to label?

You choose runs to label with a random slice for a baseline, a stratified slice for the segments you care about, and every run that failed or scored badly.

You should not try to label every run. Most of what an agent does in a given week is unremarkable, and reviewing all of it teaches a labeling team nothing it didn’t already know. The job at this stage is picking which runs are worth a human’s attention.

Three strategies cover most cases.

- **Random** gives you an honest baseline of what the agent does on an ordinary day. Use it every cycle, as the control slice you compare everything else against.
- **Stratified** gives you deliberate coverage of cases you already care about, such as a specific refund reason, a customer tier, or a tool that keeps timing out. Use it when a known segment matters more than the average run.
- **Failure-weighted** gives you the most signal per run, because the runs that went wrong carry the most information. Use it when you have an error flag or a satisfaction score to sort on.

Out of the refund agent’s 50,000 weekly runs, a reasonable pull is around 500: a random slice plus every run that hit an error or a low satisfaction score. Whatever strategy you pick, run it on a cadence, weekly to start, rather than treating it as a one-off export you remember to do after something breaks in production.

## Step 3: how do you label runs against a spec?

You label runs by writing down what good looks like as a checklist of criteria, then scoring every sampled run against that checklist instead of by gut.

This is the step almost everyone skips, and it’s the one that decides whether the dataset is worth anything.

The instinct is to label by gut: skim a run, decide it feels fine, move to the next one. That works at a hundred runs and falls apart completely at ten thousand, mostly because “feels fine” means something slightly different to every reviewer and drifts over time even for the same reviewer. The alternative is writing down, explicitly, what good looks like, and scoring every sampled run against that spec.

You’re not starting from nothing here. Your agent’s own codebase already encodes most of what it’s supposed to do: the outputs it should produce, the tools it’s allowed to call, the checks it runs before acting, the paths it should and shouldn’t take. Read the code first and most of the labeling criteria fall directly out of it.

For the refund agent, that read produces a checklist along these lines.

- **Refund within terms.** Amount within the order value and the 30-day window. Violation example: refunded a 90-day-old order.
- **No invented terms.** Only cites the published refund terms. Violation example: quoted a returns rule that does not exist.
- **Escalate disputes.** Hands chargebacks off to a human. Violation example: auto-refunded a disputed charge.

Automating this scoring step is where LLM-as-judge techniques have become the practical default, since manually reviewing thousands of runs against a rubric doesn’t scale. The approach has real limits worth knowing before you lean on it: research comparing LLM judges against human-labeled relevance data found [strong rank correlation but only fair agreement on exact labels](https://eugeneyan.com/writing/llm-evaluators/), and accuracy drops sharply on the more nuanced categories rather than the easy pass/fail calls. A [survey of LLM-as-judge methods](https://arxiv.org/html/2412.05579v2) notes it emerged specifically because manually assessing helpfulness in training data got too expensive to do at scale by hand, which is exactly the tradeoff a labeling pipeline is making. Overmind runs this scoring as evaluators against a rubric you write. Five evaluator kinds cover it, from a deterministic check to an LLM judge, and the rubric is compiled into a checklist of weighted yes/no questions, any of which you can mark as must-pass. In an eval run every variant is scored against a baseline you choose, so a changed rubric or a changed prompt is a comparison rather than a new number.

The runs that fail a criterion, whatever the customer clicked afterward, are the highest-value training data you have. The labels encode your judgment about what good looks like, and that judgment is the one thing no generic tool can supply for you. Do the first pass yourself for the first month. It’s the fastest way to find out what your criteria actually are, as opposed to what you assumed they were when you wrote the checklist.

## Step 4: how do you build the training set?

You build the training set by keeping the runs your labels approve, shaping each one into the format your training method needs, and attaching provenance to every row.

Labeled runs aren’t training examples yet. The last step shapes them for whatever method you’re about to run, and the method decides the shape.

- **Which runs you keep.** SFT: runs that passed every criterion. RL: sampled runs with their criterion scores attached.
- **What a row holds.** SFT: an input-output pair of the behavior you want repeated. RL: the run plus the scores, used as a reward signal.
- **What the model learns.** SFT: to imitate a fixed set of good examples. RL: to produce runs that score higher.
- **What labeling has to produce.** SFT: a pass/fail verdict per criterion. RL: a usable score per criterion, not just pass/fail.

Either way, the format needs to be consistent and machine-readable, with the goal and outcome attached to every run. Provenance belongs in the format too. In Overmind’s Datasets, every row keeps its link to the trace it came from, so you can always answer where a training example came from, and training uses that link to warn you when an evaluation row shares a trace with a training row. Overmind trains with supervised fine-tuning, LoRA or full, so the left-hand column is the shape its training datasets take.

This is also the stage where the case for smaller, specialized models gets concrete. Overmind’s own research argues that [every model invocation inside an agentic workflow is a natural source of high-quality training data](https://www.overmindlab.ai/research/think-smaller), precisely because the prompts are narrow and well-defined and the pass/fail signal is clean, unlike open-ended chat data. A team that instruments its model calls, clusters the resulting patterns, and fine-tunes a specialist model on them ends up with a system that improves with every production run instead of one that’s frozen at whatever a general-purpose model happened to learn at pretraining time.

Which method you feed, supervised fine-tuning or RL, is [its own separate decision](https://www.overmindlab.ai/research/types-of-fine-tuning). The dataset from steps 1 through 3 is what feeds either one.

## Which tools turn production traces into datasets?

Overmind builds evaluation and training datasets from production traces and trains an open-weight model on them in the same product. LangSmith and Langfuse add production traces to datasets for evaluation and export them for training elsewhere. Label Studio is a labeling tool you feed exported runs into.

Here is what each one does at this stage, taken from its own documentation.

- **Overmind** selects production traces into a dataset from Observability, or imports them from a Langfuse, LangSmith, Braintrust or Galileo connector. Each run lands as one row with its input, delivered output, full transcript, tool calls, model, tokens, cost and score. The dataset’s own agent then prepares and audits the rows and proposes each change as a readable Python step you approve or deny, before the dataset feeds an eval run or a training job on one of 38 open-weight base models ([datasets docs](https://docs.overmindlab.ai/core/datasets), [training docs](https://docs.overmindlab.ai/models/training)).
- **LangSmith datasets** are versioned, the LangSmith Engine can generate ground-truth examples from production traces, and a dataset exports to CSV, JSONL or OpenAI’s fine-tuning format; training happens elsewhere ([manage datasets docs](https://docs.langchain.com/langsmith/manage-datasets)).
- **Langfuse datasets** add items from a trace or in batch from the observations table, keep a `source_trace_id` on each item, and export CSV or JSON from the UI; Langfuse is open source and does not train models ([datasets docs](https://langfuse.com/docs/evaluation/experiments/datasets), [export docs](https://langfuse.com/docs/api-and-data-platform/features/export-from-ui)).
- **Label Studio** is an open-source labeling platform that now lists agent traces and LLM evals among its data types; you export runs into it, label them there, and export the labels back out. It does not capture traces or train models ([labelstud.io](https://labelstud.io/)).

Prices and formats move; every claim above was checked against the linked page on 5 October 2026.

## How do you do it in Overmind?

Four steps, from a running agent to a dataset a training job can read.

1. **Capture.** Add `overmind.init(providers="auto")` to your Python agent, or point any OpenTelemetry exporter at Overmind. From Cursor, Claude Code, OpenCode or Codex, `/overmind ensure-tracing` checks the instrumentation for you. Traces already in Langfuse, LangSmith, Braintrust or Galileo come in through a connector.
2. **Select.** In Observability, filter the root traces you want and choose **Add to dataset**, or **Data from traces** to take everything matching the filter. Pick **Evaluation**, **Training**, or **Train + eval**, which splits the source with a 30% evaluation share by default and keeps matching inputs, traces and conversations on the same side.
3. **Shape.** Open the dataset in the [Data Workshop](https://www.overmindlab.ai/product/data-workshop). Its agent prepares and audits the rows first, then proposes changes as readable Python steps you approve or deny. Each step is a version, frozen when a run uses it. Or ask your coding agent: `/overmind dataset`.
4. **Use.** Start an eval run on the evaluation half, or `/overmind finetune` on the training half. The [datasets docs](https://docs.overmindlab.ai/core/datasets) cover every option, and the [quickstart](https://docs.overmindlab.ai/quickstart) gets a first trace in within a few minutes.

## Why is the dataset the hard part?

The dataset is the hard part because it has to be rebuilt every week against new traces and sharper criteria, and most tooling stops at handing you the traces.

None of these four steps is exotic on its own. What makes the whole thing difficult is that it never stops. Production keeps producing new traces, your criteria keep getting sharper as you find edge cases you didn’t anticipate, and the dataset has to be rebuilt against what users actually did this week, not what they did last quarter. The dotted feedback line on that diagram at the top is the entire job.

It’s also exactly the gap most observability tooling leaves open. Datadog’s own writeup on GenAI tracing gets at this directly: teams are encouraged to [promote interesting production traces into curated, version-controlled “golden” datasets](https://www.datadoghq.com/blog/llm-otel-semantic-convention/) and layer evaluation metadata on top, which is essentially this same capture-to-label pipeline described from the observability side. An observability platform hands you the traces and stops there. Stitching production traces to behavioral training data to a deployed, improved model is work that mostly happens in spreadsheets and one-off scripts today, and Overmind is built to close that gap, running capture, dataset, eval and training in one open-source platform instead of leaving it as a manual export.

Owning that labeled dataset matters because differentiation in agentic AI increasingly lives in proprietary behavioral data, not in which foundation model you call. Anyone can capture traces. The labeled dataset built from them, tuned to your own definition of correct, is the thing actually worth owning.

## Frequently asked questions

### What is the difference between a trace and a training example?

A trace is the full record of one agent run, from the input through every model and tool call to the output and its timing. A training example is a curated row the model should learn from, usually an input paired with the output you want. Most traces never become training examples; sampling and labeling decide which do.

### Should failed runs go into a fine-tuning dataset?

Not as they are. Supervised fine-tuning teaches a model to copy its examples, so failed runs teach failure. Use them as evaluation cases, or correct the output first and train on the corrected version. Keep the successful runs that represent the work you want repeated.

### Can an LLM label my traces for me?

For a first pass, yes. An LLM judge scoring each run against a short written rubric is the usual start. Check a sample against human labels before trusting it, and break the rubric into small yes/no checks so you can see which one disagrees. Overmind compiles each rubric into a weighted yes/no checklist in which any item can be marked must-pass.

### How do i keep evaluation data out of my training data?

Split by run, not by row, so steps from one trace or conversation never land on both sides. In Overmind, a Train + eval dataset splits with a 30% evaluation share by default and keeps matching inputs, traces and conversations together; training reports any overlap it finds.

### Which tools turn production traces into training datasets?

Overmind builds evaluation and training datasets from production traces, then trains an open-weight model on them. LangSmith and Langfuse can add production traces to datasets for evaluation, and LangSmith exports datasets in OpenAI’s fine-tuning format. Label Studio is a labeling tool you feed exported runs into.

_Last updated 5 October 2026._

_Overmind is the model training platform for AI teams. It turns your production traces into specialized models you own._ _[Get started](https://console.overmindlab.ai/)._

## Related research

- [My summer at Overmind](https://www.overmindlab.ai/research/my-summer-at-overmind): A summer internship reflection on autonomy, hard problems, and building across the Overmind product.
- [Small is the new frontier](https://www.overmindlab.ai/research/small-is-the-new-frontier): Why specialized models trained on your own production data are becoming the practical path to owned intelligence.
- [When bigger isn’t better](https://www.overmindlab.ai/research/when-bigger-isnt-better): On specialized tasks, Overmind-trained small language models outperform frontier models on accuracy, hallucination, and cost.

---

# https://www.overmindlab.ai/research/prompt-engineering-vs-fine-tuning

# Prompt engineering vs fine-tuning

> Canonical: https://www.overmindlab.ai/research/prompt-engineering-vs-fine-tuning

- Author: Rohit Gupta
- Category: Product
- Tags: Data
- Published: 2026-07-30
- Updated: 2026-10-14

> Prompt engineering changes what you send the model. Fine-tuning changes the model's weights. Start with prompts because they are fast, cheap and reversible. Fine-tune when the eval stops improving, the task is narrow, and you have a few thousand real examples of the output you want.

![Prompt engineering vs fine-tuning](https://cdn.sanity.io/images/gv4t9qa3/production/bc5a4b0f8fd514b945fe5fff7674c79dd7bd288a-2048x2048.png)

Most teams building LLM applications start the same way: they write a system prompt, test it against a few examples, and ship. [Prompt engineering](https://docs.overmindlab.ai/agent-testing/optimisers) is cheap, fast, and reversible. [Fine-tuning](https://docs.overmindlab.ai/models/training) used to be something only teams with a training pipeline and a GPU budget could afford. Things have changed but the order of operations hasn't.

Both techniques steer model behavior. Prompting changes what you say to the model. Fine-tuning changes the model itself.

## Prompt engineering vs fine-tuning, side by side

|  | Prompt engineering | Fine-tuning |
| --- | --- | --- |
| What changes | The input | The model's weights |
| Where it lives | The context window | The model |
| Cost to try | Seconds, no infrastructure | GPU time and a training pipeline |
| Data needed | None, or a handful of examples | Thousands of labeled runs |
| Undo | Instant | Only by retraining |
| Model swap | Survives it | Does not survive it |
| Ceiling | Plateaus quickly | High, bounded by data quality |

*Prompt engineering vs fine-tuning, side by side*

![Table comparing prompt engineering and fine-tuning across seven rows. Prompt engineering changes the input, lives in the context window, costs seconds with no infrastructure, needs no data or a handful of examples, reverses instantly, survives a model swap, and plateaus quickly. Fine-tuning changes the model's weights, lives in the model, costs GPU time and a training pipeline, needs thousands of labeled runs, reverses only by retraining, does not survive a model swap, and has a high ceiling bounded by data quality.](https://cdn.sanity.io/images/gv4t9qa3/production/c280c9e392ba662b5b8e8edd8b99e7d20ee24f3a-2400x1192.png)

## Why prompting is the right first move

Prompt engineering means editing what enters the context window. No adjustment of the model's weights. You're changing the input and reading the output.

For a new agent, that's almost always the correct starting point. You learn what the model can already do before you spend anything on training. Most of the behavior changes you want, prompting will give you.

## How to tell which regime you're in

This isn't a one-time decision so much as a loop. You stay in prompt-engineering mode until the eval curve goes flat, then check whether you actually have what fine-tuning needs: a narrow task and real data.

![Decision flowchart. When new behavior is needed, ship prompt changes while the eval is still moving. Once the eval is flat for three to four changes, check whether you have a narrow task and a few thousand labeled runs: if not, build the dataset first; if yes, fine-tune.](https://cdn.sanity.io/images/gv4t9qa3/production/272459fe135b345eb693c8e046d428d70e6bd364-2042x1952.png)

*Ship prompt changes until the eval goes flat, then fine-tune.*

## Where prompting hits its ceiling

Prompting always runs out of road eventually, and the symptoms show up in predictable order:

- **The plateau.** Each prompt tweak moves the eval less.
- **Whack-a-mole.** The instruction that fixes today's failure mode causes tomorrow's.
- **Prompt bloat.** The system prompt keeps growing, slows the model down, and gets partially ignored.
- **Paying rent.** You're re-sending the same correction on every single call.

At that point, prompting stops being cheap. Engineering hours are the priciest line item in most LLM pipelines, and it's easy to burn a month nudging a prompt that plateaued weeks ago. 

## When to fine-tune

Fine-tuning updates a model's weights on examples of the behavior you want until the model does it by default, without being told every time. It's worth trying when these conditions are met:

- **The task is narrow.** Fine-tuning sharpens a model on one job, not general capability.
- **You have real data.** A few thousand labeled runs, ideally sampled from production traffic rather than synthetic examples.
- **Cost or latency matters.** A smaller, fine-tuned model [can outperform a much larger general-purpose one](https://www.overmindlab.ai/research/when-bigger-isnt-better) carrying a bloated prompt, at a fraction of the inference cost. On one product eval, a 0.5B trained specialist beat GPT-4o mini 50% to 29%.

Deciding whether to fine-tune an open-weight model or keep renting a frontier one? See [open-weight vs frontier models](https://www.overmindlab.ai/research/open-weights-vs-frontier-apis).

## Where RAG fits

Retrieval-augmented generation (RAG) is the third option, and it solves a different problem. Prompting and fine-tuning both shape how the model behaves. RAG changes what the model knows at the moment it answers, by fetching documents and putting them in the context window.

Use RAG when the answer depends on facts that change, such as product docs, policies or customer records. Use fine-tuning when the model already has the knowledge it needs but gets the format, tone or decision wrong. Fine-tuning is a poor way to teach facts that change next week, because every change means another training run.

Many production systems use all three. RAG supplies the facts, a short prompt supplies the per-request instructions, and a fine-tuned model supplies the behavior.

## Can you use prompt engineering and fine-tuning together?

Yes, and most teams should. A fine-tuned model still takes a prompt. The difference is what the prompt has to carry. Before fine-tuning, the system prompt holds every rule and every correction, and you pay to resend it on every call. After fine-tuning, the behavior lives in the weights and the prompt can shrink to the instructions that change per request.

Keep your eval set fixed across both stages. Score the prompted baseline, train, then score the fine-tuned model on the same set. That is the only fair way to tell whether the training run was worth it. [How to freeze the eval set before you train](https://www.overmindlab.ai/research/how-to-train-your-agent).

## Prompt, RAG or fine-tune? Worked examples

| Problem | Start with | Why |
| --- | --- | --- |
| Replies are too long or the wrong tone | Prompt engineering | One instruction fixes it, and you can undo it instantly |
| The agent needs this week's pricing or policy | RAG | The facts change faster than you could retrain |
| A classification or extraction step runs on every request | Fine-tuning | Narrow task, high volume, and production gives you the examples |
| The system prompt keeps growing and each fix breaks something else | Fine-tuning | You have hit the prompting ceiling |
| A new agent with no traffic yet | Prompt engineering | You have no data to fine-tune on |

*Which technique fits which problem*

## How Overmind handles both

On the prompt side, the Overmind [Optimizer](https://www.overmindlab.ai/product/agent-testing) has your coding agent rewrite prompts, tool descriptions and logic as real git diffs, replays each candidate over your dataset and scores it with your evaluators. You apply the winning diff yourself. On the fine-tuning side, you [build a dataset from production traces](https://www.overmindlab.ai/research/traces-to-training-dataset), [fine-tune an open-weight base model](https://www.overmindlab.ai/product/model-training) with LoRA or full supervised training, and benchmark the result with the same evaluators before it serves traffic. The weights are downloadable.

## FAQ

**What is the difference between prompt engineering and fine-tuning?**

Prompt engineering changes the input you send to a model and leaves the model untouched. Fine-tuning trains the model on examples of the behavior you want, which changes its weights so it behaves that way without being told. Prompting is instant and reversible. Fine-tuning takes data and GPU time, and you undo it only by retraining.

**Is fine-tuning better than prompt engineering?**

Neither is better in general. Prompting is the right first move for almost every new agent. Fine-tuning wins on narrow, high-volume tasks once prompting has plateaued and you have real examples to train on. A smaller fine-tuned model can then beat a larger prompted one at a fraction of the inference cost.

**How much data do you need to fine-tune?**

A few thousand labeled examples is a sensible starting point, ideally sampled from production traffic rather than written synthetically. Quality matters more than volume. [Types of fine-tuning](https://www.overmindlab.ai/research/types-of-fine-tuning) covers which method fits which amount of data.

**What is prompt tuning?**

Prompt tuning is not the same as prompt engineering. It trains a small set of soft prompt vectors that are prepended to the input, while the model's own weights stay frozen. It sits between the two approaches. It needs training data like fine-tuning, but changes far fewer parameters.

**Is fine-tuning still worth it?**

Yes, for narrow tasks at volume. Frontier models keep getting better at everything, but most production agents do a handful of jobs over and over. A small model trained on those jobs can match or beat a much larger one on them, at a fraction of the cost per call, and you own the weights.

**Does fine-tuning replace the prompt?**

No. A fine-tuned model still takes a prompt, but a much shorter one. The rules you used to repeat on every call move into the weights.

_Overmind is the model training platform for AI teams. It turns your production traces into specialized models you own. [Get started](https://console.overmindlab.ai)._

## Related research

- [My summer at Overmind](https://www.overmindlab.ai/research/my-summer-at-overmind): A summer internship reflection on autonomy, hard problems, and building across the Overmind product.
- [Small is the new frontier](https://www.overmindlab.ai/research/small-is-the-new-frontier): Why specialized models trained on your own production data are becoming the practical path to owned intelligence.
- [When bigger isn’t better](https://www.overmindlab.ai/research/when-bigger-isnt-better): On specialized tasks, Overmind-trained small language models outperform frontier models on accuracy, hallucination, and cost.

---

# https://www.overmindlab.ai/research/types-of-fine-tuning

# What are the different types of fine-tuning?

> Canonical: https://www.overmindlab.ai/research/types-of-fine-tuning

- Author: Rohit Gupta
- Category: Tutorials
- Tags: Models
- Published: 2026-07-30
- Updated: 2026-10-14

> What's the difference between SFT, LoRA and Distillation?

![What are the different types of fine-tuning?](https://cdn.sanity.io/images/gv4t9qa3/production/13e541368bc3dbd110205567b9469d3d1649e05e-2048x2048.png)

Not sure you need to fine-tune at all? Start with [prompt engineering vs fine-tuning](https://www.overmindlab.ai/research/prompt-engineering-vs-fine-tuning).

## Supervised fine-tuning (SFT)

SFT is where most people start, and most people should. You give the model input/output pairs and train it until it reproduces that pattern. It's teaching by worked example: here's the prompt, here's the answer, do more of this.

You'll typically need a few thousand examples for a narrow task. The catch is that SFT only works if you know what "good" looks like.

## LoRA and QLoRA

LoRA doesn't compete with SFT or RFT.

Full fine-tuning updates every weight in the model. That's slow, and it eats memory. [LoRA](https://arxiv.org/abs/2106.09685) freezes the base model and trains a small set of adapter weights on top, which gets you most of the benefit for a fraction of the compute. [QLoRA](https://arxiv.org/abs/2305.14314) pushes this further by quantising the frozen base model down to 4-bit before training the adapter. That's the reason a 7B model now fits on one consumer GPU instead of a rack of them.

Tools worth knowing:

[Unsloth](https://unsloth.ai/) is open source and runs LoRA/QLoRA 2 to 5x faster with up to 80% less VRAM, using custom kernels and 4-bit quantisation. It's become the default starting point for solo builders.

[Tinker](https://thinkingmachines.ai/tinker/), from Thinking Machines Lab, is a managed LoRA API. It handles GPU scheduling and checkpointing but still lets you control the actual training algorithm. It [launched in October 2025](https://thinkingmachines.ai/news/announcing-tinker/) and supports both SFT and RFT.  
  
[Overmind](https://www.overmindlab.ai/) is a model training platform. It runs LoRA fine-tuning on your agent traces.

## Reinforcement fine-tuning (RFT)

This used to just be called "RL," or RLHF if you wanted to impress someone. 

SFT needs you to write the correct answer. RFT only needs you to score it. You define a reward, hand it the model's output, and the model learns to produce completions that score higher across many attempts. 

Reach for this when good is easier to recognize than to demonstrate. I can't write the perfect customer support reply off the top of my head, but I can tell you whether one resolved the ticket, stayed on brand, and didn't promise something we don't offer. RFT is also the natural fit for agents: tool calls, multi-step retrieval, anything where success depends on the whole trajectory and not any single message.

GRPO (Group Relative Policy Optimization) is the algorithm doing most of the RFT work right now. It came out of [DeepSeekMath](https://arxiv.org/abs/2402.03300) and got famous when it trained [DeepSeek-R1](https://arxiv.org/abs/2501.12948). The trick is sampling a group of responses to the same prompt, then scoring each one against the group average instead of some absolute scale. That kills the need for a separate critic model, which is what made older methods like PPO so expensive to run. [DAPO](https://arxiv.org/abs/2503.14476) and [Dr.GRPO](https://arxiv.org/abs/2503.20783) are newer variants that patch specific instabilities GRPO runs into on long chain-of-thought outputs.

## Distillation

Distillation is copying a big model's behavior into a small one. Run your inputs through the big model, collect what it says, then train a smaller model to say the same things. You lose a little quality and gain a lot of speed, plus a much smaller bill.

This is helpful when a frontier model already nails your task but is too slow or too expensive to run at real volume. It's also a big part of why small language models have taken off this past year. Teams distill a specialist model instead of shipping the 400B-parameter original into production.

## Picking one

Once you know your task, the decision is mostly mechanical.

![Decision tree for choosing a fine-tuning method. If a big model already does the job but is too slow or costly, use distillation. Otherwise, if you can write the ideal output, use supervised fine-tuning; if you can only judge it, use reinforcement fine-tuning with GRPO. Every path then runs through LoRA or QLoRA.](https://cdn.sanity.io/images/gv4t9qa3/production/86d2fbfde84fb76f9d772f0178e7e2687e6dde25-2036x1734.png)

## Data is still the hard part

Every method above needs the same raw material underneath. SFT needs labeled pairs. RFT needs a reward or a judge that can rank attempts. Distillation needs a teacher and your inputs. Pick the wrong method and you lose some efficiency. Show up with bad data and none of them work. No algorithm fixes that for you.

Fine-tuning has gotten cheap. A LoRA run on a 7B model over a few thousand examples often costs a few hundred dollars. But cheap training doesn't fix bad data, and none of these tools hand you a set of real runs labeled, or ranked, against your own definition of good. That's a separate problem, and it's the one that eats the time. We cover it in [how do you turn traces into a training dataset](https://www.overmindlab.ai/research/traces-to-training-dataset) and [how to train your agent](https://www.overmindlab.ai/research/how-to-train-your-agent).

## FAQ

**Is LoRA a replacement for fine-tuning?**

No. LoRA is a way to run fine-tuning or RFT cheaper, not a different goal. You still pick SFT or RFT first, then decide whether to run it through LoRA.

**When should I use reinforcement fine-tuning instead of SFT?**

When you can judge a good output but can't write one yourself, or you're training an agent where success depends on a whole multi-step trajectory rather than one response.

**Is distillation cheaper than fine-tuning a small model from scratch?**

Usually, if a large model already performs well on your task. You're paying for inference calls to the teacher model instead of building the target behavior from labeled data by hand.

**What is GRPO and why does everyone use it now?**

Group Relative Policy Optimization scores a batch of responses against each other instead of an absolute baseline, which removes the need for a separate critic model. It's cheaper to run than older RL methods like PPO, which is most of why it's become the default for RFT.  
  
_Overmind is the model training platform for AI teams. It turns your production traces into specialized models you own. [Get started](https://console.overmindlab.ai)._

## Related research

- [My summer at Overmind](https://www.overmindlab.ai/research/my-summer-at-overmind): A summer internship reflection on autonomy, hard problems, and building across the Overmind product.
- [Small is the new frontier](https://www.overmindlab.ai/research/small-is-the-new-frontier): Why specialized models trained on your own production data are becoming the practical path to owned intelligence.
- [When bigger isn’t better](https://www.overmindlab.ai/research/when-bigger-isnt-better): On specialized tasks, Overmind-trained small language models outperform frontier models on accuracy, hallucination, and cost.

---

# https://www.overmindlab.ai/research/open-weights-vs-frontier-apis

# Open-weight vs frontier models: when to rent, when to own

> Canonical: https://www.overmindlab.ai/research/open-weights-vs-frontier-apis

- Author: Pritam Soni
- Category: Research
- Tags: Models
- Published: 2026-07-30
- Updated: 2026-10-14

> Frontier models are the most capable models available, and most are closed and rented through an API. Open-weight models publish their weights, so you can fine-tune them and run them where you choose. Rent while volume is low and the task is broad. Own once volume is high, the task is narrow, or data cannot leave your perimeter.

![Open-weight vs frontier models: when to rent, when to own](https://cdn.sanity.io/images/gv4t9qa3/production/8acf0910d23b7f1e53e144b4718249082a2c2fb4-2048x2048.png)

Developers start building AI products by tokenmaxxing. Calling frontier APIs and paying per token. 

Then the bill shows up. Or the agent feels slow. Or legal asks where the data goes. Suddenly "just use the API" might not be the right strategy.

Underneath every AI product is one choice: **rent a frontier API, or own a model**. Renting means closed weights: someone else trains it, serves it, and sets the price. Owning means picking the model, post-training it and running **self-hosted** or third-party inference.

This piece is a practical guide to open-weight vs frontier models, and when each one wins.

## The trade-offs

|  | Frontier API (closed) | Open-weight model (owned) |
| --- | --- | --- |
| Capability | Best out of the box | Strong, and tunable to your task |
| Infrastructure | None needed | Infrastructure you own or buy |
| Cost | Rises with every token | Drops hard after the upfront work |
| Speed | Network trip and a shared queue | Dedicated hardware, lower time to first token |
| Data | Prompts leave your network | Stays on your servers or in your cloud |
| Control | Vendor sets pricing and retirement dates | Your stack, your weights |
| Switching | One line of code | Retrain or re-quantize |

*Frontier API vs open-weight model, side by side*

![Table comparing closed models and open models across seven rows. Closed models give the best capability out of the box, need no infrastructure, cost more with every token, add a network trip and a shared queue, send prompts outside your network, lock you to the vendor's pricing and retirement schedule, and swap with one line of code. Open models are strong and tunable to your task, run on infrastructure you own or buy, drop hard in cost after the upfront work, serve on dedicated hardware with lower time to first token, stay on your own servers or in your own cloud, leave you your stack and your weights, and need a retrain or a re-quantise to swap.](https://cdn.sanity.io/images/gv4t9qa3/production/7eeee6a391b28cc8f95e2b11e4d07d8c2fc18469-2400x1630.png)

## LLM inference cost: when open-weight models beat the API

Renting feels cheap at the start, especially when showered with credits.

Then usage grows. The bill grows and a painful chunk of that spend is frontier prices for work [a smaller specialized model could do](https://www.overmindlab.ai/research/think-smaller).

Owning the model changes that. You pay upfront: a [**LoRA / QLoRA**](https://www.overmindlab.ai/research/types-of-fine-tuning) fine-tune, inference setup, some evals. After that, **per-token cost** is a fraction of the API. Somewhere on the volume axis the lines cross. That **crossover point** is where a self-hosted open-weights LLM stops being a science project and starts being the cheaper path.

![Chart of cost per month against tokens per month. The rent line rises linearly with usage; the own line is near-flat after an upfront build. Past the crossover point, every token is cheaper owned.](https://cdn.sanity.io/images/gv4t9qa3/production/f6f948f91b19f993b836862430560e64c4d10675-1920x1434.png)

This is not theory. A LoRA fine-tune of a 7B-8B open-weights model on a few thousand trajectories often costs a few hundred dollars.

## Beyond cost: latency, data residency, and vendor lock-in

### Your agent feels sluggish

A small model you serve has no hop to someone else's cluster and fewer parameters per forward pass. If your AI agent makes dozens of tool calls per task, **200ms vs 2s** is not a benchmark flex, it is whether the product feels alive. Lower **time to first token (TTFT)** compounds across the whole agent loop. Users notice the speed. They leave when it isn't there.

### Your data cannot leave the building

When you rent, prompts and completions leave your perimeter. For fintech, healthcare, or anyone under data-residency, GDPR, or HIPAA rules, that is not a preference. It is a hard no. A **private LLM**, self-hosted or VPC-hosted open-weights, is not an optimization. It is how you ship at all.

### The vendor moved the goalposts

Renting means their roadmap, their rate limits, [their deprecation calendar](https://www.overmindlab.ai/compare/openai-fine-tuning-alternatives). When they sunset the model your agent was tuned around, you re-tune on their schedule. Owning means your inference stack and your weights. Still a dependency, but one **you control**.

## How far behind are open-weight models?

Not far, and the gap is measured in months. [Epoch AI found](https://epoch.ai/data-insights/open-closed-eci-gap) that since January 2026 the most capable open-weight models have trailed the best closed models by about four months on average. For a narrow production task the gap matters less than that headline suggests. You are not asking a model to be good at everything, only at your job, and fine-tuning on your own data closes much of the difference that is left.

## When to use an open-weight model vs a frontier API

**Own (open-weights, self-hosted) when:**

- volume is high and climbing
- the task is narrow
- latency is a product problem
- data cannot leave your perimeter
- the model _is_ your edge
- you know what you are building

**Rent (frontier API) when:**

- volume is low or spiky
- the task is broad and messy
- latency is fine
- data has no residency limits
- raw frontier capability matters most
- you are still finding out

Not sure your task is narrow enough yet? Start with [prompt engineering vs fine-tuning](https://www.overmindlab.ai/research/prompt-engineering-vs-fine-tuning).

## FAQ: open-weight vs frontier models

**What is the difference between open-weight and frontier models?**

A frontier model is one of the most capable models available, and most of them are closed. You reach them only through the vendor's API, and the vendor trains, serves and prices them. An open-weight model publishes its trained weights, so you can download it, fine-tune it on your own data and run it on infrastructure you choose.

**Why are open-weight models cheaper to run?**

With an API you pay frontier prices on every token, forever. With an open-weight model you pay upfront for a fine-tune, inference setup and some evals, and after that the per-token cost is a fraction of the API price. Past the crossover point on the volume axis, every token is cheaper owned.

**Do I need a team of ML engineers to fine-tune an open-weights LLM?**

No. LoRA / QLoRA plus a managed inference host (such as [Overmind](https://www.overmindlab.ai/product/model-training), or vLLM on your own GPUs) is within reach of one strong engineer. The hard part is not GPUs. It is good training data and honest evals.

**Which open-weights base model should I start with?**

[Pick a family that fits your license and task](https://www.overmindlab.ai/models): Llama, Qwen, Mistral, or DeepSeek. Start small. A 7B-8B model, fine-tuned on your task, handles more production work than most teams expect.

**What if a better frontier model ships next month?**

If you rent, you get the upgrade (until the API changes). If you own, you decide whether to re-tune. Here is the part people miss: a small model trained on _your_ production trajectories often still wins on **your**[ cost, latency, and task metrics](https://www.overmindlab.ai/research/when-bigger-isnt-better). That is the scoreboard that pays the bill, not the leaderboard screenshot.

**Is my data really at risk on a frontier API?**

For most consumer SaaS a zero-retention, no-training contract covers you. For regulated data, get an explicit guarantee in writing or run a private, self-hosted LLM. Do not guess.

## Fine-tuning on production data is the real moat

Owning only works if your model is actually better at _your_ task than the rented one. Downloading open-weights does not get you there. [**Fine-tuning on your production data does**](https://www.overmindlab.ai/research/traces-to-training-dataset): traces, tool-call trajectories, preference pairs, real evals from real usage.  
  
_Overmind is the model training platform for AI teams. It turns your production traces into specialized models you own. [Get started](https://console.overmindlab.ai)._

## Related research

- [My summer at Overmind](https://www.overmindlab.ai/research/my-summer-at-overmind): A summer internship reflection on autonomy, hard problems, and building across the Overmind product.
- [Small is the new frontier](https://www.overmindlab.ai/research/small-is-the-new-frontier): Why specialized models trained on your own production data are becoming the practical path to owned intelligence.
- [When bigger isn’t better](https://www.overmindlab.ai/research/when-bigger-isnt-better): On specialized tasks, Overmind-trained small language models outperform frontier models on accuracy, hallucination, and cost.

---

# https://www.overmindlab.ai/research/anatomy-of-an-ai-agent

# The anatomy of an AI agent

> Canonical: https://www.overmindlab.ai/research/anatomy-of-an-ai-agent

- Author: Pritam Soni
- Category: Research
- Tags: Agents
- Published: 2026-07-30
- Updated: 2026-08-07

> An agent has a model, loop, tools and memory.

![The anatomy of an AI agent](https://cdn.sanity.io/images/gv4t9qa3/production/9f97b4168f325ce891c48c8088035f10aa4bddfa-2048x2048.png)

In 2024, most AI products were chatbots: a single LLM inference call, one prompt in, one completion out. In 2026, the default architecture is an **AI agent**: an agentic system that plans, calls tools, and iterates toward a goal across many steps.

Instead of returning the first model response, teams wrap the LLM in an **agent harness**, expose **tool calling** (APIs, databases, code execution), and let the system work toward an objective. Frameworks such as LangChain, Agno, CrewAI, and the OpenAI Agents SDK exist to make it easier to build the harness.

This article decomposes an AI agent into its core components. We define each part of the **AI harness** and walk through one complete agent run.

## The five components of an AI agent

Strip an agentic system down and you get five components. Four you assemble and the fifth is what they produce.

- **Model.** Selects the next action given goal and state.
- **Harness.** Runs the loop: reason, act, observe, repeat.
- **Tools.** How the agent interacts with external systems.
- **Memory.** How the agent persists and retrieves state.
- **Trajectory.** The record of one complete agent run.

### The model

The** language model** is the only component that reasons, and the only one you can train or fine-tune. Conditioned on the goal and the observations so far, it chooses the next action. Most teams start on a **frontier model API** and some migrate to an **open-weights model** they host themselves or run through a third-party provider.

More on that trade-off in [open-weights LLMs vs frontier APIs](https://www.overmindlab.ai/research/open-weights-vs-frontier-apis).

### The harness

The harness assembles what the model sees, reads the action it picks, calls the matching tool, and feeds the result back in. It also decides everything the model does not: which tools exist and how they are described, what stays in the context window, what happens when a tool call fails or comes back malformed, and when to stop, whether that is the goal being met, a step limit, or an error.  
  
This is what agent frameworks give you. It is the part of an agent you can change immediately.

### Tools and tool calling

**Tools** are how the agent reaches outside the model: query a database, call an HTTP API, run code, send an email. OpenAI shipped **function calling** in June 2023, letting the model emit a structured request to invoke a function instead of prose. Anthropic open-sourced the **Model Context Protocol (MCP)** in November 2024 to standardise how agents connect to those tools, and now everything is an MCP server.

### Memory and context

**Memory** is how the agent carries state across loop iterations. Short-term memory is the **context window**: the running transcript of the goal, prior steps, and tool outputs. Longer-term memory is **retrieval**, pulling documents or past runs from outside the window (often via RAG or a vector store). Without memory, the agent loses its way.

### The trajectory

The **trajectory**, also called the **agent trace**, is the full ordered record of a single run: the goal, every LLM call, every tool invocation, every observation, and the final outcome. It is the agent's stack trace. You do not build it; the other four components produce it. It is also the only unit of analysis at which you can honestly say whether the agent succeeded. Capture it.

## One agent run, start to finish

Watch a support agent handle a refund. The user goal enters, the agentic loop runs, an outcome exits.

1. The user goal arrives: refund this customer.
2. The model calls get_order(id); the tool returns the order details.
3. The model checks the order against the refund policy.
4. The model calls issue_refund(amount); the tool returns a confirmation.
5. The model reports back: refund done.

Four model calls, two tool calls, one outcome. That entire sequence is the **agent trajectory**. Change the goal, the tool responses, or the model's sampling behavior and the next run takes a different path. Agents are **non-deterministic**: the same prompt does not guarantee the same execution path.

## Defining your AI stack

Each layer of a typical AI stack controls an aspect of the agent. Almost none control the whole run.

- **LLM gateway.** Observes one request/response at a time. Cannot update the model.
- **Agent observability.** Sees the full trace, after the fact. Cannot update the model.
- **Agent evals.** Produce a score for the run. Cannot update the model.
- **Closed training loop.** Judged trajectories fed back into training. The only layer that can update the model.

## Improving your agent

Improving an AI agent means operating at the level of the **trajectory**, not the individual LLM call: capture the full run, judge it against your definition of success, and feed judged trajectories back into the system. There are two distinct ways to improve an agent. You can improve the harness: rewrite prompts, tighten tool definitions, and so on. Or you can improve the model itself, retraining it on the failure modes in the traces.

This is the loop [Overmind](https://www.overmindlab.ai/) is built to close.

_Overmind is the model training platform for AI teams. It turns your production traces into specialized models you own. [Get started](https://console.overmindlab.ai)._

## Related research

- [My summer at Overmind](https://www.overmindlab.ai/research/my-summer-at-overmind): A summer internship reflection on autonomy, hard problems, and building across the Overmind product.
- [Small is the new frontier](https://www.overmindlab.ai/research/small-is-the-new-frontier): Why specialized models trained on your own production data are becoming the practical path to owned intelligence.
- [When bigger isn’t better](https://www.overmindlab.ai/research/when-bigger-isnt-better): On specialized tasks, Overmind-trained small language models outperform frontier models on accuracy, hallucination, and cost.

---

# https://www.overmindlab.ai/research/so-you-have-observability-now-what

# So, you have observability. Now what?

> Canonical: https://www.overmindlab.ai/research/so-you-have-observability-now-what

- Author: Tyler Edwards
- Category: Research
- Tags: Observability
- Published: 2026-06-15
- Updated: 2026-08-07

> Observing your agent is table stakes.

![So, you have observability. Now what?](https://cdn.sanity.io/images/gv4t9qa3/production/8a9b507cc1d80e9dd4fec285dc08d5933e318ede-2160x1080.jpg)

In the not-so-distant past there was a brief period when picking an LLM meant picking a vendor. You wrote against their SDK, you paid their bill, and when their servers fell over on a Tuesday afternoon so did yours. Switching was easy enough but did require Ctrl+F'ing your way through the codebase - or asking your coding agent of choice to perform this insultingly simple task for you instead.

Then on the sixth day came the LLM gateway! One endpoint, dozens of models behind it, swap them in and out faster than the UK swaps Prime Ministers (and for international readers [here is the context on that one](https://en.wikipedia.org/wiki/Liz_Truss_lettuce)). The gateway is, on its own terms, a genuinely nice piece of engineering, and there is now a number of teams shipping model routers with various bells and whistles: Portkey, LiteLLM, OpenRouter, TrueFoundry, Kong, Cloudflare.

Sitting alongside the gateway, usually paid for separately and instrumented by a different team three weeks later, is the LLM observability stack. Langfuse, Helicone, Arize Phoenix, Braintrust, LangSmith, Datadog now too. Pick your favorite, they mostly do the same thing. They capture every prompt and every completion, stitch them into traces, and present you with dashboards.

Between the two, your AI product is being watched harder than a Premier League VAR decision.

## What is observability?

LiteLLM shipped to solve a pretty annoying problem: every provider's API was almost-but-not-quite the same. Portkey and OpenRouter commercialised the idea, TrueFoundry and Bifrost followed, and the existing API-gateway crowd (Kong, Cloudflare) extended their products to cover LLMs the moment it became clear LLMs were here to stay whether anyone liked it or not.

The observability side has a similar shape. Langfuse, Helicone, Arize Phoenix, Braintrust all turned up in 2023 to build native LLM observability from scratch. The companies that already did ML observability - Fiddler, WhyLabs, Galileo - pivoted with the arrival of ChatGPT and have spent the last three years insisting they were actually always about LLMs really anyway.

By 2026 both categories had matured enough to start being acquired. Portkey went to Palo Alto Networks in April for somewhere around $120-140M. Helicone went to Mintlify the month before. Langfuse was rolled into ClickHouse's Series D in January. Cisco picked up Galileo and folded it into Splunk.

## All data, no insight

So you now have hundreds of thousands, possibly millions, of traces composting nicely in a database somewhere. Every prompt, every completion, every tool call, every retry. The gateway has them as individual requests. The observability tool has them stitched into trajectories. Between the two, your AI product is the most thoroughly documented thing in your engineering org.

The gateway, by design, treats each call as a self-contained unit - the only way to route, cache, fall back and bill at scale. It doesn't have a concept of the trajectory because it doesn't need one. As far as the gateway is concerned, each call was a separate transaction.

The observability tool does have a concept of the trajectory. It will stitch the spans together, draw you a lovely waterfall diagram, and tell you exactly how the agent arrived at whatever it arrived at. What it won't tell you is whether what it arrived at was any good - because _good_ is a judgment, and the observability tool is, as its name suggests, there to observe.

Every call along the way can pass every check either tool runs. A 200 from the provider, the right shape of completion, no PII, latency inside SLO, and cost on budget. The agent can still invent a company policy, ignored an instruction from two steps earlier, and make three perfectly correct-looking calls on its way to [deleting a production database](https://uk.pcmag.com/ai/159249/vibe-coding-fiasco-ai-agent-goes-rogue-deletes-companys-entire-database) - and then gleefully reported its action back to you in the same way a dog might excitedly drop a dead bird at your feet.

## Where observability fails

The observability vendors aren't naive about any of this. Most of them ship some version of trace review - LangSmith has annotation queues, Braintrust has human review alongside LLM-as-judge scoring, Langfuse has dataset workflows, Arize Phoenix ships evals out of the box. Teams use them. Engineers were spending afternoons grading traces and writing scoring functions and tagging failure modes, and the tools make this less painful than it would otherwise be.

What you get at the end of all that effort is a graded slice of traces. A few thousand examples marked good or bad, often with structured scores attached, sometimes with notes. This is useful - for catching regressions, for spotting drift, for arguing with your model provider about why their new release broke something. It is not, on its own, a training or eval dataset, and it is several steps short of a model that has actually improved.

LLMs are intelligently dumb, they will learn an approximation of what you teach them, as such you need to teach them exactly what you want them to do. A graded trace is an observation. A dataset is a curated, balanced, deduplicated set of examples organized around the behaviors you want the model to learn. The difference between them is curation logic, clustering, sampling strategy, and a clear view of what the model is supposed to get better at - none of which the observability tool produces.

And even once you have the dataset, the pipeline from there to a better agent has its own shape. You need to fine-tune a model or train a LORA adapter, eval-gate the result against a held-out slice, deploy the new weights behind your routing layer, and watch the next week of traces to see whether the change actually moved the agent's behavior or just shuffled the failure modes around. Each of those steps is its own piece of infrastructure.

## Bridging the gap

Some of the vendors have started shipping bolt-ons aimed at closing parts of this. Portkey's Autonomous Fine-Tuning wires gateway logs into a provider's training API and hands off the dataset. LangChain's Engine edits prompts and code from production traces. Braintrust's Loop generates evaluators and datasets from natural-language descriptions of failure modes. Each is a real move in the right direction, and each stops one step short of the model itself. The dataset gets handed off and the trail goes cold.

This is the gap between what's possible and what's shipped. The technology to take a production trace, score it, cluster it, curate it into a dataset, fine-tune a model on it, eval-gate the result and deploy it back through your routing layer - all of that exists, in pieces, sold by different companies with different pricing pages. What doesn't exist yet is a single layer that does the whole loop. You can probably see where i’m going with this but lets not jump ahead.

## The opportunity

Cursor is the example everyone reaches for, and for good reason. They trained their Composer model on production agent traces using online reinforcement learning, and it now [beats the frontier at its size](https://cursor.com/blog/composer-2-5) on cost and performance. The model that ships to Cursor users is the model trained on what Cursor users actually do. The labeling encodes the judgment. The fine-tune encodes the labeling. The product encodes the fine-tune. It compounds to create a very powerful product.

Doing this used to require a research team and the patience of an academic institution. It doesn't anymore. Open-weights bases - Llama, Qwen, Mistral, DeepSeek - are now genuinely competitive on scoped tasks. A LoRA fine-tune on a few thousand labeled trajectories runs for low hundreds of dollars on Modal or Together or your own GPUs if you're feeling brave. Unsloth and Tinker have made the mechanics broadly accessible to anyone willing to spend a weekend with them, with the small caveat that you do still currently have to be a bit of a genius to use them properly.

What this adds up to is something most teams haven't fully reckoned with yet. The data is sitting in the gateway. The traces are sitting in the observability tool. The curation can be automated. The fine-tuning is cheap. The deployment is a config change. The only thing standing between any reasonably-sized team and a continuously-improving fleet of specialist models trained on their own production traffic is whether anyone has joined the pieces up.

## Overmind

And we arrive at the aforementioned point. We built Overmind on the same bet we wrote about in [**_Think Smaller_**](https://www.overmindlab.ai/research/think-smaller): the teams who win the next phase of agentic AI will be the ones who own their models, not the ones who rent the most powerful ones, and certainly not the ones who own the most expensive plumbing in front of those rented models.

Overmind is the layer that turns the data your gateway and observability stack are already collecting into models that improve because of it. It plugs into the trace store you've already got, runs the curation, builds the dataset, fine-tunes the model, eval-gates the result and deploys the new weights behind your existing routing layer. The gateway keeps doing what it's good at. The observability stack keeps doing what it's good at. Overmind does the bit in between that nobody else is doing.

Keep your gateway. Keep your observability stack. Just stop pretending they're enough on their own.

__

**FAQ**

> **What's the difference between an LLM gateway and LLM observability?** Gateways route and rate-limit calls between your app and model providers. Observability tools record traces and let you inspect them. Neither curates the data or trains models.  
> **Why isn't observability enough for production agents?** Observability shows you what happened. It doesn't tell you whether the trajectory was good, and it doesn't turn that judgment into model improvements. That gap is the bottleneck.  
> **What does Overmind do that observability tools don't?** Overmind sits on top of your trace store, curates the data into training-grade datasets, fine-tunes specialist models, evaluates them, and deploys the new weights behind your existing routing layer.



_Tyler Edwards is Co-Founder and CEO of Overmind. He writes about agent infrastructure, fine-tuning, and what it takes to ship AI that actually improves in production. [Connect on LinkedIn](https://www.linkedin.com/in/tyler-edwards-b393a5352/)._

## Related research

- [My summer at Overmind](https://www.overmindlab.ai/research/my-summer-at-overmind): A summer internship reflection on autonomy, hard problems, and building across the Overmind product.
- [Small is the new frontier](https://www.overmindlab.ai/research/small-is-the-new-frontier): Why specialized models trained on your own production data are becoming the practical path to owned intelligence.
- [When bigger isn’t better](https://www.overmindlab.ai/research/when-bigger-isnt-better): On specialized tasks, Overmind-trained small language models outperform frontier models on accuracy, hallucination, and cost.

---

# https://www.overmindlab.ai/research/secret-agent-overmind

# Secret agent Overmind

> Canonical: https://www.overmindlab.ai/research/secret-agent-overmind

- Author: Tyler Edwards
- Category: Product
- Tags: Agents
- Published: 2026-06-15
- Updated: 2026-08-07

> Your agent is the most documented system in your engineering org and yet you can't answer simple questions.

![Secret agent Overmind](https://cdn.sanity.io/images/gv4t9qa3/production/07294abf3d79884daa73c1cc6329c7d29d1bea4f-2160x1080.jpg)

No, Christopher McQuarrie has not asked me to direct the next instalment of Mission: Impossible, though I am keeping a keen eye on my inbox. Today, we are talking state secrets!

That is also not true. But I did spend many years building stuff and things (the OFFICIAL term) for British Intelligence, and the single most valuable thing I took away was not state secrets. Instead, it was a deeply troubling obsession with data.

“Is this person a credible threat?” is the canonical example, and a very difficult one to answer. Intelligence analysts are basically superhumans, capable of finding links between vast amounts of varied data, and one of the ways they do that is by layering context.

## Data, data, everywhere.

That data is vast: signals intelligence, human intelligence, financial records, travel manifests, satellite imagery, and a mountain of open-source material. Some of it is a tidy database, some of it is a photograph, and some of it is words scrawled on a piece of paper. The scale is hard to picture: by the NSA’s own figures, it “touches” around 29 petabytes of data a day, which is equivalent to the entire Library of Congress being inspected nearly 3,000 times over, daily.

I hear you shouting, “Well Tyler, big tech companies have access to lots of data, so why is MI5 so special?” Beyond the legal implications and the fact that we don’t (yet) live in a Blade Runner-style dystopia, the reasons are structural. Broadly speaking, any one organization only ever holds a single thin slice of the picture: what you searched, what you spent, or where you travelled. You can know someone’s entire spending history in perfect detail and still have no idea what any of it means. An intelligence agency exists precisely to sit above all those slices and join them, which is the part almost nobody else is mandated, or built, to do. The fusion, not the volume, is the whole point.

This is the core function of an intelligence agency: not simply to collect the data, but to surface the hidden correlations in vast pools of information. Which brings us, as everything eventually does, to AI agents.

## Context really is king

Your agent is the most thoroughly documented system in your engineering org. Every prompt, every completion, every tool call, every retry is captured somewhere: traces in your observability tool, scores in your eval suite, rows in your dataset. And yet you still can’t answer simple questions. Why did it do that? Where did the behavior come from? Who gave it a gun? The records are all there, sitting in separate systems, and nobody joins them up.

So we did exactly that. We built a context layer that joins all these things together: one grounded picture of how your agent is put together and what it actually does, assembled from your codebase and traces, then turned loose to power Overmind, our AI engineering copilot.

We start at the codebase, because that’s where the ground truth lives. Before Overmind touches a single trace, it reads the code the agent runs on and compiles a Capability Card: the tools the agent can call and their real contracts, its input and output schema, the prompts, and the expected output. That’s the agent at its raw logical level, taken from the source rather than inferred from behavior. Skip it, as most tooling does, and a trace is just a behavior with nothing behind it, leaving you to reverse-engineer intent from whatever the trace happened to capture.

Then we bring in the data and the traces. Every record of what the agent did is a context source: production traces, dataset rows, eval scores, all built into the same shape. For a dataset, that layer is the spine: format, schema, per-column distribution, label space, cohorts, hygiene problems, derived straight from the data.

![The Overmind console showing the PR Approval Agent (StampHog), discovered from the posthog/posthog repository. The page lists the agent's prompt, its ci, github and security tags, and the seven models it calls, including Claude Sonnet 4.6, GPT 4.1 Mini, o3, Gemini 2.5 Flash Lite and Claude Haiku 4.5. A score panel reads 50 percent, with 1 of 2 runs traced and a badge that says eval metrics not approved. Below, a flow graph links Input, Agent and Read nodes to conversation_summarizer, title_generator, session_summarizer and llm_traces_summarizer, ending at Output.](https://cdn.sanity.io/images/gv4t9qa3/production/9e04c36e92ade22c26b4406a086bbb64ab9dfaff-3024x1726.png)

*Overmind's analysis of a PostHog Agent*

The codebase context is what makes that data useful. With the Capability Card in hand, Overmind knows the full surface of the agent and can measure the data against it: how much of that surface the data actually covers, which behaviors are well represented, which tools never appear, and which paths the agent takes in production that no dataset row touches. A pile of traces tells you what happened. The codebase tells you what could happen, and the gap between the two is the thing you need to see. That delta is what decides whether the data is good enough to evaluate against or train on, or whether you’re about to write evals for a fraction of the agent or fine-tune on the same narrow slice of its total behaviors.

Codebase tied to data tied to traces is the working surface for everything else we do. The data preprocessing reads from it, the evals are written against it, the fine-tuning sets are curated out of it, and the agent optimization is steered by it. That’s the line between a tool that can tell you a behavior was wrong and one that can tell you why, as well as how to fix it.

## So, what do you know about your agent?

So much, maybe too much. But not in a useful way.

That is what Overmind is built for: not more raw evidence, but the relationships between it. Code, traces, datasets, evals, and training runs joined into one grounded view of the agent, so you can see what is really happening and where the gaps are._  
  
Tyler Edwards is Co-Founder and CEO of Overmind. He writes about agent infrastructure, fine-tuning, and what it takes to ship AI that actually improves in production. [Connect on LinkedIn](https://www.linkedin.com/in/tyler-edwards-b393a5352/)._

## Related research

- [My summer at Overmind](https://www.overmindlab.ai/research/my-summer-at-overmind): A summer internship reflection on autonomy, hard problems, and building across the Overmind product.
- [Small is the new frontier](https://www.overmindlab.ai/research/small-is-the-new-frontier): Why specialized models trained on your own production data are becoming the practical path to owned intelligence.
- [When bigger isn’t better](https://www.overmindlab.ai/research/when-bigger-isnt-better): On specialized tasks, Overmind-trained small language models outperform frontier models on accuracy, hallucination, and cost.

---

# https://www.overmindlab.ai/research/think-smaller

# Think smaller

> Canonical: https://www.overmindlab.ai/research/think-smaller

- Author: Tyler Edwards
- Category: Product
- Tags: Models
- Published: 2026-03-11
- Updated: 2026-10-14

> Frontier models are overkill for most agent tasks. Specialized small language models beat them on cost, latency, and accuracy.

![Think smaller](https://cdn.sanity.io/images/gv4t9qa3/production/d69a9f7a7d26de80aa222af3300038f4d994979e-1600x1600.webp)

The AI industry has spent three years in an arms race measured in billions of parameters, and somewhere along the way everyone agreed that the answer to every problem was a bigger model.  
  
Bigger clusters, bigger bills, bigger teams to manage the complexity. The implicit assumption is that general intelligence at scale is the end state and everything else is just waiting to catch up.

Which is obviously great for the foundational teams pushing this agenda, were it not for the fact that the overwhelming majority of problems we throw AI at don’t need otherworldly intelligence and infinite compute to solve.

Llama, Phi, Qwen, Mistral: models that would have been considered frontier-grade two years ago are now freely available, fine-tuneable, and deployable on hardware you control. The performance gap that once justified the cost and lock-in of proprietary giants is now much slimmer, so the question is no longer “which frontier model should we use?” but rather “do we need a frontier model at all?”

For most tasks, the honest answer is no.

Before you train anything, check whether a better prompt gets you there: [prompt engineering vs fine-tuning](https://www.overmindlab.ai/research/prompt-engineering-vs-fine-tuning).

## The flamethrower problem

Using a frontier model on every task is like using a flamethrower to light a candle. Technically works, but your eyebrows and your budget are gone.

The majority of subtasks in a deployed agentic system are repetitive, scoped, and non-conversational. Extract this field, classify this ticket, format this output, none of these require a model trained on the sum of human knowledge. They require a model that is fast, predictable, cheap to run, and tuned to behave exactly as expected in a well-defined context.

In fact, on well-defined, repetitive tasks, specialized SLMs do not just cost a fraction of what frontier models cost, they outperform them. A model built specifically for one thing, trained on real examples of that thing, is better at that thing than a model trained to do everything.

And this holds even as the tasks get harder. Qwen2.5-Math-7B, running Microsoft’s rStar-Math reasoning framework, scores 90% on the MATH benchmark, beating OpenAI’s o1-preview. John Snow Labs’ MedS, an 8B medical model, was preferred over GPT-4o by practising physicians in blind evaluation across every dimension: factuality, clinical relevance, and conciseness. LlaSMol, a 7B chemistry model, achieves 93% exact match on molecular prediction tasks where GPT-4 scores under 5%. On Berkeley’s function-calling leaderboard, the top spot is held by an 8B model beating GPT-4o and Claude 3.5 Sonnet.

## What does open-weights actually unlock for production AI?

The capability trajectory of small open models has been striking, yet the capability of these models is not the most exciting piece, what open weights actually unlock is control.

A frontier model you do not own, cannot audit, and cannot modify is a dependency, not infrastructure and certainly not your ‘technical moat’. When it changes, your system changes. When it halts, your system halts. When the provider decides to deprecate a version or reprice the API, you absorb it. The case for specialized small models is not just that they are better. It is that they are yours.

The physical dimension matters too. Frontier models run in data centers, behind APIs, with latency measured in hundreds of milliseconds and costs that compound with every token. In an agentic system where a single workflow might invoke a model thirty or forty times, that compounds fast. Serving a 7B SLM is roughly 10 to 30x cheaper in latency, energy, and compute than serving a 70 to 175B LLM. Small models can run on-premise, inside the security perimeter, on the device generating the data. A fraud detection model running locally, fine-tuned on that institution’s own transaction patterns, makes faster decisions with no external dependency and no data crossing a network boundary. The performance is better, the risk profile and compliance conversation are entirely different.

Even more exciting still (if you can believe it) is that a model small enough to run on a phone can be fine-tuned not just to a specific workflow but to a specific person. An agent that lives on your device and learns from how you actually work is a categorically different thing to one calling to a data center every time you ask it something. We are moving from general intelligence in the cloud to personal intelligence in your pocket. That shift is bigger than most people are currently treating it.

## Production is the best training data you have

Every invocation of a model in an agentic workflow is a natural source of high-quality training data. The prompts are well-defined, the expected outputs are narrow, and whether the workflow succeeded or failed is a clean signal. A listener at the model call interface, logging inputs, outputs, and downstream outcomes, accumulates exactly the dataset you need to continuously fine-tune a specialist model for that task, as the system runs. The production environment is generating its own improvement data in real time.

This is where the SLM case becomes compelling beyond economics. You instrument the calls, cluster the patterns, fine-tune the specialists, and the system gets better with every run. What determines whether you actually close that loop is whether you have the infrastructure to capture it, label it, and act on it.

AI security has become a pretty bloated term, so stick with me here. The version of it that actually matters is not about firewalls or prompt injection. It is about knowing what your model is doing, and knowing when it is doing something it should not. Specialized models make that question answerable. When a model has a narrow, well-defined scope, you know what it should be doing, which means you know when it is doing something else. The output distribution is tighter, the failure modes are more predictable, and the feedback loop from production behavior back to model improvement is shorter and more reliable.

General models are a different problem entirely. The failure surface is vast, constantly shifting, and practically impossible to enumerate in advance. Guardrails are the industry’s current answer, which is to say: a list of rules written by humans trying to statically account for an infinite and ever-changing set of ways a frontier model can go wrong.

## Are small specialist models harder to train?

Now I would love to just dunk on frontier models for the rest of this post, but there is a real counter-argument worth addressing. Calling an API is easy. You write a prompt, you get an answer, you iterate, you ship. The feedback loop is fast and the overhead is minimal. Building SLM-first infrastructure is a different proposition entirely.

Fine-tuning a specialist model requires training data, which requires instrumentation, which requires someone to own it. You need pipelines to capture production traces, processes to curate and label them, and infrastructure to run fine-tuning jobs on a cadence that actually moves the needle. You need model versioning, evaluation harnesses, and a deployment setup that lets you swap models without taking down the system. None of this is impossibly hard, but none of it is free either.

## Why this matters to us

Everything in this post is, admittedly, not a neutral observation. [We built Overmind](https://www.overmindlab.ai/about) on a specific bet: that the teams who win the next phase of agentic AI will be the ones who own their models, not the ones who rent the most powerful ones.

The [infrastructure to do that](https://www.overmindlab.ai/product), fine-tuning pipelines, production observability, continuous improvement loops that do not require a dedicated ML team to run, that is what we are building. Not just because it is an interesting technical problem, but because without it the agentic future we are all so excited about, the one where AI runs diagnostics in hospitals or runs locally on your phone and adapts to you as an individual, simply cannot happen.

So we are building that future.

_Overmind builds real-time supervision and fine-tuning infrastructure for autonomous AI systems. If you are [deploying agentic AI](https://docs.overmindlab.ai/),_ _get in touch_

## _  
_FAQ

**Are small language models better than frontier models?**  
For repetitive, scoped tasks they are. A model trained specifically for one job, on real examples of that job, outperforms a generalist on cost, latency and accuracy. Frontier models still win on open-ended reasoning and conversation.

**What is a specialist SLM?**   
A small language model (typically 1B to 8B parameters) fine-tuned on a narrow task using your own production data. Llama, Phi, Qwen and Mistral are common open-weight bases.

**How much cheaper is a 7B SLM compared to a 70B+ LLM?**   
Roughly 10 to 30x cheaper in latency, energy and compute when self-hosted, and the gap widens at high request volumes.

**Can I run a specialist SLM on-device?**   
Yes. 7B and 8B models run on modern phones and laptops, which enables per-user fine-tuning, offline operation, and no data leaving the device.



_Tyler Edwards is Co-Founder and CEO of Overmind. He writes about agent infrastructure, fine-tuning, and what it takes to ship AI that actually improves in production. [Connect on LinkedIn](https://www.linkedin.com/in/tyler-edwards-b393a5352/)._

## Related research

- [My summer at Overmind](https://www.overmindlab.ai/research/my-summer-at-overmind): A summer internship reflection on autonomy, hard problems, and building across the Overmind product.
- [Small is the new frontier](https://www.overmindlab.ai/research/small-is-the-new-frontier): Why specialized models trained on your own production data are becoming the practical path to owned intelligence.
- [When bigger isn’t better](https://www.overmindlab.ai/research/when-bigger-isnt-better): On specialized tasks, Overmind-trained small language models outperform frontier models on accuracy, hallucination, and cost.

---

# https://www.overmindlab.ai/about

# Our mission

Turn every company into a frontier lab

- [Read More](#cutscene)

> Canonical: https://www.overmindlab.ai/about

Overmind is a London team building the infrastructure that lets agentic AI teams own their models rather than rent the most expensive plumbing in front of them.

## The humans of Overmind

Individually brilliant, collectively unstoppable. Meet the minds behind our machine.

- **Tyler Edwards** — Co-Founder & CEO
- **Sam Brunt** — Co-Founder & COO
- **Freyam Mehta** — Founding engineer
- **Pritam Soni** — Founding engineer
- **Rohit Gupta** — Founding engineer
- **Sam Morris** — Founding GTM
- **Freyam Mehta** — Founding engineer
- **Pritam Soni** — Founding engineer
- **Rohit Gupta** — Founding engineer
- **Sam Morris** — Founding GTM

Backed by the best

## Help us shape a new era

At Overmind, we're building the infrastructure layer that autonomous AI is missing. If you want to work on hard problems that actually matter, we'd love to hear from you.

- [View Open Roles](https://www.overmindlab.ai/careers)

## Research on machines, written by humans

Exploits, In-depth write-ups, and game-breaking bugs explored by our team.

---

# https://www.overmindlab.ai/contact

# Contact

> Canonical: https://www.overmindlab.ai/contact

Talk to the team

Overmind is the model training platform for AI teams. Whether you want to trace an agent, curate a dataset from production traffic, train a model you own, or talk through an enterprise deployment, the fastest path is a short call with the people who build the platform. Email works too, and every reply comes from an engineer.

## Book a call

A 30-minute walkthrough of tracing, evals, training, and serving on your own agents. Bring a real workload if you can.

Book a call: https://www.overmindlab.ai/contact (opens the scheduling popup)

## Documentation and support

Quickstart, SDK references, the REST API, and inference docs. Support requests from the console reach the same engineers.

[Read the docs](https://docs.overmindlab.ai/)

## Console

Sign in to manage projects, API keys, traces, datasets, evals, training runs, and deployed models.

[Open the console](https://console.overmindlab.ai)

## Careers

Open engineering and research roles. If you want to work on training infrastructure for agents, we would like to hear from you.

[See open roles](https://www.overmindlab.ai/careers)

## Company

Overmind is a UK-based company. Registered company details and the postal address are published here as soon as they are set in the site settings; until then, email is the quickest way to reach us.

## Elsewhere

- [Discord](https://discord.gg/XGsbf49EHG)
- [Github](https://github.com/overmind-core/overmind)
- [Twitter (X)](https://x.com/overmindlab)
- [LinkedIn](https://www.linkedin.com/company/over-mind-ai/)
- [Hugging Face](https://huggingface.co/OvermindLab)

---

# https://www.overmindlab.ai/careers

# Careers at Overmind

Build what's next

- [See Openings](#section-a24d089da777)

> Canonical: https://www.overmindlab.ai/careers

Help make intelligence as easy to create as it is to consume. Explore careers at Overmind, including our founding engineer role and open application.

## Find your role

Help make intelligence as easy to create as it is to consume. Find your role, or choose your fighter and show us what you’d bring.

- [Choose your fighter](https://www.overmindlab.ai/careers/choose-your-fighter): Open application
- [Founding engineer](https://www.overmindlab.ai/careers/founding-engineer): Full time, London

Backed by the best

## The humans of Overmind

Individually brilliant, collectively unstoppable. Meet the minds behind our machine.

- **Tyler Edwards** — Co-Founder & CEO
- **Sam Brunt** — Co-Founder & COO
- **Freyam Mehta** — Founding engineer
- **Pritam Soni** — Founding engineer
- **Rohit Gupta** — Founding engineer
- **Sam Morris** — Founding GTM
- **Freyam Mehta** — Founding engineer
- **Pritam Soni** — Founding engineer
- **Rohit Gupta** — Founding engineer
- **Sam Morris** — Founding GTM

---

# https://www.overmindlab.ai/legals/privacy

# Privacy policy

> Canonical: https://www.overmindlab.ai/legals/privacy

Our rule set

*Last updated: 2026-10-03*

## Who we are

Overmind Technology Ltd. ("Overmind", "we", "us") provides an AI Security & Observability platform. We are registered in the UK. Contact: [support@overmindlab.ai](mailto:support@overmindlab.ai) Address: Walworth Town Hall 151 Walworth Road, Office 302, London, England, SE17 1RS

## Scope of this notice

- Account Data: This Notice applies to the personal data we collect about our business clients (administrators, buyers) and website visitors. Overmind is the Data Controller for this data.
- Customer Content: We process prompt inputs, model outputs and other content submitted through the platform on behalf of our Clients. The Client is the Data Controller and Overmind is the Data Processor. Our Terms of Service and applicable Data Processing Addendum govern this processing. The integration disclosures below explain how connected workflows use Customer Content.

## Data we collect (account data)

- Identity & Contact Data: Name, email, job title, and company name of account administrators.
- Billing Data: Payment information (processed via our secure payment provider; we do not store full card numbers).
- Technical Data: IP address, browser type, and usage data regarding how you interact with our dashboard (for security and product improvement).

## How we use account data

**We process Account Data to:**

- Register new customers (Performance of Contract).
- Process payments and manage subscriptions (Performance of Contract).
- Secure our platform and prevent fraud (Legitimate Interest).
- Notify you about changes to our service (Legal Obligation).

**  
Where we store data:**

- Our primary infrastructure is hosted on Amazon Web Services (AWS) in the Ireland (eu-west-1) region.
- For UK Clients: Transfers to the EEA (Ireland) are covered by the UK "Adequacy Regulations."
- For EU Clients: Data remains resident within the EEA.

## Data security

We utilize industry-standard encryption (AES-256 for data at rest, TLS 1.2+ for data in transit). Please review our [Trust Center](https://overmindlab.ai/) for detailed security architecture.

## Your rights

Under the UK GDPR, you have rights to access, correct, or delete your Account Data. To exercise these rights, please contact [support@overmindlab.ai](mailto:support@overmindlab.ai).

## Connected tools and integrations

When you connect Overmind to an agent client such as Codex or ChatGPT, the connection can access projects your Overmind account is authorized to use. Depending on your requests, the integration processes project and capability identifiers, traces and their inputs and outputs, dataset content, evaluation results, and training or inference requests and results. We use this information to perform the requested workflows, maintain access controls and operate the service.

## Recipients of integration data

Results requested through the integration are returned to the connected agent client. Cloud hosting and storage, authentication, operational monitoring and analytics providers support the service. Model, evaluation, training and inference providers may process workflow content when those workflows are used. These recipients depend on the connected services and selected workflow; their handling of data is also subject to their applicable terms and privacy notices.

## Customer-content retention and deletion

Section 5.5 of our Terms of Service states that Customer Data is retained during the Subscription Period and for 30 days thereafter unless an Order Form specifies otherwise. Upon termination or expiration, customers may request deletion; Customer Data will be deleted or anonymized within 90 days of termination, except where retention is required by law. Customers should export their data before termination. See https://www.overmindlab.ai/legals/terms.

Our current production application-log retention setting is 365 days. Automated production database backups have a rolling 7-day retention setting. These operational settings do not define the retention of manual snapshots, exported copies or third-party analytics.

## Connection controls

OAuth access tokens expire after one hour. The connection remains authorized through rotating refresh tokens until revoked. Revoking the connection prevents continued use of that authorization; it does not itself delete content already stored in Overmind or copies already returned to the connected client. Contact support@overmindlab.ai for data-access, correction or deletion requests.

---

# https://www.overmindlab.ai/legals/terms

# Terms of service

> Canonical: https://www.overmindlab.ai/legals/terms

*Last updated: 2026-03-01*

**OVERMIND TECHNOLOGY LIMITED**

## **1. INTRODUCTION**

1.1 These Terms of Service ("Terms") constitute a legally binding agreement between Overmind Technology Limited, a company incorporated in England and Wales with registered number 16585996 and registered office at Walworth Town Hall, 151 Walworth Road, Office 302, London, England, SE17 1RS ("Overmind," "we," "us," or "our") and you ("Customer," "you," or "your").

1.2 These Terms govern your access to and use of Overmind's AI Security and Observability Platform ("Platform" or "Services"), including but not limited to:

- Input protection and threat detection
- MCP Gateway controls
- Agentic observability and monitoring
- Adaptive intelligence and policy enforcement
- Browser AI security extensions
- Related software, documentation, and support services

1.3 By accessing or using our Services, you acknowledge that you have read, understood, and agree to be bound by these Terms. If you are entering into these Terms on behalf of an organization, you represent and warrant that you have the authority to bind that organization to these Terms.

## **2. DEFINITIONS**

2.1 "Authorized Users" means your employees, contractors, and agents who are authorized to use the Services under your subscription.

2.2 "Customer Data" means all data, content, and information submitted, uploaded, or processed by you or your Authorized Users through the Services.

2.3 "Documentation" means the technical documentation, user guides, and other materials provided by Overmind relating to the Services.

2.4 "Subscription Period" means the period during which you have purchased access to the Services, as specified in your Order Form.

2.5 "Order Form" means the ordering document, statement of work, or online order specifying the Services, pricing, and Subscription Period.

2.6 "AI Agents" means autonomous or semi-autonomous software systems that utilize large language models or other AI technologies to perform tasks, make decisions, or interact with tools and data sources.

2.7 "Security Incident" means any unauthorized access, use, disclosure, modification, or destruction of Customer Data or the Services.

### **3. SERVICE PROVISION**

3.1 **Access to Services**

- Subject to your compliance with these Terms and payment of applicable fees, Overmind grants you a non-exclusive, non-transferable, limited right to access and use the Services during the Subscription Period.
- You may only use the Services for your internal business purposes and in accordance with the Documentation.

3.2 **Service Levels**

- Overmind will use commercially reasonable efforts to make the Services available 99.5% of the time during each calendar month, excluding scheduled maintenance and circumstances beyond our reasonable control.
- Service level commitments and remedies, if any, are specified in your Order Form or separate Service Level Agreement.

3.3 **Updates and Modifications**

- Overmind may update, modify, or enhance the Services from time to time. We will provide reasonable notice of material changes that adversely affect your use of the Services.
- Overmind reserves the right to discontinue features or functionality with 90 days' prior written notice.

3.4 **Support Services**

- Technical support is provided in accordance with your subscription tier as specified in your Order Form.
- Support hours, response times, and channels vary by subscription level.  


### **4. CUSTOMER OBLIGATIONS**

4.1 **Account Security**

- You are responsible for maintaining the confidentiality of your account credentials and for all activities that occur under your account.
- You must notify Overmind immediately of any unauthorized use of your account or any other security breach.
- You must implement reasonable security measures for your Authorized Users' access to the Services.  


4.2 **Acceptable Use** You agree not to:

- Use the Services for any unlawful purpose or in violation of any applicable laws or regulations
- Attempt to gain unauthorized access to the Services or related systems
- Interfere with or disrupt the integrity or performance of the Services
- Reverse engineer, decompile, or disassemble any portion of the Services
- Use the Services to transmit malware, viruses, or other harmful code
- Remove, alter, or obscure any proprietary notices on the Services
- Use the Services to compete with Overmind or develop competing products
- Exceed usage limits specified in your subscription tier  


4.3 **Compliance**

- You are responsible for ensuring that your use of the Services and your Customer Data comply with all applicable laws, regulations, and industry standards.
- You must obtain all necessary consents and permissions for the processing of data through the Services.
- You acknowledge that you are responsible for configuring the Services appropriately for your compliance requirements.  


4.4 **AI Agent Configuration**

- You are responsible for the configuration, deployment, and behavior of any AI Agents you monitor or secure using the Services.
- You must ensure that your AI Agents comply with applicable laws and do not violate third-party rights.
- Overmind is not responsible for the actions, outputs, or decisions made by your AI Agents.  


### **5. DATA PROTECTION AND SECURITY**

5.1 **Data Processing**

- Overmind processes Customer Data solely to provide the Services and as instructed by you in accordance with these Terms.
- The parties' data protection obligations are set forth in the Data Processing Addendum ("DPA"), which is incorporated into these Terms by reference.
- Overmind complies with applicable data protection laws, including the UK GDPR and Data Protection Act 2018.

5.2 **Security Measures**

- Overmind implements and maintains appropriate technical and organizational security measures to protect Customer Data, including:
   - Encryption of data in transit and at rest
   - Access controls and authentication mechanisms
   - Regular security assessments and penetration testing
   - Security monitoring and incident response procedures
   - ISO 27001 certified information security management system

5.3 **Security Incidents**

- Overmind will notify you without undue delay upon becoming aware of a Security Incident affecting your Customer Data.
- Overmind will take reasonable steps to investigate, remediate, and mitigate Security Incidents.
- You acknowledge that notification may be delayed if required by law enforcement or to prevent further harm.

5.4 **Data Location and Transfers**

- Customer Data is processed and stored in data centers located in the United Kingdom and European Union.
- If Customer Data is transferred outside the UK or EU, Overmind will ensure appropriate safeguards are in place, including Standard Contractual Clauses where required.

5.5 **Data Retention and Deletion**

- Overmind retains Customer Data during the Subscription Period and for 30 days thereafter, unless otherwise specified in your Order Form.
- Upon termination or expiration of your subscription, you may request deletion of your Customer Data. Overmind will delete or anonymize Customer Data within 90 days of termination, except where retention is required by law.
- You are responsible for exporting your Customer Data before termination.  


### **6. INTELLECTUAL PROPERTY RIGHTS**

6.1 **Overmind IP**

- Overmind retains all right, title, and interest in and to the Services, including all software, technology, Documentation, and any modifications or derivative works thereof.
- These Terms do not grant you any intellectual property rights in the Services except as expressly stated.

6.2 **Customer Data**

- You retain all right, title, and interest in and to your Customer Data.
- You grant Overmind a limited license to use, process, and store Customer Data solely to provide the Services.

6.3 **Feedback**

- If you provide feedback, suggestions, or recommendations regarding the Services ("Feedback"), Overmind may use such Feedback without obligation or compensation to you.
- You grant Overmind a perpetual, irrevocable, worldwide, royalty-free license to use, incorporate, and exploit Feedback in any manner.

6.4 **Aggregated Data**

- Overmind may collect and analyze aggregated, anonymized data derived from use of the Services for purposes of improving the Services, developing new features, and industry research.
- Such aggregated data does not identify you or any individual and is owned by Overmind.  


### **7. FEES AND PAYMENT**

7.1 **Subscription Fees**

- You agree to pay all fees specified in your Order Form in accordance with the payment terms stated therein.
- Fees are based on your selected subscription tier and may include base platform fees, per-agent monitoring fees, and usage-based charges.

7.2 **Payment Terms**

- Unless otherwise specified, fees are payable annually in advance.
- All fees are non-refundable except as expressly provided in these Terms.
- Overmind will invoice you in accordance with your Order Form, and payment is due within 30 days of the invoice date.

7.3 **Taxes**

- All fees are exclusive of taxes, duties, and similar governmental charges.
- You are responsible for paying all applicable taxes associated with your use of the Services, except for taxes based on Overmind's income.

7.4 **Late Payment**

- If payment is not received when due, Overmind may charge interest at the rate of 1.5% per month (or the maximum rate permitted by law, whichever is less) on overdue amounts.
- Overmind may suspend your access to the Services if payment is more than 30 days overdue, upon 10 days' written notice.

7.5 **Fee Changes**

- Overmind may increase fees upon renewal of your Subscription Period by providing at least 60 days' prior written notice.
- Fee increases during the current Subscription Period require your written consent.  


### **8. TERM AND TERMINATION**

8.1 **Term**

- These Terms commence on the date you first access the Services and continue for the Subscription Period specified in your Order Form.
- Subscriptions automatically renew for successive periods equal to the original Subscription Period unless either party provides written notice of non-renewal at least 30 days before the end of the then-current period.

8.2 **Termination for Convenience**

- You may terminate these Terms at the end of the then-current Subscription Period by providing written notice at least 30 days before renewal.
- Termination for convenience does not entitle you to a refund of prepaid fees.

8.3 **Termination for Cause** Either party may terminate these Terms immediately upon written notice if:

- The other party materially breaches these Terms and fails to cure such breach within 30 days of receiving written notice
- The other party becomes insolvent, enters bankruptcy proceedings, or ceases business operations

8.4 **Suspension**

- Overmind may suspend your access to the Services immediately without notice if:
   - Your use of the Services poses a security risk or threatens the stability of the Services
   - You are in breach of Section 4.2 (Acceptable Use)
   - Your account is more than 30 days overdue on payment
- Overmind will provide notice of suspension and an opportunity to cure where reasonably practicable.

8.5 **Effect of Termination** Upon termination or expiration:

- Your right to access and use the Services immediately terminates
- You must cease all use of the Services and delete any copies of Overmind's Confidential Information
- Outstanding fees become immediately due and payable
- You may request export of your Customer Data within 30 days of termination
- Sections 6 (Intellectual Property), 7 (Fees), 9 (Confidentiality), 10 (Warranties), 11 (Limitation of Liability), and 13 (General Provisions) survive termination  


### **9. CONFIDENTIALITY**

9.1 **Definition** "Confidential Information" means all non-public information disclosed by one party to the other, including technical, business, financial, and customer information, that is marked as confidential or would reasonably be considered confidential.

9.2 **Obligations**

- Each party agrees to protect the other party's Confidential Information using the same degree of care it uses to protect its own confidential information, but no less than reasonable care.
- Neither party may use the other party's Confidential Information except as necessary to fulfill its obligations or exercise its rights under these Terms.
- Neither party may disclose the other party's Confidential Information to third parties except to employees, contractors, and advisors who have a need to know and are bound by confidentiality obligations.

9.3 **Exceptions** Confidential Information does not include information that:

- Is or becomes publicly available through no breach of these Terms
- Was rightfully known prior to disclosure
- Is rightfully received from a third party without breach of confidentiality obligations
- Is independently developed without use of the disclosing party's Confidential Information
- Must be disclosed pursuant to law or court order (provided the receiving party gives reasonable notice to allow the disclosing party to seek protective measures)

9.4 **Duration**

- The confidentiality obligations continue for three years after the termination of these Terms or, for trade secrets, for as long as the information remains a trade secret under applicable law.  


### **10. WARRANTIES AND DISCLAIMERS**

10.1 **Mutual Warranties** Each party represents and warrants that:

- It has the legal power and authority to enter into these Terms
- These Terms constitute a valid and binding obligation
- Its performance will not violate any agreement or obligation to third parties

10.2 **Overmind Warranties** Overmind warrants that:

- The Services will perform materially in accordance with the Documentation
- Overmind will use industry-standard measures to prevent the introduction of viruses or malicious code into the Services
- Overmind will comply with applicable laws in providing the Services

10.3 **Customer Warranties** You warrant that:

- You have all necessary rights and consents to provide Customer Data to Overmind for processing
- Your use of the Services and Customer Data will comply with applicable laws
- You will not use the Services in any manner that infringes third-party rights

10.4 **Disclaimer** EXCEPT AS EXPRESSLY PROVIDED IN THESE TERMS, THE SERVICES ARE PROVIDED "AS IS" AND "AS AVAILABLE." TO THE MAXIMUM EXTENT PERMITTED BY LAW, OVERMIND DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE, AND NON-INFRINGEMENT.

OVERMIND DOES NOT WARRANT THAT:

- The Services will meet your specific requirements
- The Services will be uninterrupted, timely, secure, or error-free
- The results obtained from the Services will be accurate or reliable
- Any errors in the Services will be corrected

10.5 **AI Technology Disclaimer**

- You acknowledge that AI technology, including the adaptive intelligence features of the Services, may produce outputs that are unpredictable, incomplete, or inaccurate.
- Overmind does not warrant the accuracy, completeness, or reliability of AI-generated threat detections, policy recommendations, or security assessments.
- You are responsible for independently verifying and validating security decisions based on the Services.  


### **11. LIMITATION OF LIABILITY**

11.1 **Exclusion of Damages** TO THE MAXIMUM EXTENT PERMITTED BY LAW, NEITHER PARTY WILL BE LIABLE FOR ANY INDIRECT, INCIDENTAL, SPECIAL, CONSEQUENTIAL, OR EXEMPLARY DAMAGES, INCLUDING DAMAGES FOR LOSS OF PROFITS, REVENUE, DATA, USE, GOODWILL, OR OTHER INTANGIBLE LOSSES, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

11.2 **Liability Cap** EXCEPT FOR EXCLUDED CLAIMS (AS DEFINED BELOW), EACH PARTY'S TOTAL CUMULATIVE LIABILITY ARISING OUT OF OR RELATED TO THESE TERMS WILL NOT EXCEED THE AMOUNTS PAID OR PAYABLE BY YOU TO OVERMIND IN THE TWELVE (12) MONTHS PRECEDING THE EVENT GIVING RISE TO LIABILITY.

11.3 **Excluded Claims** The limitations in Sections 11.1 and 11.2 do not apply to:

- Your payment obligations
- Either party's breach of confidentiality obligations
- Either party's indemnification obligations
- Your breach of intellectual property rights
- Liability that cannot be excluded or limited under applicable law
- Death or personal injury caused by negligence
- Fraud or fraudulent misrepresentation

11.4 **AI Agent Liability**

- You acknowledge that Overmind provides security and observability tools for AI Agents but does not control the actions, outputs, or decisions of your AI Agents.
- Overmind is not liable for any damages arising from the behavior, outputs, or decisions of AI Agents you deploy or monitor using the Services.
- You are solely responsible for the actions and outputs of your AI Agents and any consequences thereof.

11.5 **Basis of the Bargain** The parties acknowledge that the limitations of liability in this Section 11 are fundamental elements of the basis of the bargain and that Overmind would not provide the Services without such limitations.

### **12. INDEMNIFICATION**

12.1 **Customer Indemnification** You will indemnify, defend, and hold harmless Overmind and its officers, directors, employees, agents, and affiliates from and against any claims, liabilities, damages, losses, costs, and expenses (including reasonable attorneys' fees) arising out of or relating to:

- Your use of the Services in breach of these Terms
- Your Customer Data, including any claim that Customer Data infringes third-party rights
- Your AI Agents' actions, outputs, or decisions
- Your violation of applicable laws or regulations
- Your negligence or willful misconduct

12.2 **Overmind Indemnification** Overmind will indemnify, defend, and hold harmless you and your officers, directors, employees, and agents from and against any claims by third parties that the Services, when used in accordance with these Terms, infringe such third party's intellectual property rights.

12.3 **Indemnification Process**

- The indemnified party must promptly notify the indemnifying party of any claim
- The indemnifying party has sole control of the defense and settlement, provided that it may not settle any claim without the indemnified party's consent if such settlement imposes obligations on or admits liability of the indemnified party
- The indemnified party will provide reasonable cooperation in the defense
- The indemnified party may participate in the defense with its own counsel at its own expense

12.4 **Remedies** If the Services become, or in Overmind's opinion are likely to become, subject to an infringement claim, Overmind may, at its option and expense:

- Procure the right for you to continue using the Services
- Modify the Services to be non-infringing without material loss of functionality
- Replace the Services with non-infringing equivalent functionality
- If none of the above are commercially reasonable, terminate these Terms and refund prepaid fees for the unused portion of the Subscription Period  


### **13. GENERAL PROVISIONS**

13.1 **Governing Law** These Terms are governed by and construed in accordance with the laws of England and Wales, without regard to conflict of law principles.

13.2 **Jurisdiction** The parties irrevocably submit to the exclusive jurisdiction of the courts of England and Wales for any dispute arising out of or relating to these Terms.

13.3 **Entire Agreement** These Terms, together with any Order Forms, DPA, and other documents expressly incorporated by reference, constitute the entire agreement between the parties regarding the subject matter and supersede all prior agreements, understandings, and communications, whether written or oral.

13.4 **Amendments**

- Overmind may update these Terms from time to time by posting the updated Terms on its website.
- Material changes will be notified to you via email or through the Services at least 30 days before taking effect.
- Your continued use of the Services after such notice constitutes acceptance of the updated Terms.
- If you do not agree to the updated Terms, you may terminate these Terms in accordance with Section 8.2.

13.5 **Waiver** No failure or delay by either party in exercising any right under these Terms will constitute a waiver of that right. No waiver will be effective unless in writing and signed by an authorized representative of the waiving party.

13.6 **Severability** If any provision of these Terms is held to be invalid, illegal, or unenforceable, the remaining provisions will remain in full force and effect, and the invalid provision will be modified to the minimum extent necessary to make it valid and enforceable.

13.7 **Assignment**

- You may not assign or transfer these Terms or any rights or obligations hereunder without Overmind's prior written consent.
- Overmind may assign these Terms in connection with a merger, acquisition, or sale of all or substantially all of its assets upon notice to you.
- Any attempted assignment in violation of this Section is void.

13.8 **Independent Contractors** The parties are independent contractors. These Terms do not create a partnership, franchise, joint venture, agency, fiduciary, or employment relationship between the parties.

13.9 **Force Majeure** Neither party will be liable for any failure or delay in performance due to causes beyond its reasonable control, including acts of God, natural disasters, war, terrorism, labor disputes, or government actions (excluding payment obligations).

13.10 **Notices**

- All notices under these Terms must be in writing and delivered to the addresses specified in the Order Form or, if none, to Overmind's registered office.
- Notices may be delivered by email, registered mail, or recognized courier service.
- Notices are deemed received: (a) when delivered if by email (with confirmation of delivery); (b) five business days after mailing if by registered mail; or (c) when delivered if by courier.
- Notices to Overmind should be sent to: legal@overmindlab.ai

13.11 **Export Compliance** You agree to comply with all applicable export and import control laws and regulations in your use of the Services. You represent that you are not located in, under the control of, or a national or resident of any country subject to comprehensive sanctions.

13.12 **Third-Party Components** The Services may include or incorporate third-party software components subject to separate license terms. Such third-party licenses are available upon request.

13.13 **Publicity**

- Unless otherwise agreed in writing, Overmind may identify you as a customer and use your name and logo in Overmind's marketing materials, website, and customer lists.
- You may request that Overmind cease such use by providing written notice to Overmind.

13.14 **Survival** The following sections survive termination or expiration of these Terms: Sections 5.5 (Data Retention), 6 (Intellectual Property), 7 (Fees for services rendered prior to termination), 9 (Confidentiality), 10.4-10.5 (Disclaimers), 11 (Limitation of Liability), 12 (Indemnification), and 13 (General Provisions).

### **14. COMPLIANCE AND REGULATORY**

14.1 **Financial Services Compliance**

- If you operate in a regulated financial services environment, you acknowledge that you are responsible for ensuring your use of the Services complies with applicable regulations, including FCA requirements, MiFID II, PSD2, and other financial services regulations.
- Overmind will provide reasonable assistance in your compliance efforts, including audit trail functionality and compliance reporting features.

14.2 **AI-Specific Regulations**

- You acknowledge that the regulatory landscape for AI systems is evolving and that you are responsible for monitoring and complying with applicable AI-specific regulations, including the EU AI Act and UK AI regulations.
- Overmind will use commercially reasonable efforts to update the Services to support compliance with major AI regulations as they come into force.

14.3 **Audit Rights**

- Upon reasonable notice (at least 30 days), you may audit Overmind's compliance with its security and data protection obligations under these Terms, provided such audit: (a) occurs no more than once per year; (b) is conducted during business hours; (c) does not unreasonably interfere with Overmind's operations; and (d) is subject to reasonable confidentiality obligations.
- In lieu of an audit, Overmind may provide relevant certifications (ISO 27001, SOC 2, etc.) and assessment reports.  


### **15. DEFINITIONS AND INTERPRETATION**

15.1 **Interpretation**

- Headings are for convenience only and do not affect interpretation
- References to "including" mean "including without limitation"
- References to Sections are to sections of these Terms
- Words in the singular include the plural and vice versa
- References to a party include its permitted successors and assigns

15.2 **Order of Precedence** In the event of conflict between documents, the following order of precedence applies:

1. Order Form
2. Data Processing Addendum
3. These Terms
4. Documentation  


### **16. CONTACT INFORMATION**

For questions about these Terms or the Services, please contact:

  
**Overmind Technology Limited** Walworth Town Hall 151 Walworth Road, Office 302 London, England SE17 1RS

Email: legal@overmindlab.ai Website: www.overmindlab.ai

  
**ACCEPTANCE**

By accessing or using the Services, you acknowledge that you have read, understood, and agree to be bound by these Terms of Service.