Agent testing
Prove every change before you ship it
Generate evaluations from your agent’s context, compare models and prompts on the same data, and test code changes in your own repository. Inspect the scores, evidence and winning diffs before deciding what to ship.
Measure the result. Inspect the change.
Generated evaluators and reproducible experiments connect production failures to changes in your code.
Evaluators grounded in the capability
Overmind generates checks from the capability’s purpose, prompt, tools and available trace evidence. Inspect the generated library, edit your own evaluators, and organize them into eval sets. Generative testing and live trace scoring use the roles appropriate to each evaluator.
Compare on the same evidence
Evaluate recorded outputs or generate fresh responses from model and prompt variants. Each run pins its dataset version and evaluator snapshots. Compare against a baseline, inspect per-metric scores and judge reasoning, and add human annotations to samples that need review.
Improve prompts, tools and code
Run /overmind optimize from your agent’s repository. Your coding agent writes candidate diffs and replays them locally; Overmind scores the results against your chosen eval set. Inspect iteration scores and candidate patches, then apply the winning change through your normal code review.
Test the model before you change it
Use /overmind backtest to replay the same dataset through candidate models while keeping the agent’s prompts and logic fixed. Compare quality alongside latency and cost, and inspect the samples behind the averages before choosing the next model or training experiment.
What you get
Evaluation
Reusable evaluator library
Combine LLM judges and deterministic checks in capability-scoped or project-wide eval sets.
Reproducible comparisons
Pin data, prompts and evaluators so results remain tied to what the experiment actually tested.
Inspectable results
Read score rationales and sample outputs, with incomplete generations distinguished from scored results.
Optimization
Repository-level changes
Test edits to prompts, tool definitions and application logic in the environment where your agent runs.
Model backtesting
Compare models over a shared dataset and evaluation set with your existing harness.
Bounded experiments
Choose iteration and candidate budgets, follow progress, and inspect the selected candidate and its diff.
Stop guessing, start measuring
From a production failure to a tested change in your repository.