Agent Testing
Prove every change before you ship it
Catch regressions before your users do. Overmind writes the evals from your agent's own context graph, tests every candidate against your baseline, and turns the winning change into a GitHub PR.
Evals you did not have to write
The context graph knows what each agent does. That knowledge becomes the test suite.
Eval generation from context
The repo scan builds a capability card for each agent: what it is designed to do, the tools it can call, and how it can fail. Rubrics are written from that card and compiled into weighted pass/fail checks you review and edit before anything is scored.
Scoped to each agent
A triage agent is not scored on a planner’s criteria. Rubrics are scoped per agent, and every run reports per-metric scores you can compare across models, prompts, and releases.
The Optimiser improves the whole agent
Candidates are real git diffs across prompts, tool definitions, and agent logic, written by a coding agent and run against your actual repo through the Overmind CLI. Every candidate is scored on the same eval set as your baseline, so a gain is a measured result, not a hunch.
The winning change is a PR
When a candidate beats the baseline, one click opens the GitHub PR: the winning diff with the baseline-versus-candidate scores in the description. If gains flatten across experiments, Overmind says so and recommends training instead of more prompt surgery.
What you get
A test suite generated from context and an optimisation loop that ends in a reviewable change.
Evals
Rubric generation
Rubrics grounded in each agent's purpose, tools, and failure modes, straight from the repo scan.
Multiple evaluator kinds
Judge, trajectory, deterministic, statistical, and agentic evaluators, scored at output, turn, or trajectory level.
Per-metric scores with rationale
Each metric has a pass threshold, and every judge score carries its reasoning. A low mark is an explanation.
Optimisation
Same-set benchmarking
Baseline and every candidate scored on the identical eval set. Differences come from the change alone.
GitHub PR output
One click turns the winning diff into a pull request with its scores attached. Review, comment, merge.
Plateau detection
When best scores stop improving across experiments, Overmind says so and recommends training instead.
Stop guessing, start measuring
Evals from context. Improvements as PRs.