Behavior

An agent reports success on almost every run. Some of those runs promised a refund it could not give, answered without looking anything up, or retried the same call forty times and then said “done”. Reading transcripts by hand does not scale past the first week, and “the agent said it worked” is the only signal most teams have. The Behavior section on each agent’s page answers three questions: is this agent behaving, where is it not, and what do we do about it. Judges read every run and say whether the agent behaved. Repeated failures are grouped into named behaviors. Behaviors that keep recurring become incidents. And each failure can become a fix that is tested against the runs it claims to repair before anything changes. In one line: you get the failures the agent called a success, and a tested fix for each one.

---

​

What you need first

Judges grade runs. A run is a sealed decision trace: what the agent was asked, what it read, what it did and what it answered. Runs reach SenseLab in three ways, and any of them is enough to start: Two things make verdicts sharper:

• Pass task_input and response_text when you commit. The request and the answer are what most judges read. Without them a judge sees what the agent did but not what it was asked or what it said, and will often return Uncertain.
• Instrument the run. Steps, LLM calls, tokens and cost are optional, but a judge that can see the step tree can point its fail at the exact step. Instrument your agent covers every path.

---

​

Your first verdict

The Overview tab opens with a three-step setup card. It takes about two minutes. Once judges exist and verdicts are in, the card collapses to a single line and the Overview becomes the weekly picture.

---

​

Verdicts

A verdict is one of four things: Only Pass and Fail count. The fail rate is fail verdicts over decided verdicts (pass + fail), so a flaky judge does not read as a failing agent. The Overview shows, for the current window (two weeks by default):

• Coverage — the share of runs at least one judge graded. Below 100% means sampling, disabled judges, or runs sealed before a judge existed.
• Fail rate, with its trend per day.
• Disagreement — runs the agent recorded as a success that a judge failed. This is the best single signal that the agent’s own outcome records cannot be trusted alone.
• Open incidents, verdicts by judge (one bar per judge), the top behaviors by how many runs they were seen in, and the fixes waiting for a decision.

Every number is a link to the runs behind it.

---

​

Runs

The Runs tab lists every run with its verdicts, duration, tokens and cost (once the agent is instrumented), and the attributes the agent stamped on it. Filter by judge, verdict, behavior, duration, errors, or any attribute. Grade on a row grades that one run now. Open a run to see its step tree on the left and, on the right, the behaviors found in it, each step’s payload, the memory it read, and the response it gave, with the evidence each judge quoted highlighted in place. From the run you can grade it, flag a behavior, propose a fix (when a judge has failed it), or compare it with another run.

---

​

Judges

A judge is a rule a run is graded against, written in plain language. A name and a one-sentence description — “The agent states something no tool result or memory supports” — is a complete judge: the rubric the grader uses is written from the description, and you can edit it directly if you want more control. In the rubric, {{trace}} is where the run goes and {{evidence}} is an optional grounding report of each claim in the response and where it is supported. For each judge you choose:

• Share of runs to grade. The share of new runs this judge grades as they land. Below 100% it samples — enough to see a trend, fewer verdicts to read. Grade recent runs and backfills grade every run regardless.
• Model. Which model grades. Leave it on the account default unless you have a reason.
• Enabled. A judge that is off keeps its history and its verdicts; it just stops grading.

Before enabling a judge, dry-run it: it grades 20 recent runs without storing anything and shows what it would fail and what it would cost, so you tune the wording on real runs rather than in the abstract. Backfill grades the past with a new or edited judge so you have a baseline the same day. Every edit is a new version. A judge’s page shows its history with a diff between any two versions, its detection rate per day, how often it disagreed with what the agent reported, and the runs it graded. A judge that suddenly fails everything can be read against what it said last week; a verdict always records the judge version it was graded under. Writing a judge that works

• One failure per judge. “Promised something it cannot do” and “answered without checking” are two judges, not one.
• Say what passes as well as what fails. The starter judges do this: “recovering, retrying once, or telling the user passes”.
• Hedged statements and questions are not claims. Say so if your judge is about claims.
• Dry-run, read the fails, adjust, dry-run again. Then enable and backfill.

---

​

Behaviors

A behavior is a named pattern of failure: “Skips the smoke test after promotion”. Behaviors come from two places:

• Discovered. When several runs fail the same way, SenseLab proposes a named behavior and links the runs. It is marked novel until someone reviews it. Rediscover runs discovery again on recent verdicts.
• Flagged by you. From any run, Flag a behavior: a name and one sentence is enough, and you say whether it counts as a failure or as something to watch.

Each behavior shows how many runs it was seen in, the share of all runs affected, and an occurrence chart — when it happens, and on which slice of traffic. Behaviors are what incidents and fixes attach to.

---

​

Incidents and alerts

A behavior that keeps recurring across runs opens an incident — one open incident per behavior. Open ones are still happening; acknowledge one to say it is being looked at, resolve it when it has stopped, or mute it for a while if you have decided to live with it. The Incidents tab has a segmentation explorer: pick a behavior and a dimension — model, agent version, tool, or any attribute your agent stamps on its runs — and see where the failures concentrate. The dimensions available are the ones your runs carry; attributes are how you add more. An incident can be shared into a Room, so the team’s agents know about it, or as a public report with a link for someone who is not on SenseLab. The alerts bell at the top of the section fires when:

• a behavior with a shipped fix comes back,
• a behavior’s rate drifts sharply against the previous window,
• discovery promotes a behavior nobody has named,
• an incident opens on a behavior you watch,
• grading spend nears the account’s monthly cap.

A muted incident silences alerts on its behavior. The account also receives a weekly digest email covering every agent.

---

​

Fixes

A fix is a claim with a test attached, not an edit and a hope. Every fix carries a prediction — the failing runs it repairs, the passing runs it must not disturb, and the prevention it expects — and is replayed against that prediction before anything changes. Fixes are proposed from any run a judge failed (Propose fix on the run), and SenseLab proposes them on its own when an incident opens.

​

Where the fix goes

The first thing a fix does is say what kind of failure this was, so the repair goes where the fault is:

​

How a fix is tested

The fix is staged on its own memory branch, so the live agent does not read it yet. Then:

1. Replay test. The failing runs are replayed with the fix in place and graded by the judge that failed them. The passing runs on the same work are replayed too, so a fix that repairs one thing by breaking another is caught. The result is Test passed, Test failed, or Test inconclusive — inconclusive means too little of the run could be reconstructed to call it, and is not treated as a fail.
2. Your agent’s replay (optional). With a replay webhook set on the Overview, each case is sent, signed, to your own agent, which runs it for real on the fix’s branch and reports back. The verdict is then not only SenseLab’s replay of your agent.
3. Canary (optional, Teams and Enterprise). The fix goes live on its branch for a share of the agent’s sessions while the rest stay on the current knowledge. It passes when the fixed behavior recurs significantly less on the branch than on main, fails if the branch does worse, and is called inconclusive if the window closes without a significant difference. It never widens on its own, and writes from either arm still land on your main knowledge.

​

Shipping, verifying, rolling back

Ship it merges a knowledge or procedure fix into the agent’s memory — the next run reads it. A code fix becomes Handed off once you copy the handoff. A week after shipping, a verify sweep compares what happened to what the fix predicted: did the behavior stop recurring in new runs, and did the passing runs stay passing? The fix becomes Verified or Regressed. Roll back is always one click and nothing in memory is deleted: the previous version is restored and the fix’s entries are retired.

​

Proof

Attach a fix to a suite to pin the runs it claims to repair as cases you can re-check any time, and the fix shows the behavior’s rate before and after shipping against what it predicted. “We fixed it” becomes a number with a date.

​

Who decides: the repair policy

One setting per agent, on the Overview: Automatic policies only act once your runs carry what they need to tell a fix’s effect from everything else: which model ran each run, which version of your agent ran it, and, for canaries, an outcome confirmed from outside the agent — CI, a monitor, a reviewer. The Overview shows the policy as Active, Warming up (waiting for enough runs that carry them) or Paused (runs are missing one), with what to add. Until then a fix that passes waits for you. Attributes are how you stamp model and agent version on a run.

---

​

Investigate

The box above the tabs takes a question in your own words — why did the agent promise refunds this week? — and answers from this agent’s runs, verdicts, behaviors and fixes, with each thing it refers to as a link to the exact slice.

---

​

From your agent or scripts

Everything above is also available to your agents. The SenseLab MCP server (installed with an API key — see MCP Setup) exposes the Behavior tools; the most useful: The same operations are REST endpoints under /api/v1/eval/ on https://amfs-login.sense-lab.ai, authenticated with your API key. These tools act on your SenseLab account; connected to a local store they return a message saying so.

---

​

Troubleshooting

---

​

What’s next