Skip to main content
Your agents make the same kinds of decisions over and over, and a general-purpose model charges general-purpose prices for every one of them. Managed Models turns the decisions that worked into a small model of your own: SenseLab builds the training set from traces already in your account, runs the fine-tune, scores the result against decisions the model was never shown, and serves it. There is nothing to export, no training infrastructure to stand up, and no ML engineer in the loop. The ML layer improves what your agents read; this changes the model they are.

​
What the model learns

A decision trace already records what an agent was asked, what it knew, what it did, and how that turned out. Training pairs the request with the first tool call the agent made, across the decisions your own outcomes marked successful. What comes out is a model that answers a request with an action: the tool to call, and the arguments to call it with.
A managed model is not a smaller general-purpose chat model. It is trained to emit one tool call in your agent’s own vocabulary, which makes it good at the routine decisions it has seen many times and no use at all for open-ended conversation. Keep a general model in the loop for everything else.
Decisions that went badly are never training targets — teaching a model the action that caused an incident is an expensive way to repeat it. They are kept for evaluation instead, where they answer a more useful question.

​
What you need first

A Pro plan or above, plus a Managed Models subscription. The subscription is $199/month per account and unlocks training for the whole account. Your plan decides how many models may serve at once: one on Pro, three on Teams, a hundred on Enterprise. Agents that record what they did. A decision becomes a training example only if it carries the request that started the work, at least one recorded action, and an outcome. In practice that means your agents calling amfs_record_action as they act and amfs_commit_outcome(..., task_input=...) when the work is done. Traces missing the request text or the action still count as traces, and the readiness panel names which half is missing.
If decisions are arriving but none of them are eligible, it is almost always the request text. An agent that commits outcomes without task_input will never become trainable no matter how long it runs.
Enough of them. Readiness is measured per agent over a rolling 90 days: Below 100 a run isn’t worth the queue slot. Between 100 and 200 it’s allowed but flagged, because the holdout gets thin enough that the scoring below stops meaning much. A model trained on a single tool has nothing to generalise across. The dashboard shows where you stand as a count and a rate rather than a verdict — “142 of 200, about a week at your current rate” — so the gap is something you can close. A model is scoped to one agent, because that is where the loop boundary is.

​
Creating and training a model

1

Subscribe

On Models, subscribe to Managed Models. This is separate from your plan subscription and covers the whole account.
2

Create the model

Pick the agent to clone, give the model a name, and choose a tier — see Choosing a tier below if you’re unsure.
3

Check the estimate

Before a run starts, SenseLab shows how many tokens it is expected to consume, how much of your monthly training allowance that uses, and what it would cost if it goes past. The estimate is there to inform you, not to block you — if it can’t be measured it says so rather than quietly reporting zero.
4

Start training

The run builds your dataset, submits it, tunes, and then scores the result. The tuning itself is the long part and takes hours, so there is nothing to watch — you’re notified when it finishes. One run at a time per account.

​
Choosing a tier

Both tiers are the same product. Same subscription, same training allowance, same evaluation, same endpoint, same limits. What differs is how capable the model underneath is, and what its tokens cost you to serve. Most loops want the default, and it’s worth remembering why: the job you’re handing the model is to pick the tool your agent already picks, on a request shaped like ones it has seen many times. That’s a narrow task. A more capable base model earns its price when the choice itself is hard — when which tool is right depends on reasoning through the request rather than recognising it. The reason to spend a minute on the decision is that the tier is fixed for the life of the model. Moving a loop to the other tier means creating a second model and paying for another training run, so it’s a decision rather than a setting. Note also that the gap between the tiers is wider on output than on input, so a loop that emits long tool calls feels the difference more than the table suggests. If you’re unsure, start on the default and let the evaluation report settle it. Per-tool agreement shows you which task shapes the cheaper tier handles and which it misses; if it falls short on the shapes that matter to you, that’s the case for the pro tier — made with your own numbers rather than a guess.

​
How your model is evaluated

Every run ends with the same question: is this version good enough to put in front of your traffic? SenseLab answers it, on the server. Going live is not a button you press against a model that agrees with your agent 40% of the time. The bars are ours by default, and you can move them or add cases of your own — both covered below.

​
What it’s scored on

The most recent slice of your decisions — about 15% — is held back and never given to the trainer. Everything below is measured on that slice alone. Two details make it worth trusting: The holdout is the recent slice, not a random sample. The question a gate has to answer is whether the model will handle the decisions that come next, which is a question about the future. A randomly chosen decision usually has near-identical siblings from the same afternoon sitting in the training data, so it scores well for the wrong reason. It is not the trainer’s validation set. The provider gets its own validation slice and uses it to fit the run, so by the time tuning finishes that data is no longer held out in any useful sense. Scoring against it would produce numbers that are optimistic by construction — and then show them to you as evidence. The holdout also keeps the decisions that went wrong, even though training excludes them. That’s deliberate, and it’s what makes the last row of the table below answerable at all.

​
The bars a version has to clear

These are the defaults every model starts with: Agreement is measured over the successful decisions only. The two halves of the holdout ask opposite questions: the successes ask whether the model does what your agent did, and the failures ask whether it repeats what went wrong. Mixing them would penalise a model for declining to reproduce an incident. Tool agreement is the headline because picking the wrong tool is the failure you notice, while argument drift usually isn’t — a one-hour window where your agent used six is often equally correct. That’s also why exact agreement sits lower. The report carries more than the gate uses: a graded argument score that gives partial credit per field, agreement broken down per tool so you can see which task shape is weak, typical and worst-case latency, and a projected cost per thousand calls against what you’re paying now.
The honest limit: agreement is not correctness. SenseLab can measure whether the small model does what your agent did. It cannot measure whether that was the right thing to do — your outcome labels are the only correctness signal in the system, and they are already spent choosing which decisions are worth imitating. The failure half of the holdout is the one place this flips, and there agreement is the bad result.

​
Setting your own bars

Our defaults are a starting point, not a judgement about your domain. Each bar in the table above can be set per model, under Evaluation on the model page. Raising a bar is unrestricted. If you want 95% tool agreement before anything serves your traffic, you can have it — you’ll simply promote less often, which is the trade you’re choosing. Lowering stops where the number stops meaning anything: agreement can’t go below 50%, because under half a model is closer to guessing than agreeing; valid actions can’t go below 80%, because prose instead of an action is broken rather than lenient; and the sample-size floors stay at 20, because smaller samples swing several points on a single decision. We serve these models under our own name, so the floor under the floor is ours to keep. Anything outside the range is rejected rather than quietly clamped, so the number you set is always the number that decides. Exact agreement is the one bar with no floor at all. It’s the weakest measure in the report — a one-hour window scored against a six-hour one counts as total disagreement — so if you read the graded argument score beside it and conclude it’s noise for your tools, you’re probably right.
Bars you don’t set keep following the platform default. That’s deliberate: if we revise a default later, the change reaches every model whose owner never had an opinion about that bar, instead of freezing today’s number into every model ever created. Every report says which numbers are yours and which are ours.

​
Adding your own cases

Agreement across a holdout tells you how a version behaves on average. It can’t tell you whether it still handles the incident — the one you never want repeated. For that, you can give a model an eval set of your own, in two ways:
  • Pin a past decision. Choose a real decision from the model’s agent, and the expected action is taken from the decision itself rather than retyped, so a case can’t drift from what actually happened.
  • Write a case. Describe a situation your traces don’t contain yet, and name the tool and arguments you expect. This is how you test for something before it happens to you.
A model can hold up to 500 cases. Every case is answered by the model on every run and every rescore, so the set is meant to be the ones that matter. Pinned decisions are withheld from every training split, and that exclusion is re-checked before scoring, since a leaked case would score better rather than worse. Cases are scored on exactly the same harness as the holdout — so when you compare “94% on their set, 71% on mine”, the two numbers were computed the same way. The report lists your cases one by one rather than only in aggregate, because a customer who wrote six cases is asking about those six. Each says what the model chose and how that reads: it agreed or disagreed on a case that went well, avoided or repeated on one you pinned because it went wrong, and separately whether the answer wasn’t a valid action at all.
By default your cases report without deciding anything. Turn on require eval cases and a set of at least five can add a reason to withhold a promotion — but it can never satisfy one. The holdout still has to pass on its own; your cases can only stop a version, never wave it through.
Each bar on your set has its own floor of five, applied to the half it belongs to: valid actions across the whole set, agreement over the cases that went well, and repetition over the ones that went wrong. A set of three successes and two failures enforces nothing about either, and the report says so bar by bar — “these stopped the promotion” and “these were scored and decided nothing” are different sentences, and reading the second as the first is how you end up trusting a gate that isn’t protecting you. One boundary worth knowing: a case pinned while a run is already training may already be inside that run’s training data, so it’s reported as pinned after training and starts counting from the next run.

​
What happens next

If a version clears every bar it is promoted and starts serving, without you doing anything. If your plan has no free serving slot or billing is unsettled, the version waits and you promote it yourself when that’s resolved. If it misses a bar, it isn’t promoted, and the report says which bar and by how much in plain words — “tool agreement is 68%, needs 75%”. Nothing about your currently live model changes. You can retrain when more decisions have accumulated, and if a run finished but couldn’t be scored you can rescore it without paying for a new run. The report can be shared as a link that needs no login, so a budget holder can read the numbers without an account. Links expire after 30 days and you can revoke one at any time.

​
Calling your model

Your model is served behind an OpenAI-compatible endpoint, so most code needs a base URL, a key, and a model string:
The model string is the id shown on the model’s Connect tab, and the key is an ordinary SenseLab API key. For Anthropic-dialect clients, point them at /api/v1/messages instead; the OpenAI path also supports streaming. A few expectations differ from a general model, and they follow from what the model is:
  • Send the request, not the conversation. Only the latest user message is used, and SenseLab pulls the relevant memory in for you. A long messages array doesn’t produce a multi-turn call.
  • Answers are short by design. Output is capped, because the answer is a tool call rather than an essay.
  • You don’t pass tool definitions. The model was trained on your agent’s tools, so the answer is the call. Tool and function parameters are ignored.
  • Failed calls aren’t billed.

​
When the tuned model can’t answer

Some requests won’t suit the tuned model, so SenseLab falls back to a general one rather than handing your agent something it can’t execute. That happens when the answer isn’t a parseable action, when it names a tool your agent doesn’t use, when it’s blocked on safety grounds, and when the tuned model is briefly unavailable or rate limited. You can nominate which model to fall back to, or leave it to SenseLab. Either way your caller gets an ordinary successful response. Fallback calls are charged at the fallback model’s own rate rather than your tier’s, and they’re counted separately in usage — a climbing fallback count is the number to watch, because it means you’re paying general-model prices for a model you tuned to avoid them.
The fallback only covers a model that is live. A model that is suspended or out of date refuses the request instead, so keep your own path to a general model for that case.

​
Keeping a model current

A tuned model is a snapshot of a system that keeps moving. SenseLab watches for the ways the world goes out from under it, and marks a model out of date when:
A model marked out of date stops serving, and calls to it are refused rather than answered. That’s deliberate: each of the reasons above degrades quality without producing a single error, so a model in that state would keep sounding confident. Retraining is how you get back.
Every run adds a version rather than replacing one. You can promote any version that passed its evaluation, and rolling back to the previous one takes effect immediately — useful if a newly promoted model behaves worse in production than its scores suggested. Retraining as decisions accumulate is the same pipeline, and the loop compounds: your tuned model’s own runs are recorded like any other, so the next tune starts from a better baseline than the last one did.

​
What it costs

Training is metered rather than capped, because a customer with years of history should be able to train on all of it. Serving has no included allowance and bills from the first token. Every model starts with a $500 monthly spend cap. You’re warned at 80%, and at 100% the model stops answering until you raise the cap — a cap is a floor under surprise invoices, not a performance setting. Billing & Metering covers how this appears on your invoice.

​
Worth knowing

  • Training and serving are US-only. There is no region choice.
  • One training run at a time per account, and runs queue behind other accounts, so a run can wait before it starts.
  • If your subscription lapses, models stop serving but their weights are kept for 30 days. After that they’re deleted and the model has to be retrained.
  • This is driven from the dashboard and the REST API. There are no MCP tools for creating or training a model: your agents produce the decisions, and people decide when to turn them into a model.

​
What’s next

Immutable Traces

The sealed decision traces every managed model is trained from.

ML Layer

How outcomes sharpen retrieval and calibrate confidence, without training a model.