Instrument your agent

A run reaches SenseLab flat unless the agent says otherwise: the memory it read, the tools it called and the contexts it recorded, in order, with no durations, no nesting, no token counts and no cost. The Runs page shows that list and a hint — Want tokens, cost and nested steps? Instrument your agent. Instrumenting adds three things to each run:

Nested steps with durations. Which tool call belonged to which plan, and how long each took.
LLM calls. Model, input and output tokens, latency and cost per call, and total_tokens / total_cost_usd on the run.
Attributes. Dimensions to filter and group runs by: customer, task type, environment, ticket.

Judges are shown the same tree, so a fail can point at the exact step. How runs are graded, and what happens to the failures, is covered in Behavior.

Pick a path

You have	Use
An agent already emitting OpenTelemetry — LangChain, LangGraph, LlamaIndex, CrewAI, DSPy through OpenInference, the OTel GenAI instrumentors, or Traceloop	OpenTelemetry — point the exporter at SenseLab; no code changes
An agent that uses SenseLab through MCP (Cursor, Claude Code, Codex)	MCP tools — the agent records steps and LLM calls with tool calls
An agent built on the SenseLab Python SDK	Python SDK — two calls add attributes, tokens and cost to the run it already commits
Anything else, in any language	HTTP — post the run with its spans in one request

All of them land as the same sealed trace. You can mix them across agents.

OpenTelemetry

SenseLab accepts OTLP/HTTP at POST /v1/traces, in protobuf or JSON, gzip optional. Authenticate with your API key in X-AMFS-API-Key or Authorization: Bearer amfs_.... Point an existing exporter at it with environment variables:

export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://amfs-login.sense-lab.ai/v1/traces"
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_HEADERS="X-AMFS-API-Key=${AMFS_API_KEY}"
export OTEL_SERVICE_NAME="deploy-agent"   # becomes the agent id

The dashboard’s Connect page carries this block with your environment’s host filled in, and a run’s flat step tree links to it with the agent’s id already set. Use the _TRACES_ variables as written: exporters append /v1/traces to the generic OTEL_EXPORTER_OTLP_ENDPOINT themselves. Or configure it in code:

import os

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider(resource=Resource.create({"service.name": "deploy-agent"}))
provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(
            endpoint="https://amfs-login.sense-lab.ai/v1/traces",
            headers={"X-AMFS-API-Key": os.environ["AMFS_API_KEY"]},
        )
    )
)
trace.set_tracer_provider(provider)

Every OTLP trace becomes one run. The root span is the run; its children are the steps.

What is read from your spans

SenseLab understands the OpenTelemetry GenAI conventions, OpenInference, and its own amfs.* attributes. The first match wins.

Run field	Read from
Agent id	`amfs.agent_id` (resource or root span), `gen_ai.agent.name`, then `service.name`
Session id	`amfs.session_id`, `session.id`; defaults to the OTLP trace id
Task input / response	The root span’s `input.value` / `output.value`
Outcome	`amfs.outcome_type` on the root, else the root span’s status: `OK` → success, `ERROR` → failure. `UNSET` — what most frameworks leave on a span that did not fail — records no outcome, so set `amfs.outcome_type` if you want one
Outcome reference	`amfs.outcome_ref`
Step kind	`amfs.span.kind`, `openinference.span.kind` (`CHAIN`, `LLM`, `TOOL`, `RETRIEVER`, `AGENT`), `gen_ai.operation.name`, or inferred from the attributes below
Model	`gen_ai.request.model`, `gen_ai.response.model`, `llm.model_name`
Tokens	`gen_ai.usage.input_tokens` / `gen_ai.usage.output_tokens` (or the older `gen_ai.usage.prompt_tokens` / `completion_tokens`), or `llm.token_count.prompt` / `llm.token_count.completion`
Cost	`amfs.llm.cost_usd`
Tool name	`tool.name`, `gen_ai.tool.name`
Attributes	Resource attributes and the root span’s attributes

Cost is not estimated from the model name on this path. Set amfs.llm.cost_usd on each LLM span if you want spend on the run; without it the run shows tokens and no cost.

Batches and retries

Exporters batch by time, so one trace often arrives in several requests. Late spans are merged into the run as a new version in the same session chain, and the Runs page shows the latest. A request that repeats spans already recorded writes nothing. The response is the standard OTLP one: 200 with an empty body on success, partial_success listing rejected spans otherwise — never a 4xx for a mapping problem, so the exporter does not retry spans that were accepted. Bodies are capped at 4 MiB and each account at 600 requests a minute.

MCP tools

An agent working through the SenseLab MCP server already gets a step for every memory read, recorded action and context. Four tools add the rest, and amfs_commit_outcome seals everything into the run.

Tool	What it records
`amfs_start_span(name, kind, input)` → `span_id`	Opens a step and makes it current; everything until the matching end nests under it
`amfs_end_span(span_id, output, status, error)`	Closes it, measuring the duration
`amfs_record_span(name, kind, input, output, duration_ms)`	A step that already finished, in one call
`amfs_record_llm_call(model, input_tokens, output_tokens, cost_usd, latency_ms)`	An LLM call — tokens and cost roll up to the run
`amfs_set_trace_attributes({...})`	Dimensions to filter runs by

A run that reads as a tree:

amfs_set_trace_attributes({"customer": "acme", "task_type": "deploy"})
plan = amfs_start_span("plan", kind="chain", input={"ticket": "INC-2047"})
  amfs_record_llm_call(model="gpt-5-mini", input_tokens=812, output_tokens=64, cost_usd=0.00033)
amfs_end_span(plan, output={"chosen": "roll back to v41"})
amfs_record_span("deploy_rollback", kind="tool", input={"to": "v41"}, output="ok", duration_ms=8300)
amfs_commit_outcome("deploy-142", "success",
                    task_input="roll api back to v41",
                    response_text="Rolled api back to v41; p99 is back under 400 ms.")

kind is one of tool, retrieval, agent, chain, memory_read, memory_write, llm or custom. Inputs and outputs are scanned for secrets and capped before storage. Spans left open are closed at commit.

The response

The one thing no tool call can capture is what the agent said. SenseLab sees its own tools, so a run committed through MCP has a task, steps and an outcome — and no response, which is what a judge grading the answer reads. Pass response_text on amfs_commit_outcome: the final message to the user, in full. Without it the run’s response is empty and any judge that asks about the answer fails on "response_text": null.

Put these in the agent’s instructions rather than hoping it discovers them: “open a span for each sub-task with amfs_start_span, record every model call with amfs_record_llm_call, close spans with amfs_end_span, and pass task_input and response_text on amfs_commit_outcome.”

Requires amfs-mcp-server-pro 0.1.53 or later; response_text needs 0.1.54.

Python SDK

An agent built on the Python SDK and connected through the HTTP adapter already commits a run with every memory read, recorded action and context as a step. Two calls add the attributes and the LLM calls, and commit_outcome seals them in with the rest:

mem.set_session_attributes({"customer": "acme", "task_type": "deploy"})

resp = client.chat.completions.create(...)
mem.record_llm_call(
    resp.model, resp.usage.prompt_tokens, resp.usage.completion_tokens,
    provider="openai", cost_usd=0.00033, latency_ms=412.0,
)

mem.record_action("deploy_rollback", {"to": "v41"}, result="ok", duration_ms=8300)

mem.commit_outcome(
    "deploy-142", OutcomeType.SUCCESS,
    task_input="roll api back to v41",
    response_text="Rolled api back to v41; p99 is back under 400 ms.",
)

Call	What it records
`set_session_attributes({...})`	Dimensions to filter runs by. Scalars only, at most 20 keys; merged into the bag the next `commit_outcome` stamps on the run
`record_llm_call(model, input_tokens, output_tokens, cost_usd=, latency_ms=, provider=)`	An LLM call — tokens and cost roll up to the run
`record_action(tool_name, arguments, result=, success=, duration_ms=)`	A tool call, as a step on the run

Both the attributes and the calls can also be handed straight to commit_outcome(attributes={...}, llm_calls=[...]) for a session that did not record them as it went.

This path gives a run its attributes, tokens and cost. It does not nest steps — they stay the flat list the run already had. For a step tree with durations, send the run over OpenTelemetry or HTTP. Cost is not estimated here either: pass cost_usd on each call, or the run shows tokens and no cost.

Requires amfs 0.5.10 or later. The TypeScript SDK does not have these calls yet; use OpenTelemetry or HTTP from Node.js.

HTTP

POST /api/v1/traces persists a run in one request. Steps, LLM calls and attributes travel inside session_metadata under spans, llm_calls and attributes; the response carries the sealed run’s id as immutable_trace_id.

curl -X POST https://amfs-login.sense-lab.ai/api/v1/traces \
  -H "X-AMFS-API-Key: $AMFS_API_KEY" \
  -H "Content-Type: application/json" \
  -d @run.json

run.json

{
  "agent_id": "deploy-agent",
  "session_id": "sess-142",
  "outcome_ref": "deploy-142",
  "outcome_type": "success",
  "task_input": "roll api back to v41",
  "response_text": "Rolled back to v41; error rate recovered.",
  "session_started_at": "2026-09-05T12:00:00Z",
  "session_ended_at": "2026-09-05T12:00:12Z",
  "session_metadata": {
    "attributes": { "customer": "acme", "task_type": "deploy" },
    "spans": [
      { "span_id": "root", "parent_span_id": null, "name": "deploy-agent run", "kind": "session",
        "status": "ok", "started_at": "2026-09-05T12:00:00Z", "ended_at": "2026-09-05T12:00:12Z",
        "duration_ms": 12000 },
      { "span_id": "plan", "parent_span_id": "root", "name": "plan", "kind": "chain",
        "status": "ok", "started_at": "2026-09-05T12:00:00Z", "ended_at": "2026-09-05T12:00:03Z",
        "duration_ms": 3000, "input_payload": { "goal": "roll back" }, "output_payload": { "chosen": "v41" } },
      { "span_id": "llm-1", "parent_span_id": "plan", "name": "gpt-5-mini", "kind": "llm",
        "status": "ok", "started_at": "2026-09-05T12:00:00Z", "ended_at": "2026-09-05T12:00:02Z",
        "duration_ms": 2300,
        "llm": { "call_id": "llm-1", "model": "gpt-5-mini", "provider": "openai",
                 "input_tokens": 812, "output_tokens": 64, "cost_usd": 0.00033, "latency_ms": 2300 } },
      { "span_id": "tool-1", "parent_span_id": "root", "name": "deploy_rollback", "kind": "tool",
        "status": "ok", "started_at": "2026-09-05T12:00:03Z", "ended_at": "2026-09-05T12:00:11Z",
        "duration_ms": 8300, "input_payload": { "to": "v41" }, "output_payload": "ok" }
    ],
    "llm_calls": [
      { "call_id": "llm-1", "model": "gpt-5-mini", "provider": "openai",
        "input_tokens": 812, "output_tokens": 64, "cost_usd": 0.00033, "latency_ms": 2300 }
    ]
  }
}

Span fields

Field	Notes
`span_id`, `parent_span_id`	Any strings unique within the run. Exactly one span has `parent_span_id: null`; it is the run
`name`	Short and stable — the same step should have the same name every run
`kind`	`session` for the root; `tool`, `retrieval`, `agent`, `chain`, `memory_read`, `memory_write`, `llm` or `custom` otherwise
`status`	`ok`, `error` or `unset`; add `error` text on a failure
`started_at`, `ended_at`, `duration_ms`	ISO 8601 timestamps and milliseconds. Include `duration_ms`; it is what the Runs page shows
`input_payload`, `output_payload`	Any JSON; scanned for secrets and capped, with the overflow kept by reference
`llm`	An LLM call object, on `llm` spans. List every call in `llm_calls` too, with the same `call_id`
`attributes`	Flat scalars for this step

llm_calls is what total_tokens and total_cost_usd are summed from. Leave cost_usd out of a call and the run’s cost is left blank rather than understated.

What you see afterwards

Open Agents → your agent → Runs and pick a run. The header shows its duration and, once LLM calls are recorded, agent spend — tokens and cost. The Steps pane is the tree you sent; clicking an llm step shows model, tokens and cost for that call, and the instrumentation hint is gone. Judges that fail the run point at spans in this tree.

Runs that arrived before you instrumented keep their flat shape. Sealed traces are never rewritten.

Product

Getting Started

Guides

Self-Hosting

Core Concepts

Pro Features

Instrument your agent

Pick a path

OpenTelemetry

What is read from your spans

Batches and retries

MCP tools

The response

Python SDK

HTTP

Span fields

What you see afterwards

​Pick a path

​OpenTelemetry

​What is read from your spans

​Batches and retries

​MCP tools

​The response

​Python SDK

​HTTP

​Span fields

​What you see afterwards

Pick a path

OpenTelemetry

What is read from your spans

Batches and retries

MCP tools

The response

Python SDK

HTTP

Span fields

What you see afterwards