Attach Claude

This page takes you from nothing to a finding, using only Claude and an org token. No console, and no help from us. It takes five operations. Each one is shown with what to ask, the tool call Claude makes, and what comes back.

Words this page uses

MCP (Model Context Protocol): the standard way an assistant such as Claude calls tools on a server. Litmus runs an MCP server; Claude is its client.
Tool: one operation on that server, with a name and arguments. MCP tools lists all of them.
Org token: the secret that proves a call is from your org. It has a scope: read can only read, push can also start runs, admin can also manage tokens.
Sandbox (also called an environment): a test copy of your system that your agent is run against. It is never your production system.
Question (also called a task): one request your agent is given in a sandbox, such as “set the assignee of task 7 to Alice”. Each question has an index, such as 0.3.
Verifier (also called a check): the rule that decides whether an answer to a question met it, for example “after the run, task 7’s assignee is Alice”.
Try: one attempt by your agent at one question. Litmus makes five tries per question per version, so one lucky or unlucky try does not decide anything.
Deciding calls: the tool calls in a try that the verdict rests on, and the checks the try failed.
Golden set: the questions your org holds fixed and re-asks of every version. Each question carries a tag, the use case it belongs to.
Served version: one version of your own MCP server, at one commit, that Litmus serves to your agent. Its id starts with sr_.
Run group (or group): one run of a set of questions on one version. Its id starts with job_. A job is anything that runs in the background; you poll it until it is done.
Compare: two run groups side by side, question by question: where one version met and the other did not, and which calls differ.
Pair-equal: whether the two versions in a compare were tested the same way: same harness, grader, documents, time budget, server instructions, model, question text, sandbox, data seed and agent. If not, Litmus will not compare them.
Copy defect: a known fault in our test copy (for example, a document the copy does not serve). A result that could be caused by our copy, and not by your agent, is held back until the copy is fixed.
Finding: a frozen report of what a compare showed, with the questions and calls it cites.
Dry run: a start that only prices the run and starts nothing.

1. Install

Nothing is installed on your side for Claude Code. In a terminal, with your workspace URL and token in your shell (see step 2):

claude mcp add --transport http litmus "$LITMUS_URL/v0/mcp" \
  --header "Authorization: Bearer $LITMUS_TOKEN"

To keep the token out of every file, put this in .mcp.json instead. Claude Code reads ${VAR} from your environment:

{
  "mcpServers": {
    "litmus": {
      "type": "http",
      "url": "${LITMUS_URL}/v0/mcp",
      "headers": { "Authorization": "Bearer ${LITMUS_TOKEN}" }
    }
  }
}

Claude Desktop speaks only stdio, so it runs a small bridge. Install it with pip install litmus-ai-sdk, then add:

{ "mcpServers": { "litmus": { "command": "litmus-ai", "args": ["mcp", "--stdio"] } } }

The bridge reads LITMUS_URL and LITMUS_TOKEN from the environment, or what litmus-ai init stored. To list fewer tools, add ?chapters= to the URL. This page’s calls use org (the whoami check in step 2), ontology, sandboxes, runs, agents (list_agents, for your agent’s id in (e)) and findings:

claude mcp add --transport http litmus \
  "$LITMUS_URL/v0/mcp?chapters=org,ontology,sandboxes,runs,agents,findings" \
  --header "Authorization: Bearer $LITMUS_TOKEN"

Any tool can still be called; the chapters narrow only the list Claude sees.

2. Auth

An org admin mints a token in the console under Settings → Tokens. The token is shown once. Operations 1 to 4 need a read token. Operation 5’s start needs push, so ask for a push token if you will start runs.

export LITMUS_URL="https://<your workspace>"
export LITMUS_TOKEN="<the token>"     # never paste it into a chat

Check it: ask Claude “call the Litmus whoami tool”. It answers with your org and the token’s scope. You never need to name your org: every tool defaults to the org your token is for.

3. The five operations

Each is one or more tool calls. Each answers in well under ten seconds.

(a) List the golden set

Ask: “List our Litmus golden set, with each question’s tag and how many are compared.”

{ "name": "golden_set_get", "arguments": {} }

Answers (trimmed):

{
  "questions": 1,
  "held": [{ "environment": "<sandbox id>", "index": "0.0", "cell": "lookup" }],
  "cells": [{ "cell": "lookup", "share": 1.0, "n": 1 }],
  "compared": { "n": 0, "of": 1, "line": "0 of 1 compared (same stack); 0 with tries on one stack only" }
}

held is the golden set: each question’s sandbox, index and tag (cell). compared says how many of them have five valid tries on both versions: n of of. Keep a question’s environment and index; (b) reads it. The answer is a kept view, built off the request and rebuilt as runs land, so it reads the same while the workspace is busy. It says so: every kept answer on this page carries served_from: "kept" and kept_at, when the copy you are reading was built. Right after a restart, or for a record nobody has read yet, it can answer 202 with warming: true and retry_after (seconds) instead: nothing is wrong; the copy is being built off the request (a build is cut and queued again if it runs past 300 seconds while another waits), and the answer never waits on it. Ask again after that many seconds. A kept block beside served_from means the copy is an older build’s, served while a newer one is made; kept.as_of says when it was built.

(b) Read one question

Ask: “Show question 0.0’s verifier, and the calls its tries made.” Two calls. The verifier, as it runs:

{ "name": "environment_task_checks_get",
  "arguments": { "env_id": "<sandbox id>", "index": "0.0" } }

{
  "index": "0.0",
  "checks": [{ "code": "field", "tier": "deterministic", "how": "query",
               "query": "read the system after the run\nfrom tasks\nwhere id = 7\nexpect assignee = \"Alice\" on a matching row" }],
  "last_run": { "run": "run_…", "trials": 5, "checks": [ … ] },
  "served_from": "kept", "kept_at": "2026-10-30T09:12:04Z"
}

tier says how the check decides: deterministic means code reads the record, so the same try always gets the same verdict. The tries, call by call. Find a run group with runs_groups_list, then:

{ "name": "jobs_calls", "arguments": { "job_id": "job_…" } }

{
  "tally": [{ "version": "…", "met": 0, "failed": 4 }],
  "tries": [{
    "task": "task 00", "trial": 1, "terminated": "agent_failure",
    "calls": [{ "seq": 2, "tool": "list_tasks", "answer": "data" }],
    "checks_failed": ["fact: the request's row holds the value asked"],
    "deciding": { "fact": [{ "check": "field" }] }
  }],
  "served_from": "kept", "kept_at": "2026-10-30T09:12:04Z"
}

Each try lists its calls (the tool and the kind of answer it got: data, empty, error:…, refused), the checks it failed, and which checks decided it (deciding). Both answers are kept copies, built when the run landed or its grade moved; kept_at says when. While a run is still landing its copy is rebuilt behind the page every few seconds.

(c) Read the compare of two served versions

Ask: “Compare run groups A and B, with A as main. What separates them?”

{ "name": "compare_runs",
  "arguments": { "group_a": "job_…", "group_b": "job_…", "main": "a" } }

Answers (trimmed):

{
  "pair_equal": { "equal": true, "gated": true, "differs": [] },
  "rows": [{ "task": "0.0", "a": { "met": 5, "valid": 5 }, "b": { "met": 0, "valid": 5 } }],
  "checkpoints": { … },
  "calls_vs_main": { … },
  "built_on": { "version": "…", "counts": [ … ] },
  "served_from": "kept", "kept_at": "2026-10-30T09:12:04Z"
}

The compare is kept per pair and rebuilt when either side lands a try or is graded again; built_on names the build and the grade counts it reflects. pair_equal is the first gate: equal: true means both versions were tested the same way, so the difference is the versions’. rows is each question: met out of valid tries on each side. calls_vs_main names the calls one version makes that the other does not. The two gates, when they refuse, answer an error instead (see Refusals below): pair_unequal if the versions were not tested the same way, and copy_defect_open if a question that separates them is blocked by a defect in our copy.

(d) Read one finding

Ask: “List our written findings and read the newest one, with its copy state.” Three calls:

{ "name": "weekly_reports", "arguments": { "kind": "written" } }

{ "name": "report_get", "arguments": { "report_id": "rpt_…" } }

{
  "id": "rpt_…", "kind": "written",
  "source": { "from": "compare", "main": "job_…", "other": "job_…" },
  "body": { "markdown": "In short: on the proposed version, 1 of 1 question moved down … Calls one version makes that the other does not …" },
  "tier": "deterministic: a projection of stored records, frozen at make time",
  "served_from": "kept", "kept_at": "2026-10-30T09:12:04Z"
}

The finding’s text cites its questions and the calls that differ. Its copy state, the registry of open copy defects:

{ "name": "copy_defects_list", "arguments": {} }

{
  "open": { "tickets": 0, "presence": 0 },
  "closed": 0,
  "blocked": { "questions": 0, "findings": 0 },
  "repair": { "repairable": 0, "harness": 0, "no_recipe": 0, "repairing": 0 },
  "defects": []
}

open counts the open defects: tickets are registered under a ticket of ours, presence were opened by the copy-presence detector. Both 0 means no known copy defect blocks anything. blocked is how many questions and findings the open ones hold back; repair says how many the product can close itself (repairable) and how many wait on a code change (harness). If a defect is open and blocks a question this finding cites, report_get is refused with copy_defect_open instead of answering.

(e) Price a run, then start it

Ask: “What would a one-try run of our agent on this sandbox cost? Then start it.” First the price; dry_run starts nothing. Find your agent’s id with list_agents.

{ "name": "run_selection",
  "arguments": { "environment": "<sandbox id>", "agent": "agt_…",
                 "trials": 1, "tries": 3, "dry_run": true } }

{
  "dry_run": true,
  "price": { "usd": 0.96, "per_try_usd": 0.32, "tries": 3, "basis": "list",
             "line": "Priced at $0.96 for 3 tries ($0.32 a try, from the model's list price).",
             "spend": { "spent_usd": 0.0, "cap": { "usd": 50.0 } } },
  "would_refuse": null, "model_calls": 0, "cost_usd": 0
}

tries is how many tries to price: questions × tries per question. would_refuse is the sentence a start would be refused with, if it would cross your daily spend cap. Then the start, with a push token:

{ "name": "run_selection",
  "arguments": { "environment": "<sandbox id>", "agent": "agt_…", "trials": 1 } }

It answers at once with the run group’s id (group) and starts nothing else. Poll it until status is done:

{ "name": "job_status", "arguments": { "job_id": "job_…" } }

A start spends model calls on your workspace’s provider key.

Refusals

When Litmus refuses a call, the tool answers with isError set, the reason (cause), a machine word (code) where there is one, and next: one sentence saying what to do. Read next first.

Refusal	What it means	What to do
`401`	Litmus did not recognise the token: none was sent, it was mistyped, or it was revoked or expired. This comes back as an HTTP error before any tool runs.	Send a valid token as `Authorization: Bearer <token>`; an admin mints one under Settings → Tokens.
`403`	The token is real, but not allowed this call: it is for another org, or its scope is too low (a `read` token starting a run). Nothing was changed.	Call it on your own org (leave the org out), or ask an admin for a token with the scope named.
`409` `copy_defect_open`	A question this answer cites is blocked by a known fault in our test copy. The result could be our copy’s fault, not your agent’s, so we do not show it, not even with a warning. The refusal names the defect and the questions.	Wait for the defect to close; its ticket says who is fixing it, and the questions are re-run then. Meanwhile, read pages that do not cite those questions.
`409` `pair_unequal`	The two versions were not tested the same way, so a difference between them could be the setup’s, not the versions’. `differs` names what was not the same (for example `server_instructions_passed`).	Run both versions again on one setup, then compare those two runs.

A 404 means the record does not exist for your org; a record of another org answers the same way, so check the id. A tool whose route answers 202 is not a refusal: it started a job, and next names the job_status call to poll — or, with warming: true in the answer, the value is being built off the request, and retry_after says in how many seconds to ask again.

Get started

API reference

Tools

Words this page uses

1. Install

2. Auth

3. The five operations

(a) List the golden set

(b) Read one question

(c) Read the compare of two served versions

(d) Read one finding

(e) Price a run, then start it

Refusals

​Words this page uses

​1. Install

​2. Auth

​3. The five operations

​(a) List the golden set

​(b) Read one question

​(c) Read the compare of two served versions

​(d) Read one finding

​(e) Price a run, then start it

​Refusals

Words this page uses

1. Install

2. Auth

3. The five operations

(a) List the golden set

(b) Read one question

(c) Read the compare of two served versions

(d) Read one finding

(e) Price a run, then start it

Refusals