Skip to main content
A dataset upload adds questions you already know the answers to beside the ones Litmus draws, graded the same way. In the console it is Datasets → New dataset → Upload a file (admins only). This page states the file it takes and the calls behind it, so the same thing can be done with a token.

​
File shape

A CSV or JSONL file with one question per row. Any other column is named in the preview and ignored. A CSV needs a header row; a JSONL file is one JSON object per line. One example row, as CSV:
The same row, as JSONL:

​
Limits

  • At most 500 rows in one file, and at most 8,000 characters in a prompt or a golden_response. A file over either is refused whole, with the number.
  • One seeding request takes at most 25 rows. The console refuses a file of more before sending; with a token, send the rest with pairs (row indexes) in another request.
  • A row with no prompt or no golden_response is not a question. The console names it by row number and sends nothing until it is fixed.

​
What happens to each row

Litmus turns each golden response into the facts a correct answer must state, each quoting its words, and shows you each one; you can correct any of them on the question’s page. These facts are model-derived, and are marked so. A row that no part of your workspace’s test copy is about is refused uncited, by row number: it cannot be graded there.

​
The three calls

  1. POST /v0/evidence with kind: golden_pairs and the rows as content stores the file. The answer’s id is the evidence id.
  2. POST /v0/orgs/{org}/environments/{env_id}/tasks with {"from_pairs": "<evidence id>"} seeds one question per row on that environment. It answers a job; when the job is done its result lists stored_as (the new question indexes) and refused ({pair, why}, pair counted from 0).
  3. POST /v0/orgs/{org}/datasets with {"name": "...", "environments": [{"id": "<env_id>", "indexes": [<stored_as>]}]} declares the dataset as exactly those questions. Declaring needs an admin token.
Or two calls: send "dataset": {"name": "..."} with step 2 ({"from_pairs": "<evidence id>", "dataset": {"name": "..."}}) and the job declares the set itself when it lands — exactly the indexes it stored, with origin uploaded — and reports it as result.dataset (declared, id, indexes, or why when nothing was stored). A name already taken is refused at the call, before anything is seeded.