# Woodstitch API — human ratings for TTS

Paid human listeners judge your audio and give you a number back. Use it when
automated metrics (WER, speaker-sim, predicted MOS) can't answer the question.

Two templates, chosen with `"template"` on create:

| `template` | A listener hears | and answers | you get |
|---|---|---|---|
| `tts_preference.v1` *(default)* | two clips | which one they prefer (forced choice, no tie) | preference for side B, 0.5 = no preference |
| `tts_detect.v1` | one clip | human or AI, plus confidence 1–4 | share of clips taken for human, vs a real-human baseline |

Use **preference** to pick between two systems or checkpoints.
Use **detection** to find out whether output passes as real.

**What Woodstitch does not do:** MOS, CMOS, MUSHRA or any 1–5 rating scale. These two
tasks are the whole product. If you need a mean opinion score, this is not the right tool;
if you need to know which of two outputs people prefer, or whether output passes as human,
it is. API studies are forced-choice; the `ties` fields in results are kept for schema
stability and are 0 on API studies.

`GET /v1/templates` returns the same thing as JSON, no key needed.

Base URL: `https://api.woodstitch.com/v1`
Auth: `Authorization: Bearer ws_live_...`
Machine-readable spec: [`/openapi.json`](https://woodstitch.com/openapi.json) (OpenAPI 3.1). This page as markdown: [`/docs.md`](https://woodstitch.com/docs.md).
MCP server for Claude Code, Cursor and other agents: `claude mcp add woodstitch -e WOODSTITCH_API_KEY=ws_live_... -- uvx woodstitch-mcp` ([woodstitch-mcp on PyPI](https://pypi.org/project/woodstitch-mcp/)).

---

## Read this first (especially if you are an AI agent)

**Always show the quote to the user before calling `/start`.** Step 1 is free and
tells you the exact cost. Step 2 spends real money and cannot be undone once
raters begin answering.

**Pass `max_total_usd` on every start.** It is a hard ceiling; the call fails
rather than overspending.

**Send `Idempotency-Key` on every POST.** If a call times out, retry with the
**same** key — you get the original response back instead of a second charge.

**Poll as often as you like.** `GET /v1/evals/{id}` is cheap and returns live
numbers. The first judgements usually land within minutes of starting, and the
study completes when the last one does, which depends on its size and how many
listeners are online. `poll_after_s` is a suggestion, not a limit.

**Duplicate protection is automatic.** Job identity is a hash of the audio
*content* plus the question and rating count. Re-running the same script with
different local paths returns the existing evaluation (`"duplicate": true`)
rather than buying it twice.

**Partial scores are published live.** `GET /v1/evals/{id}` returns the current
win rate, CI, and counts from whatever has been rated so far, flagged
`"provisional": true` with the n it came from. Poll it as often as you like.

Know what you are looking at: deciding to stop because the number crossed a
threshold you were watching is optional stopping, and it inflates false
positives badly — a test built for p<0.05 behaves more like p<0.2 under
continuous peeking. Treat an interim figure as a preview. Whether to act on one
is your call.

---

## No key yet? Start without one

`POST /v1/start` (no auth) creates an account in the background, puts the first-study credit on it, and
returns an `api_key` to use on every later call. Send `source` (your git remote) so the account has a name;
`email` is optional. Nothing is asked of the person. One account per call, rate-limited per address.

```bash
curl -sX POST https://api.woodstitch.com/v1/start -H "Content-Type: application/json" \
  -d '{"source": "github.com/your-org/your-repo"}'
# -> {"api_key":"ws_live_…","credit_usd":25,"next":"POST /v1/files, then POST /v1/evals …"}
```

To open the account in a browser later, attach an email with `POST /v1/account/email {"email": "…"}` and sign
in at https://woodstitch.com/app (a sign-in link is emailed). The step-by-step recipe an agent should follow is
[llms.txt](https://woodstitch.com/llms.txt).

---

## The whole flow

```bash
# 0. Upload your audio. Skip this if it already sits at public URLs. Any number of files per call.
curl -sX POST https://api.woodstitch.com/v1/files \
  -H "Authorization: Bearer $WOODSTITCH_KEY" \
  -F "file[]=@base/001.wav" -F "file[]=@cand/001.wav" -F "file[]=@base/002.wav" -F "file[]=@cand/002.wav"
# -> {"files":[{"id":"file_7c1e…","name":"001.wav","duration_s":12.4,"format":"wav"}, ...]}

# 1. Create — measures every clip, returns a price. CHARGES NOTHING.
curl -sX POST https://api.woodstitch.com/v1/evals \
  -H "Authorization: Bearer $WOODSTITCH_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: ckpt-42-create" \
  -d '{
    "name": "ckpt-42 vs baseline",
    "question": "Which audio sample do you prefer if you were listening to an audiobook?",
    "ratings_per_pair": 5,
    "pairs": [
      {"id":"utt-001","a":"file_7c1e…","b":"file_09aa…"},
      {"id":"utt-002","a":"file_b3d0…","b":"file_55f2…"}
    ]
  }'
# -> {"eval_id":7,"quote":{"id":3,"total_usd":5.80},"nothing_charged_yet":true,
#     "dashboard_url":"https://woodstitch.com/app/evaluations/7",...}

# 2. Start — THIS SPENDS MONEY. Confirm the amount with a human first.
curl -sX POST https://api.woodstitch.com/v1/evals/7/start \
  -H "Authorization: Bearer $WOODSTITCH_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: ckpt-42-start" \
  -d '{"quote_id":3,"max_total_usd":10.00}'

# 3. Poll as often as you like: live numbers, first judgements usually within minutes.
curl -s https://api.woodstitch.com/v1/evals/7 -H "Authorization: Bearer $WOODSTITCH_KEY"

# 4. Results
curl -s https://api.woodstitch.com/v1/evals/7/results -H "Authorization: Bearer $WOODSTITCH_KEY"
```

---

## Pricing

Free to check, no key needed:

```bash
curl -s https://api.woodstitch.com/v1/pricing
```

Priced per judgement, by **how many seconds of audio the rater hears** —
measured server-side from your audio, never from a number you declare, and
never rounded up to a band.

```
usd_per_judgement = audio_seconds / 3600 × 18.00      (rounded up to the cent)
```

| Template | `audio_seconds` is | 15 s clips | 30 s clips | 60 s clips |
|---|---|---|---|---|
| `tts_preference.v1` | `dur_a + dur_b` — the rater hears both clips | $0.15 | $0.30 | $0.60 |
| `tts_detect.v1` | the clip's own duration — one clip | $0.08 | $0.15 | $0.30 |

A 10 s + 50 s pair costs exactly what a 30 s + 30 s pair costs. Detection costs
half of preference at the same clip length because the rater hears one clip
instead of two.

**Every pair or clip is priced independently.** One long outlier does not
reprice the rest, and `dry_run` returns each unit's measured listening time
and its price so you can see exactly what you are buying before paying.

```
preference:  total = sum over pairs of (usd_per_judgement × ratings_per_pair)
detection:   total = sum over clips of (usd_per_judgement × ratings_per_clip)
```

200 pairs × 30 s clips × 5 ratings = 1,000 judgements = **$300.00**
600 clips × 30 s × 8 ratings = 4,800 judgements = **$720.00**

Clips must be 0.5–120 s. Up to 5,000 pairs or clips per evaluation.

### Billing is metered

You hold a balance and it drains as judgements arrive — there is no up-front
reservation of the whole study.

- Money is committed when a task is **handed to a rater**, not when the answer
  comes back, so a rater mid-task is always paid even if you cancel.
- `/start` needs a positive balance, not the whole quote. Run out of funds and
  the eval **pauses** (`pause_reason: insufficient_funds`) rather than failing.
  Top up and it resumes by itself.
- `POST /v1/evals/{id}/cancel` stops it. You are charged only for work already
  handed out; `max_additional_charge_if_canceled_now_usd` on the progress
  response tells you that figure before you decide.
- Abstentions are not charged.

How listeners are recruited, paid and screened is in the
[platform methodology](https://woodstitch.com/research/methodology). Rate
quality is the product.

---

## How many ratings do I need?

This is the question most people get wrong. Preference gaps are small, and
required sample size explodes as the gap narrows:

| True gap (Elo) | Expected preference | Judgements for 80% power |
|---|---|---|
| 50 | 57% | ~400 |
| 30 | 54% | ~1,100 |
| 20 | 53% | ~2,400 |
| 10 | 51.4% | ~9,500 |
| 2 | 50.3% | ~237,000 |

Those are idealized. Ties and repeated raters push the real number higher.
**If you are trying to resolve a sub-10-Elo difference, this is the wrong tool** —
no human panel will settle it affordably.

A good default for "is my new checkpoint better than baseline?" is
**200 pairs × 5 ratings**, which reliably detects moderate-to-large differences.

---

## Endpoints

### `POST /v1/files` — upload audio
Multipart. One file as `-F file=@x.wav`, or several as repeated `file[]` parts (up
to 100 files or 1 GB per call, 50 MB per file; a plain repeated `file=` keeps
only the last one, and the API tells you so). Each file is decoded, measured and
stored; you get back an id to use anywhere create takes a URL.

```bash
curl -sX POST https://api.woodstitch.com/v1/files \
  -H "Authorization: Bearer $WOODSTITCH_KEY" \
  -F "file[]=@base/001.wav" -F "file[]=@cand/001.wav"
```
```json
{"object":"list","count":2,"files":[
  {"id":"file_7c1e4f…","object":"file","name":"001.wav","duration_s":12.4,"format":"wav","bytes":1190456,"created_at":"2026-11-05T19:02:11Z"},
  {"id":"file_09aa31…","object":"file","name":"001.wav","duration_s":12.1,"format":"wav","bytes":1162004,"created_at":"2026-11-05T19:02:11Z"}
]}
```

A single-file call returns that file object directly (with the same `files`
list attached). Files are returned in the order you sent them and carry their
original `name`, so pairing two folders by filename is a one-liner. Ids are
private to your account. If one file in a call is bad, the call fails and
nothing from it is recorded; fix the file and resend the call. Re-sending the
same bytes is harmless: storage is content-addressed, so nothing is duplicated.

Also: `GET /v1/files` (your uploads, newest first), `GET /v1/files/{id}`, and
`DELETE /v1/files/{id}` to remove an upload from your account. Studies that
already used a file keep their copy.

### `POST /v1/evals` — create and price
Free. Resolves every clip (a file id or a URL it downloads), validates it
decodes, measures duration, stores an immutable copy, and returns a quote.

| Field | Required | Default | Notes |
|---|---|---|---|
| `template` | no | `tts_preference.v1` | Or `tts_detect.v1`. |
| `pairs` | preference | — | Array of `{id, a, b}`. Max 5000. `a`/`b` are file ids from `POST /v1/files` or public http(s) URLs. |
| `clips` | detection | — | Array of `{id, audio, truth, source, speaker, passage, duration_bucket}`. Max 5000. `audio` is a file id or a URL. |
| `ratings_per_pair` | no | 5 | 1–50. Independent judgements per pair. (`judgments_per_item` is accepted as an alias.) |
| `ratings_per_clip` | no | 5 | Detection's name for the same thing. |
| `pairs[].system_a`, `pairs[].system_b` | no | — | Labels for the two sides (model ids). Never shown to raters. Needed for `per_participant` rules on `system`. |
| `pairs[].passage`, `pairs[].speaker` | no | — | Text id and voice id, for no-repeat rules and your own analysis. |
| `per_participant` | no | — | No-repeat rules for one listener; see [Controlling who hears what](#controlling-who-hears-what). Works on both templates. |
| `allocation` | detection only | — | Study-wide balancing. Refused on preference studies, where every pair gets `ratings_per_pair`. |
| `question` | no | per template | Max 200 chars. Shown verbatim to raters. |
| `name` | no | "eval" | Your label. |
| `dry_run` | no | false | Price it and throw it away. Nothing is stored, no `eval_id`. |

See [the detection template](#the-detection-template-tts_detectv1) for what
each clip field means and why `speaker` and `passage` are required.

**Pricing without committing:** send `"dry_run": true` to get the quote and
`pair_seconds` — every pair's measured listening time and its price per
judgement — with nothing persisted. Use it to spot an over-long pair before
you commit.

The create response also carries a `warnings` array: it flags an underpowered
design, and flags any pair whose two sides are in different audio formats
(a WAV-vs-MP3 pair risks measuring your encoder instead of your model).

Audio: wav, mp3, flac, ogg, m4a. Each clip 0.5–120s, max 50MB. Give each clip as
a file id from `POST /v1/files` (uploaded from disk) or as a public http(s) URL.
A URL must stay reachable for the duration of this call only — we keep our own
copy. Redirects are not followed, so use direct or presigned links.

**Which side is which:** `a` and `b` are preserved exactly as you send them.
Raters see them in randomized order and are never told what either one is.

**Use the same format and bitrate on both sides.** If `a` is 48kHz WAV and `b`
is 128kbps MP3, you are partly measuring your encoder, not your model. Export
both sides identically.

**`pairs[].id`** must be unique within the evaluation, max 64 characters. It is
echoed back in `per_pair` so you can join results to your own data.

**Comparing several systems at once.** Send one pair per matchup, label the
sides with `system_a` / `system_b`, give each text a `passage` id, and add a
`per_participant` block so no listener hears one system often enough to
recognise it:

```json
"pairs": [
  {"id":"p01-x-y","a":"https://.../x_p01.wav","b":"https://.../y_p01.wav",
   "system_a":"x","system_b":"y","passage":"p01"}
],
"per_participant": {"max_judgements": 16, "unique_by": ["passage"], "max_per": {"system": 7}}
```

The result is still per pair; the export carries the labels so you can tally
wins per system yourself.

### `POST /v1/evals/{id}/start` — begin (spends money)
| Field | Required | Notes |
|---|---|---|
| `quote_id` | no | If given, must match the current quote, else `quote_mismatch`. |
| `max_total_usd` | no | Hard ceiling. Fails with `over_max` rather than overspending. |

Needs a positive balance and a key whose spend cap can cover the quote. Nothing
is debited here; billing is metered from the first judgement.

### `GET /v1/evals/{id}` — progress and live score
Returns the submitted `question`, `pairs`, `ratings_per_pair`, `cost_usd` and
`started_at`, plus `judgements.done` / `.target` / `.percent`,
`pairs_with_data`, and the same `metrics` / `counts` shape as `/results` —
computed from whatever has been rated so far.

`"provisional": true` until the evaluation completes. Status values: `ready`,
`running`, `completed`, `canceled`, `failed`.

Everything an evaluation needs to be resumed is in this response, so an agent
that restarts only needs the `eval_id`.

Every response that names an evaluation also carries `dashboard_url`, the
page where a person can watch the same study in the browser
(`https://woodstitch.com/app/evaluations/{id}`, sign-in required). Hand it to
the user when you report the `eval_id`.

### `GET /v1/evals/{id}/results` — final numbers
409 `not_ready` while the evaluation is still running. Available once it is
`completed`, or `canceled` — a canceled study returns whatever was judged
before the stop, with `"incomplete": true` and `judgements_target` so you can
see how far it got.

```json
{
  "metrics": {
    "human_pref/b_win_rate": 0.7222,
    "human_pref/b_win_rate_ci95_low": 0.6667,
    "human_pref/b_win_rate_ci95_high": 0.8333,
    "human_pref/tie_rate": 0.1111,
    "human_pref/pairs": 3,
    "human_pref/judgements": 9
  },
  "counts": {"a_wins": 2, "b_wins": 6, "ties": 1},
  "per_pair": [{"id":"utt-001","a_wins":1,"b_wins":2,"ties":0}]
}
```

`metrics` is a flat numeric dict, so it drops straight into a tracker:

```python
wandb.log(result["metrics"])
```

**Reading the score.** `b_win_rate` is `(b_wins + 0.5*ties) / total` — the share
of preference for **your `b` side**. `0.5` means indistinguishable. Ties count
half to each side, because a pair nobody can tell apart is evidence of parity,
not evidence for either one.

**The confidence interval is bootstrapped by pair, not by vote.** Multiple
judgements on the same pair are not independent, so a naive binomial interval
would be far too narrow. If the interval spans 0.5, you have not shown a
difference.

### `GET /v1/evals/{id}/export` — per-judgement rows
One row per judgement, with every clustering dimension: rater (pseudonymous,
stable within the eval), source, speaker, passage, duration bucket, answer,
confidence, time spent. Add `?format=csv` for CSV.

This is what you fit your own mixed-effects model on.

### `POST /v1/evals/{id}/cancel` — stop early
Charges you only for work already handed to raters. Anyone mid-task finishes
and is paid. Check `max_additional_charge_if_canceled_now_usd` on the progress
response first if you want to know that number before deciding.

### `POST /v1/start` — account + key + first-study credit, no auth
See [Start without one](#no-key-yet-start-without-one). Returns `api_key`, `credit_usd`, `account`.

### `POST /v1/account/email` — attach an email to the account
`{"email": "…"}`. Needed once before the dashboard sign-in link can find an account created by `/start` without an email.

### `POST /v1/account/topup` — add funds
A paused evaluation resumes automatically when the money lands.

### `GET /v1/account` — balance and spend
### `GET /v1/evals` — your last 50 evaluations
### `GET /v1/files` — your uploads · `GET /v1/files/{id}` · `DELETE /v1/files/{id}`
### `GET /v1/templates` — both templates, their inputs and metrics (no auth)
### `GET /v1/pricing` — rate card for both templates (no auth)

---

## The detection template — `tts_detect.v1`

A listener hears **one** clip and answers *"was this made by a human or AI?"*,
then rates their confidence 1–4. Unlike preference, this template has a right
answer, so you send ground truth with every clip.

### What you must send per clip

| Field | Required | Notes |
|---|---|---|
| `id` | yes | Unique within the eval. Echoed back in results. |
| `audio` | yes | Public http(s) URL. Same formats and limits as preference. |
| `truth` | yes | `"human"` or `"ai"`. **Never** sent to a listener. |
| `source` | yes* | The model id (`"eleven_v4"`). Real narration uses the reserved value `"human"` — and may omit the field. |
| `speaker` | yes | Voice id for AI, narrator id for human. |
| `passage` | yes | The text id. **Use the same id for every rendering of the same text, across every source.** |
| `duration_bucket` | no | Your design condition, e.g. `"15s"`, `"30s"`, `"60s"`. Free text, max 16 chars. |

`speaker` and `passage` are required because the result resamples over them to
work out its uncertainty. Without them there is no honest interval to report.

```bash
curl -sX POST https://api.woodstitch.com/v1/evals \
  -H "Authorization: Bearer $WOODSTITCH_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: realism-001" \
  -d '{
    "template": "tts_detect.v1",
    "name": "realism vs human narration",
    "ratings_per_clip": 8,
    "clips": [
      {"id":"h-p01","audio":"https://.../human_p01.wav",
       "truth":"human","source":"human","speaker":"narrator_03","passage":"p01","duration_bucket":"30s"},
      {"id":"e-p01","audio":"https://.../eleven_p01.wav",
       "truth":"ai","source":"eleven_v4","speaker":"rachel","passage":"p01","duration_bucket":"30s"}
    ]
  }'
```

### You need real human recordings in the same study

A fooling rate on its own means nothing: if 40% of your clips are called human,
that could be a realistic model or an inattentive panel, and you cannot tell
which. The real-human baseline is what separates the two, so include genuine
narration of the **same passages** and let the API compute the gap.

Create returns a warning if you leave it out, and no `baseline_deficit`.

### The result

```json
{
  "metric": "human_label_rate",
  "results": [
    {"source":"eleven_v4","duration":"30s","human_label_rate":0.41,"n":480,
     "ci95":{"tested_stimuli":{"low":0.37,"high":0.46},
             "model":        {"low":0.28,"high":0.55}},
     "baseline_deficit":0.37,
     "baseline_deficit_ci95":{"tested_stimuli":{"low":0.31,"high":0.43},
                              "model":        {"low":0.19,"high":0.54}}}
  ]
}
```

`human_label_rate` is the share of judgements that called the recording human.
For an AI source that is the fooling rate; for `source: "human"` it is the
recognition rate. `baseline_deficit` is the human rate minus this source's
rate — **0 means indistinguishable from real narration on these stimuli.**

### Two intervals, and why you must not mix them up

Every number comes with two confidence intervals:

| `scope` | Answers | Resamples |
|---|---|---|
| `tested_stimuli` | "What if a **different panel** heard these exact clips?" | raters |
| `model` | "What if I re-ran this with **different voices and passages** from the same model?" | raters, voices and passages together |

The `model` interval is typically 3–4× wider. That is not pessimism, it is the
cost of a claim about a model rather than about a set of files.

This was measured, not assumed. Simulated against a known ground truth on a
4-source / 5-voice / 30-passage / 400-rater design, at nominal 95%:

| interval | covers its own estimand | covers the model-level truth |
|---|---|---|
| `tested_stimuli` | 96.7% | **46.7%** |
| `model` | — | 93.8–96.0% |

**Quoting the narrow interval as a statement about a model is wrong more than
half the time.** If you are going to publish a sentence of the form *"model X
fools listeners Y% of the time"*, the `model` interval is the one that belongs
next to it. Use `tested_stimuli` when the claim really is about the specific
files you tested — a release gate on this build, for instance.

The response also carries a `generalisation` block stating in machine-readable
form what the data does and does not support, and `coverage` with the counts
behind it.

### More voices, not more ratings

If a source has few voices, buying more judgements will not fix the `model`
interval — between-voice variance is what makes it wide, and only more voices
reduce it. With fewer than 8 voices the response says
`"to_the_model": "not_established"` and tells you so in `warnings`.

The honest way to report a 5-voice study is *"these five voices"*, not
*"this model"*.

### Controlling who hears what

By default every clip gets `ratings_per_clip` judgements and any rater may be
served any clip. Two optional blocks change that. `per_participant` works on
both templates (for preference, the `system` field reads the pair's
`system_a` / `system_b` labels); `allocation` is detection only. They are named for their
**scope**, which is the thing to get right: a rule either applies across the
whole study or within one participant.

```json
{
  "template": "tts_detect.v1",
  "clips": [ ... ],

  "allocation": {
    "total_judgements": 2400,
    "balance_by": ["truth", "duration_bucket"]
  },

  "per_participant": {
    "max_judgements": 12,
    "counts": { "truth": { "human": 6, "ai": 6 } },
    "unique_by": ["speaker", "passage"]
  }
}
```

**`allocation`** — study-wide targets.

| Field | Means |
|---|---|
| `total_judgements` | How many judgements to buy in total. |
| `balance_by` | Target **equal completed judgements** in each combination of these fields. |

`balance_by` is what lets a small group of clips carry as much weight as a
large one. With 36 human clips and 216 AI clips, `balance_by: ["truth"]` gives
each human clip ~33 judgements and each AI clip ~6 — so the human baseline is
estimated as precisely as the models, which a single uniform count cannot do.

**Balance on every field whose comparison you care about.** A field you leave
out will drift, because the remainder from integer division lands wherever it
happens to land. If the study compares durations, name `duration_bucket`.

**`per_participant`** — one person's experience.

| Field | Means |
|---|---|
| `max_judgements` | Most clips this rater will ever hear in this study. Not per sitting — per person. |
| `counts` | Required mix within that, e.g. 6 human and 6 AI. Must sum to `max_judgements`. |
| `unique_by` | Fields that may not repeat for a rater. |

`unique_by` is the one that protects a repeated-measures design. If a rater
hears voice V at 60 seconds and again at 15 seconds, they recognise the voice
and carry over their earlier answer — and because longer clips are more
recognisable, that contamination flows from the long arm into the short one
and flattens the very effect being measured. `unique_by: ["speaker",
"passage"]` prevents it. Valid fields: `speaker`, `passage`, `source`,
`truth`, `duration_bucket`, `system`.

**`max_per`** caps how often a rater may meet the same value of a field, where
`unique_by` is the special case of a cap of 1. It exists because an A/B field
like `system` is too scarce to be unique — five systems make only two disjoint
pairs — so `"max_per": {"system": 7}` lets a rater work through the study while
stopping anyone hearing one system often enough to start recognising it.

These two are independent: no repeated speaker **and** no repeated passage,
not merely no repeated speaker-passage pair.

**Per-participant balance is not inferred from global balance.** They are
different designs. If each rater must hear 6 human and 6 AI, say so in
`counts`; `balance_by` alone only balances the totals.

### What create tells you

The response carries `effective_design` — the compiled plan, so you can check
what you are buying before paying:

```json
"effective_design": {
  "judgements_total": 2400,
  "ratings_per_clip": { "min": 6, "max": 34 },
  "judgements_by": {
    "truth": { "human": 1200, "ai": 1200 },
    "duration_bucket": { "15s": 800, "30s": 800, "60s": 800 }
  },
  "per_participant": { "max_judgements": 12, ... }
}
```

An impossible design is refused at create time rather than stalling halfway
through a paid study. If `unique_by` includes `passage` and a rater needs 12
clips but you supplied 8 passages, that is `infeasible_session` with the
numbers in the message.

### Expect a shortfall

These constraints mean some raters cannot be given a valid session and are
simply turned away — a safe outcome, because the alternative is serving a clip
that violates the design.

Simulated on a 12-passage, 6-voice, 3-duration study: **91% of target at 200
raters, 96% at 250**, and flat after that. More raters past that point mostly
get declined. Plan for a shortfall and read the completed counts in the
results rather than assuming every target is met.

### Confidence

Raters pick 1 (guessing) to 4 (very confident). It is **exploratory only**: it
never affects rater pay, never filters a judgement, never gates quality — and
raters are told that, which is the only reason the numbers mean anything.
Results include an accuracy-by-confidence breakdown as a calibration check.

A listener who cannot judge (audio failed) records an **abstention**, not
confidence 1. Abstentions are reported separately, the rater is still paid, and
**you are not charged** — the clip goes back out to someone else.

### Fit your own model

`GET /v1/evals/{id}/export` returns one row per judgement with every clustering
dimension (rater, speaker, passage, duration, answer, confidence, timing).
Add `?format=csv` for CSV. Rater ids are pseudonyms, stable inside the eval.

If you want the hierarchical logistic model this API does **not** fit, that
export is what you fit it on.

### Design checks before you pay

Create runs these and returns anything it finds in `warnings`:

- no human baseline, or no AI clips at all
- lopsided stimulus counts (a baseline resting on far fewer recordings)
- a source with only one voice — that result describes the voice, not the model
- fewer than 8 voices per source, or fewer than 10 passages
- **no passage shared by every source** — sources then get compared on different
  text, and an easy-text model simply looks more human
- mixed audio formats, which in a detection study is the most dangerous
  confound there is: listeners learn to hear the codec, not the voice
- too few judgements to separate anything

Run with `"dry_run": true` first to see all of it for free.

---

## Errors

Always:

```json
{"error": {"code": "insufficient_funds", "message": "Balance is $0.00. Add funds to begin.", "hint": "Top up at https://woodstitch.com/app"}}
```

| Code | HTTP | Meaning |
|---|---|---|
| `no_key` / `bad_key` | 401 | Missing or invalid `Authorization: Bearer`. |
| `insufficient_funds` | 402 | Balance is empty. Billing is metered, so `/start` needs a positive balance, not the whole quote; a running study pauses instead if it runs dry. |
| `spend_cap` | 402 | This key's cap would be exceeded. |
| `over_max` | 400 | Quote exceeds your `max_total_usd`. |
| `idempotency_conflict` | 409 | Key reused with a different body. |
| `not_ready` | 409 | Results requested while still running. |
| `bad_allocation` | 400 | `allocation` sent on a preference study, or malformed. |
| `bad_audio` | 400 | Clip won't decode, or is outside 0.5–120s. |
| `fetch_failed` | 400 | We couldn't download the URL. |
| `bad_file` | 400 | A file id that is not on this account, or whose audio is gone. |
| `no_files` / `upload_failed` / `too_many_files` | 400 | Nothing usable in a `POST /v1/files` call; the message names the file. |
| `upload_too_large` | 413 | A call over 1 GB, or a file over 50 MB. Send fewer files per call. |
| `quote_expired` | 400 | Quotes last 24h. Create again. |

---

## What raters actually do

Sign in, hear clip A and clip B **in full** (the choice is locked until both
finish playing), and pick A, B, or "sounds the same." They're paid per answer.
They are never told what the systems are, and side order is randomized per
rater, so position bias can't favour either side.