Paid human listeners judge your audio and give you a number back. Use it when automated metrics (WER, speaker-sim, predicted MOS) can't answer the question.
Two templates, chosen with "template" on create:
template | A listener hears | and answers | you get |
|---|---|---|---|
tts_preference.v1 (default) | two clips | which one they prefer, or tie | preference for side B, 0.5 = no preference |
tts_detect.v1 | one clip | human or AI, plus confidence 1–4 | share of clips taken for human, vs a real-human baseline |
Use preference to pick between two systems or checkpoints. Use detection to find out whether output passes as real.
GET /v1/templates returns the same thing as JSON, no key needed.
Base URL: https://api.woodstitch.com/v1 Auth: Authorization: Bearer ws_live_...
Always show the quote to the user before calling /start. Step 1 is free and tells you the exact cost. Step 2 spends real money and cannot be undone once raters begin answering.
Pass max_total_usd on every start. It is a hard ceiling; the call fails rather than overspending.
Send Idempotency-Key on every POST. If a call times out, retry with the same key — you get the original response back instead of a second charge.
Do not poll in a tight loop. A typical evaluation takes hours. Record the eval_id, tell the user, and move on. Poll at most every ~15 minutes (poll_after_s tells you when).
Duplicate protection is automatic. Job identity is a hash of the audio content plus the question and rating count. Re-running the same script with different local paths returns the existing evaluation ("duplicate": true) rather than buying it twice.
Partial scores are published live. GET /v1/evals/{id} returns the current win rate, CI, and counts from whatever has been rated so far, flagged "provisional": true with the n it came from. Poll it as often as you like.
Know what you are looking at: deciding to stop because the number crossed a threshold you were watching is optional stopping, and it inflates false positives badly — a test built for p<0.05 behaves more like p<0.2 under continuous peeking. Treat an interim figure as a preview. Whether to act on one is your call.
# 1. Create — downloads your audio, measures it, returns a price. CHARGES NOTHING.
curl -sX POST https://api.woodstitch.com/v1/evals \
-H "Authorization: Bearer $WOODSTITCH_KEY" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: ckpt-42-create" \
-d '{
"name": "ckpt-42 vs baseline",
"question": "Which audio sample do you prefer if you were listening to an audiobook?",
"ratings_per_pair": 5,
"pairs": [
{"id":"utt-001","a":"https://your-bucket/base/001.wav","b":"https://your-bucket/cand/001.wav"},
{"id":"utt-002","a":"https://your-bucket/base/002.wav","b":"https://your-bucket/cand/002.wav"}
]
}'
# -> {"eval_id":7,"quote":{"id":3,"total_usd":5.80},"nothing_charged_yet":true,...}
# 2. Start — THIS SPENDS MONEY. Confirm the amount with a human first.
curl -sX POST https://api.woodstitch.com/v1/evals/7/start \
-H "Authorization: Bearer $WOODSTITCH_KEY" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: ckpt-42-start" \
-d '{"quote_id":3,"max_total_usd":10.00}'
# 3. Poll — humans take HOURS. Don't sit in a loop; check back later.
curl -s https://api.woodstitch.com/v1/evals/7 -H "Authorization: Bearer $WOODSTITCH_KEY"
# 4. Results
curl -s https://api.woodstitch.com/v1/evals/7/results -H "Authorization: Bearer $WOODSTITCH_KEY"Free to check, no key needed:
curl -s https://api.woodstitch.com/v1/pricingPriced per judgement, by how long a rater listens — measured server-side from your audio, never from a number you declare.
| Listening time | tts_preference.v1 | tts_detect.v1 |
|---|---|---|
0 < d <= 10s | $0.25 | $0.16 |
10 < d <= 30s | $0.58 | $0.32 |
30 < d <= 60s | $1.08 | $0.58 |
60 < d <= 120s | $2.09 | $1.08 |
Detection costs less at the same clip length because the rater hears one clip instead of two.
Which band a unit falls in:
- preference — the pair's two clips together, as (dur_a + dur_b) / 2. A rater hears both, so a 10s + 50s pair costs exactly what a 30s + 30s pair costs. - detection — the clip's own duration.
Boundaries are inclusive at the top: exactly 30.0s is the $0.58 band, 30.1s is $1.08.
Every pair or clip is priced independently. One long outlier does not reprice the rest, and dry_run returns each unit's measured duration and rate so you can see exactly where you sit before paying.
preference: total = sum over pairs of (rate × ratings_per_pair)
detection: total = sum over clips of (rate × ratings_per_clip)200 pairs × 25s clips × 5 ratings = 1,000 judgements = $580.00 600 clips × 30s × 8 ratings = 4,800 judgements = $1,536.00
You hold a balance and it drains as judgements arrive — there is no up-front reservation of the whole study.
- Money is committed when a task is handed to a rater, not when the answer comes back, so a rater mid-task is always paid even if you cancel. - Run out of funds and the eval pauses (pause_reason: insufficient_funds) rather than failing. Top up and it resumes by itself. - POST /v1/evals/{id}/cancel stops it. You are charged only for work already handed out; max_additional_charge_if_canceled_now_usd on the progress response tells you that figure before you decide. - Abstentions are not charged.
Raters are paid $15/hour of listening — above the ~$8/hr floor the large panels enforce. Rate quality is the product.
This is the question most people get wrong. Preference gaps are small, and required sample size explodes as the gap narrows:
| True gap (Elo) | Expected preference | Judgements for 80% power |
|---|---|---|
| 50 | 57% | ~400 |
| 30 | 54% | ~1,100 |
| 20 | 53% | ~2,400 |
| 10 | 51.4% | ~9,500 |
| 2 | 50.3% | ~237,000 |
Those are idealized. Ties and repeated raters push the real number higher. If you are trying to resolve a sub-10-Elo difference, this is the wrong tool — no human panel will settle it affordably.
A good default for "is my new checkpoint better than baseline?" is 200 pairs × 5 ratings, which reliably detects moderate-to-large differences.
POST /v1/evals — create and priceFree. Downloads every clip, validates it decodes, measures duration, stores an immutable copy, and returns a quote.
| Field | Required | Default | Notes |
|---|---|---|---|
template | no | tts_preference.v1 | Or tts_detect.v1. |
pairs | preference | — | Array of {id, a, b}. Max 5000. Public http(s) URLs. |
clips | detection | — | Array of {id, audio, truth, source, speaker, passage, duration_bucket}. Max 5000. |
ratings_per_pair | no | 5 | 1–50. Independent judgements per pair. |
ratings_per_clip | no | 5 | Detection's name for the same thing. |
question | no | per template | Max 200 chars. Shown verbatim to raters. |
name | no | "eval" | Your label. |
dry_run | no | false | Price it and throw it away. Nothing is stored, no eval_id. |
See the detection template for what each clip field means and why speaker and passage are required.
Pricing without committing: send "dry_run": true to get the quote, a per-band tally, and pair_seconds — every pair's measured duration and its rate — with nothing persisted. Use it to find pairs sitting just over a band boundary before you commit.
The create response also carries a warnings array: it flags an underpowered design, and flags any pair whose two sides are in different audio formats (a WAV-vs-MP3 pair risks measuring your encoder instead of your model).
Audio: wav, mp3, flac, ogg, m4a. Each clip 0.5–120s, max 50MB. URLs must stay reachable for the duration of this call only — we keep our own copy.
Which side is which: a and b are preserved exactly as you send them. Raters see them in randomized order and are never told what either one is.
Use the same format and bitrate on both sides. If a is 48kHz WAV and b is 128kbps MP3, you are partly measuring your encoder, not your model. Export both sides identically.
pairs[].id must be unique within the evaluation, max 64 characters. It is echoed back in per_pair so you can join results to your own data.
POST /v1/evals/{id}/start — begin (spends money)| Field | Required | Notes |
|---|---|---|
quote_id | no | If given, must match the current quote, else quote_mismatch. |
max_total_usd | no | Hard ceiling. Fails with over_max rather than overspending. |
GET /v1/evals/{id} — progress and live scoreReturns the submitted question, pairs, ratings_per_pair, cost_usd and started_at, plus judgements.done / .target / .percent, pairs_with_data, and the same metrics / counts shape as /results — computed from whatever has been rated so far.
"provisional": true until the evaluation completes. Status values: ready, running, completed, canceled, failed.
Everything an evaluation needs to be resumed is in this response, so an agent that restarts only needs the eval_id.
GET /v1/evals/{id}/results — final numbers409 not_ready until complete.
{
"metrics": {
"human_pref/b_win_rate": 0.7222,
"human_pref/b_win_rate_ci95_low": 0.6667,
"human_pref/b_win_rate_ci95_high": 0.8333,
"human_pref/tie_rate": 0.1111,
"human_pref/pairs": 3,
"human_pref/judgements": 9
},
"counts": {"a_wins": 2, "b_wins": 6, "ties": 1},
"per_pair": [{"id":"utt-001","a_wins":1,"b_wins":2,"ties":0}]
}metrics is a flat numeric dict, so it drops straight into a tracker:
wandb.log(result["metrics"])Reading the score. b_win_rate is (b_wins + 0.5*ties) / total — the share of preference for your b side. 0.5 means indistinguishable. Ties count half to each side, because a pair nobody can tell apart is evidence of parity, not evidence for either one.
The confidence interval is bootstrapped by pair, not by vote. Multiple judgements on the same pair are not independent, so a naive binomial interval would be far too narrow. If the interval spans 0.5, you have not shown a difference.
GET /v1/evals/{id}/export — per-judgement rowsOne row per judgement, with every clustering dimension: rater (pseudonymous, stable within the eval), source, speaker, passage, duration bucket, answer, confidence, time spent. Add ?format=csv for CSV.
This is what you fit your own mixed-effects model on.
POST /v1/evals/{id}/cancel — stop earlyCharges you only for work already handed to raters. Anyone mid-task finishes and is paid. Check max_additional_charge_if_canceled_now_usd on the progress response first if you want to know that number before deciding.
POST /v1/account/topup — add fundsA paused evaluation resumes automatically when the money lands.
GET /v1/account — balance and spendGET /v1/evals — your last 50 evaluationsGET /v1/templates — both templates, their inputs and metrics (no auth)GET /v1/pricing — rate card for both templates (no auth)tts_detect.v1A listener hears one clip and answers "was this made by a human or AI?", then rates their confidence 1–4. Unlike preference, this template has a right answer, so you send ground truth with every clip.
| Field | Required | Notes |
|---|---|---|
id | yes | Unique within the eval. Echoed back in results. |
audio | yes | Public http(s) URL. Same formats and limits as preference. |
truth | yes | "human" or "ai". Never sent to a listener. |
source | yes* | The model id ("eleven_v4"). Real narration uses the reserved value "human" — and may omit the field. |
speaker | yes | Voice id for AI, narrator id for human. |
passage | yes | The text id. Use the same id for every rendering of the same text, across every source. |
duration_bucket | no | Your design condition, e.g. "15s", "30s", "60s". Free text, max 16 chars. |
speaker and passage are required because the result resamples over them to work out its uncertainty. Without them there is no honest interval to report.
curl -sX POST https://api.woodstitch.com/v1/evals \
-H "Authorization: Bearer $WOODSTITCH_KEY" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: realism-001" \
-d '{
"template": "tts_detect.v1",
"name": "realism vs human narration",
"ratings_per_clip": 8,
"clips": [
{"id":"h-p01","audio":"https://.../human_p01.wav",
"truth":"human","source":"human","speaker":"narrator_03","passage":"p01","duration_bucket":"30s"},
{"id":"e-p01","audio":"https://.../eleven_p01.wav",
"truth":"ai","source":"eleven_v4","speaker":"rachel","passage":"p01","duration_bucket":"30s"}
]
}'A fooling rate on its own means nothing: if 40% of your clips are called human, that could be a realistic model or an inattentive panel, and you cannot tell which. The real-human baseline is what separates the two, so include genuine narration of the same passages and let the API compute the gap.
Create returns a warning if you leave it out, and no baseline_deficit.
{
"metric": "human_label_rate",
"results": [
{"source":"eleven_v4","duration":"30s","human_label_rate":0.41,"n":480,
"ci95":{"tested_stimuli":{"low":0.37,"high":0.46},
"model": {"low":0.28,"high":0.55}},
"baseline_deficit":0.37,
"baseline_deficit_ci95":{"tested_stimuli":{"low":0.31,"high":0.43},
"model": {"low":0.19,"high":0.54}}}
]
}human_label_rate is the share of judgements that called the recording human. For an AI source that is the fooling rate; for source: "human" it is the recognition rate. baseline_deficit is the human rate minus this source's rate — 0 means indistinguishable from real narration on these stimuli.
Every number comes with two confidence intervals:
scope | Answers | Resamples |
|---|---|---|
tested_stimuli | "What if a different panel heard these exact clips?" | raters |
model | "What if I re-ran this with different voices and passages from the same model?" | raters, voices and passages together |
The model interval is typically 3–4× wider. That is not pessimism, it is the cost of a claim about a model rather than about a set of files.
This was measured, not assumed. Simulated against a known ground truth on a 4-source / 5-voice / 30-passage / 400-rater design, at nominal 95%:
| interval | covers its own estimand | covers the model-level truth |
|---|---|---|
tested_stimuli | 96.7% | 46.7% |
model | — | 93.8–96.0% |
Quoting the narrow interval as a statement about a model is wrong more than half the time. If you are going to publish a sentence of the form "model X fools listeners Y% of the time", the model interval is the one that belongs next to it. Use tested_stimuli when the claim really is about the specific files you tested — a release gate on this build, for instance.
The response also carries a generalisation block stating in machine-readable form what the data does and does not support, and coverage with the counts behind it.
If a source has few voices, buying more judgements will not fix the model interval — between-voice variance is what makes it wide, and only more voices reduce it. With fewer than 8 voices the response says "to_the_model": "not_established" and tells you so in warnings.
The honest way to report a 5-voice study is "these five voices", not "this model".
By default every clip gets ratings_per_clip judgements and any rater may be served any clip. Two optional blocks change that. They are named for their scope, which is the thing to get right: a rule either applies across the whole study or within one participant.
{
"template": "tts_detect.v1",
"clips": [ ... ],
"allocation": {
"total_judgements": 2400,
"balance_by": ["truth", "duration_bucket"]
},
"per_participant": {
"max_judgements": 12,
"counts": { "truth": { "human": 6, "ai": 6 } },
"unique_by": ["speaker", "passage"]
}
}allocation — study-wide targets.
| Field | Means |
|---|---|
total_judgements | How many judgements to buy in total. |
balance_by | Target equal completed judgements in each combination of these fields. |
balance_by is what lets a small group of clips carry as much weight as a large one. With 36 human clips and 216 AI clips, balance_by: ["truth"] gives each human clip ~33 judgements and each AI clip ~6 — so the human baseline is estimated as precisely as the models, which a single uniform count cannot do.
Balance on every field whose comparison you care about. A field you leave out will drift, because the remainder from integer division lands wherever it happens to land. If the study compares durations, name duration_bucket.
per_participant — one person's experience.
| Field | Means |
|---|---|
max_judgements | Most clips this rater will ever hear in this study. Not per sitting — per person. |
counts | Required mix within that, e.g. 6 human and 6 AI. Must sum to max_judgements. |
unique_by | Fields that may not repeat for a rater. |
unique_by is the one that protects a repeated-measures design. If a rater hears voice V at 60 seconds and again at 15 seconds, they recognise the voice and carry over their earlier answer — and because longer clips are more recognisable, that contamination flows from the long arm into the short one and flattens the very effect being measured. unique_by: ["speaker", "passage"] prevents it. Valid fields: speaker, passage, source, truth, duration_bucket.
These two are independent: no repeated speaker and no repeated passage, not merely no repeated speaker-passage pair.
Per-participant balance is not inferred from global balance. They are different designs. If each rater must hear 6 human and 6 AI, say so in counts; balance_by alone only balances the totals.
The response carries effective_design — the compiled plan, so you can check what you are buying before paying:
"effective_design": {
"judgements_total": 2400,
"ratings_per_clip": { "min": 6, "max": 34 },
"judgements_by": {
"truth": { "human": 1200, "ai": 1200 },
"duration_bucket": { "15s": 800, "30s": 800, "60s": 800 }
},
"per_participant": { "max_judgements": 12, ... }
}An impossible design is refused at create time rather than stalling halfway through a paid study. If unique_by includes passage and a rater needs 12 clips but you supplied 8 passages, that is infeasible_session with the numbers in the message.
These constraints mean some raters cannot be given a valid session and are simply turned away — a safe outcome, because the alternative is serving a clip that violates the design.
Simulated on a 12-passage, 6-voice, 3-duration study: 91% of target at 200 raters, 96% at 250, and flat after that. More raters past that point mostly get declined. Plan for a shortfall and read the completed counts in the results rather than assuming every target is met.
Raters pick 1 (guessing) to 4 (very confident). It is exploratory only: it never affects rater pay, never filters a judgement, never gates quality — and raters are told that, which is the only reason the numbers mean anything. Results include an accuracy-by-confidence breakdown as a calibration check.
A listener who cannot judge (audio failed) records an abstention, not confidence 1. Abstentions are reported separately, the rater is still paid, and you are not charged — the clip goes back out to someone else.
GET /v1/evals/{id}/export returns one row per judgement with every clustering dimension (rater, speaker, passage, duration, answer, confidence, timing). Add ?format=csv for CSV. Rater ids are pseudonyms, stable inside the eval.
If you want the hierarchical logistic model this API does not fit, that export is what you fit it on.
Create runs these and returns anything it finds in warnings:
- no human baseline, or no AI clips at all - lopsided stimulus counts (a baseline resting on far fewer recordings) - a source with only one voice — that result describes the voice, not the model - fewer than 8 voices per source, or fewer than 10 passages - no passage shared by every source — sources then get compared on different text, and an easy-text model simply looks more human - mixed audio formats, which in a detection study is the most dangerous confound there is: listeners learn to hear the codec, not the voice - too few judgements to separate anything
Run with "dry_run": true first to see all of it for free.
Always:
{"error": {"code": "insufficient_funds", "message": "Need $580.00, balance is $100.00.", "hint": "Top up at https://woodstitch.com/billing"}}| Code | HTTP | Meaning |
|---|---|---|
no_key / bad_key | 401 | Missing or invalid Authorization: Bearer. |
insufficient_funds | 402 | Balance below the quote. |
spend_cap | 402 | This key's cap would be exceeded. |
over_max | 400 | Quote exceeds your max_total_usd. |
idempotency_conflict | 409 | Key reused with a different body. |
not_ready | 409 | Results requested before completion. |
bad_audio | 400 | Clip won't decode, or is outside 0.5–120s. |
fetch_failed | 400 | We couldn't download the URL. |
quote_expired | 400 | Quotes last 24h. Create again. |
Sign in, hear clip A and clip B in full (the choice is locked until both finish playing), and pick A, B, or "sounds the same." They're paid per answer. They are never told what the systems are, and side order is randomized per rater, so position bias can't favour either side.