# Woodstitch

> Paid human listeners judge text-to-speech audio blind. Two tasks: pairwise
> preference (two clips of the same text, which do people prefer) and
> human-vs-AI detection (one clip, does it pass as human). Not offered: MOS,
> CMOS, MUSHRA or any 1-5 scale. Plain HTTPS API, base URL
> https://api.woodstitch.com/v1. Price: $18 per hour of audio a listener
> hears, per judgement, measured from your clips. Quote first, charge only
> when the study starts.

## If you were asked to "set up a Woodstitch listening test"

You are an agent in someone's repo. Do the whole thing here; no sign-up, no
key to copy, no form. The person expects to see a plan and a price, say yes,
and get a win rate back. Steps:

1. **Work out the comparison from the repo.** Find which TTS providers,
   models or voices the project uses (config, env names, SDK calls, recent
   commits). The usual test is the voice the product uses now versus one
   alternative the repo can also render (the previous provider, a second
   configured voice, or a different model or setting). If the person named
   the two sides, use those. If only one side can be rendered, say so and
   ask which alternative to use before doing anything else.
2. **Pick 20 lines the product actually speaks.** Real strings from the
   codebase or content, 1-3 sentences each, ideally 10-40 seconds of audio.
   Fewer is fine for a first look; the quote scales with audio length.
3. **Render every line with both sides** using the project's own code or
   keys. Same text, same format and sample rate on both sides (a WAV-vs-MP3
   pair measures the encoder, not the voice). Never upload or send provider
   secrets to Woodstitch.
4. **Get a key.** One call, nothing asked of the person:

       curl -s -X POST https://api.woodstitch.com/v1/start \
         -H "Content-Type: application/json" \
         -d '{"source": "<git remote origin, e.g. github.com/org/repo>"}'

   Response: `api_key` (store it in the project environment as
   `WOODSTITCH_API_KEY`; shown once) and `credit_usd` (25). `email` is
   optional; include the user's email only if they give it. Send the key as
   `Authorization: Bearer <key>` from here on.
5. **Upload** the clips: `POST /v1/files`, multipart, repeated `file[]`
   parts (up to 100 per call). Each returns an id.
6. **Create and price (charges nothing):**

       POST /v1/evals
       {"template": "tts_preference.v1",
        "name": "<project>: <A> vs <B>",
        "question": "Which would you rather listen to?",
        "ratings_per_pair": 5,
        "pairs": [{"id": "l01", "a": "file_…", "b": "file_…", "system_a": "<A>", "system_b": "<B>", "passage": "l01"}, …],
        "per_participant": {"unique_by": ["passage"]}}

   Add header `Idempotency-Key: <uuid>` to every POST; a retry with the same
   key returns the same response instead of acting twice. The response has
   `quote.total_usd`, `eval_id`, `warnings` and `nothing_charged_yet: true`.
   20 pairs of ~20 s at 5 ratings is about $20.
7. **Show the person the plan and the price** (what is compared, how many
   lines, how many listeners, the quote, what the $25 credit covers) and
   wait for a yes. Do not start without it.
8. **Start:** `POST /v1/evals/{id}/start` with `{"max_total_usd": <quote>}`.
   This spends the credit. The ceiling is enforced server-side.
9. **Results:** `GET /v1/evals/{id}` any time (provisional while running;
   first judgements within minutes, 100 judgements usually within hours).
   When `status` is `completed`, `GET /v1/evals/{id}/results` gives
   `human_pref/b_win_rate` with a 95% interval: 0.5 means indistinguishable,
   an interval that spans 0.5 does not establish a preference.
   `GET /v1/evals/{id}/export?format=csv` is every judgement. Tell the person
   the key is saved and they can ask for the result later.

Full reference: https://woodstitch.com/docs.md (same page rendered:
https://woodstitch.com/docs). Machine-readable spec:
https://woodstitch.com/openapi.json. Each call's errors carry a `hint`.

## Everything else

- Any later study: same endpoints with the saved key. Balance and spend:
  `GET /v1/account`. Add funds: sign in at https://woodstitch.com/app. If
  the account was started without an email, attach one first with
  `POST /v1/account/email {"email": "..."}` so the sign-in link can find it.
- `GET /v1/templates` and `GET /v1/pricing` need no key.
- Detection studies (`tts_detect.v1`): one clip per item with `truth`
  human/ai, `source`, `speaker`, `passage`; include real human recordings of
  the same passages as the baseline. Details in docs.md.
- Who the listeners are, how they are paid and screened, how audio is kept:
  https://woodstitch.com/research/methodology. Client audio is never
  published or used for training.
- MCP server (optional, same API): `uvx woodstitch-mcp` with
  WOODSTITCH_API_KEY set; Claude Code plugin:
  `/plugin marketplace add woodstitch-llc/claude-plugin` then
  `/plugin install tts-human-eval@woodstitch`.
- Published studies run on the same panel: https://woodstitch.com/research/
- Questions: support@woodstitch.com