Methodology
One protocol for every use case. What differs is the scripts, the listener question and, for phone, the audio condition.
Entries and voice selection
An entry is a vendor model; it is evaluated through up to three stock voices per use case. Unless the vendor nominates, voices are chosen by a fixed rule: the vendor’s quickstart default, then the voices the vendor itself recommends for that use case in its docs or voice library, then its usage ranking, then a seeded random sample (seed 20270101). US-English voices only, no gender quota, no subjective screening by Woodstitch. Every voice page records why it was chosen.
Rendering
Every clip is rendered once by Woodstitch from the vendor’s public API, or from the published checkpoint for open-weight models, at default settings, and frozen. Technical retries only (HTTP errors, empty or truncated audio), all logged. Each clip is loudness-matched with one static gain to a common integrated loudness (BS.1770), no compression, true-peak ceiling −1 dBTP. Phone clips first pass through 300–3400 Hz band-limiting, 8 kHz resampling and G.711 μ-law.
Listeners
Paid US listeners, vetted on device and network signals and identity-verified before payout, judging blind pairs in the browser with a forced choice. Left/right order is balanced.
Design and statistics
Each voice meets 12 voices from other vendors, six judgments per pair, each on a different one of 12 scripts. Scores are Bradley–Terry log-strengths with a weak prior; intervals and rank ranges come from 200 bootstrap resamples of listeners and scripts. The model score is the mean, over its evaluated voices, of their Bradley–Terry strengths, shown on an Elo scale where 1000 is the average voice in the use case. There are no tiers.
Nominations and disputes
Any vendor can nominate up to three voices per use case per edition. A disputed render is re-rendered under the same protocol and both results stay on the record.
Editions and data
While an edition is collecting, this page shows preliminary results that update as judgments arrive. When it is frozen, the edition is published with the scripts, every evaluated clip we hold the rights to publish, per-judgment data and each voice’s provenance.