Which TTS model is best for audiobooks?
Five commercial text-to-speech systems read the same sixteen passages from classic novels, each in its own stock narration voices. 78 paid listeners heard them in pairs and chose the one they would rather listen to. Report at the preregistered sample of 480 judgments. Listen before you read.
Listen first
Which reading would you rather keep listening to?
Two of the five systems read the same passage. Hear both, choose one, then see which system was which and how the study's listeners voted on this exact pair. The clips are the study's own stimuli, as the raters heard them: stock voices, one shared gain, nothing else touched.
All 20 published clips (four passages × five systems)
Four of the sixteen passages are published here; the remaining audio is in use in the running extension and is released when it closes (§5). Panel counts on a single pair are three judgments, so a 3–0 is a hint, not a verdict; the ranking in Table 1 pools all 481.
Question When five current text-to-speech systems read the same passage of an audiobook, each in a stock narration voice of the matching gender, which would ordinary listeners rather keep listening to?
Study ElevenLabs v4, Gemini 3.8 Flash TTS, Fish Audio Drama 3, Cartesia Sonic 3 and Qwen 3.1 TTS Flash each read sixteen passages of about 45 seconds, one from each of sixteen nineteenth-century novels, using four of the provider's own stock voices (two female, two male) cycled across the passages. Every clip was transcribed to confirm it read the text, resampled to one rate and given one shared gain. 78 paid, untrained listeners heard two readings of the same passage, blind to the system, and made a forced choice. Ten system pairs × sixteen passages = 160 items at three judgments each: 480 planned, 481 collected.
Findings ElevenLabs v4 won 56.2% of its pairings (95% CI 49 – 63 resampling listeners), Gemini 3.8 Flash TTS 54.4% (47 – 62) and Fish Audio Drama 3 52.6% (45 – 60). Those three cannot be separated: ElevenLabs is ranked first in 49% of bootstrap replicates, Gemini in 33% and Fish in 16%, and the head-to-head series between them are 23–25, 26–22 and 24–24. Cartesia Sonic 3 won 46.1% (38 – 54) and Qwen 3.1 TTS Flash 40.6% (34 – 48); Qwen lost its series against every other system and is last in 81% of replicates. The preregistered sample answers the question it was sized for, which system is best, with "one of three"; it was not sized to order those three.
Scope Stock voices, ~45-second passages of public-domain literary prose, lay listeners hearing each passage once. The result is listener preference between short readings, not long-form narration quality over hours, not voice cloning, and not the systems' best possible output with tuning.
Key findings
- Three systems share the top: ElevenLabs v4 (108 wins of 192, 56.2%), Gemini 3.8 Flash TTS (105 of 193, 54.4%) and Fish Audio Drama 3 (101 of 192, 52.6%). Each of the three finishes in the top three in at least 85% of listener-resampled replicates; none finishes first in more than half.
- Qwen 3.1 TTS Flash is last (78 of 192, 40.6%; CI 34 – 48): it lost every head-to-head series (15–33 to ElevenLabs, 17–31 to Gemini, 22–26 to Fish, 24–24 to Sonic) and ranks last in 81% of replicates.
- Cartesia Sonic 3 (46.1%; CI 38 – 54) sits between: below the leading three in 77% of replicates, above Qwen in 83%.
- The level of the clip did not drive the result. ElevenLabs, the quietest system after the shared gain (−26.6 dB RMS against −21.0 to −23.2 dB for the others), won most; the louder clip of a pair was chosen 45% of the time.
- Preference differed by voice gender in a way the study was not designed to test: ElevenLabs won 61% of pairings in male-voiced passages but 51% in female-voiced ones; Gemini the reverse (59% female, 50% male). Cells are ~96 judgments and these are observed differences, not established effects (§2.4).
1What we tested
Public text-to-speech leaderboards rank systems on short, varied utterances. An audiobook is a different job: tens of hours of continuous prose in one voice, where pacing, paragraph pauses, breath and the handling of long nineteenth-century sentences matter more than a punchy line. We wanted to know which of the current commercial systems a listener would rather hear reading a novel, measured the only way that question can be measured, by asking listeners who do not know which system they are hearing.
The question is which system to buy, so there is no human narrator arm; an earlier draft had one and it answered a different question. Each of five systems read the same sixteen passages in four of its own stock narration voices. Listeners heard two systems reading the same passage and chose the one they would rather listen to for a long audiobook. The primary outcome is each system's share of pairings won; the secondary outcomes are the head-to-head series and whether the result depends on the voice's gender or on the passage.
2Results
2.1 Three systems lead and cannot be ordered; one trails; one is last
Table 1 ranks the five systems by Bradley-Terry strength fitted to all 481 judgments; the win rate gives the same order. ElevenLabs v4 won 56.2% of its pairings (listener-resampled 95% CI 49 – 63; resampling listeners and passages together, 44 – 68), Gemini 3.8 Flash TTS 54.4% (47 – 62; 42 – 67) and Fish Audio Drama 3 52.6% (45 – 60; 39 – 65). All three intervals include each other's point estimates, and only ElevenLabs' listener-resampled interval does not include 50%, narrowly. ElevenLabs ranks first in 49% of replicates, Gemini in 33% and Fish in 16%: the order among them is unknown, which is different from their being equal. Cartesia Sonic 3 won 46.1% (38 – 54; 33 – 59). Qwen 3.1 TTS Flash won 40.6% (34 – 48; 28 – 54) and ranked last in 81% of replicates.
Side of presentation did not matter: the left-hand clip was chosen on 48.6% of judgments (CI 43 – 54). Each system was on the left in half of its items by design (§4.5).
Table 1. Ranking by Bradley-Terry strength. Win rate is wins over all pairings the system appeared in. The first 95% interval resamples listeners (3,000 replicates); the second resamples listeners and passages independently and is wider. Strength is on a natural-log scale, mean zero; a difference of 0.20 predicts the stronger system wins 55% of their pairings. P(1st) and P(last) are the shares of listener-resampled replicates in which the system ranked first or fifth.
| System | Provider | Won | Lost | Win rate | 95% CI (listeners) | 95% CI (listeners + passages) | Strength | P(1st) | P(last) |
|---|---|---|---|---|---|---|---|---|---|
| ElevenLabs v4 | ElevenLabs | 108 | 84 | 56.2% | 49 – 63 | 44 – 68 | +0.20 | 49% | 0% |
| Gemini 3.8 Flash TTS | 105 | 88 | 54.4% | 47 – 62 | 42 – 67 | +0.14 | 33% | 0% | |
| Fish Audio Drama 3 | Fish Audio | 101 | 91 | 52.6% | 45 – 60 | 39 – 65 | +0.08 | 16% | 2% |
| Cartesia Sonic 3 | Cartesia | 89 | 104 | 46.1% | 38 – 54 | 33 – 59 | -0.12 | 2% | 17% |
| Qwen 3.1 TTS Flash | Alibaba | 78 | 114 | 40.6% | 34 – 48 | 28 – 54 | -0.30 | 0% | 81% |
2.2 Head to head
Every system met every other on all sixteen passages, three times each, so each series is 48 judgments (one item received a fourth judgment, §4.5). Table 2 gives the series. Among the leading three the series are 23–25 (ElevenLabs over Gemini), 26–22 (ElevenLabs over Fish) and 24–24 (Gemini over Fish): each within four of even. The clear results are at the bottom: Qwen lost all four of its series, and ElevenLabs beat Qwen 33–15, the most lopsided series in the study.
Table 2. Head-to-head series, read as row wins – column wins over the judgments in which the two systems met.
| ElevenLabs | Gemini | Fish | Sonic | Qwen | |
|---|---|---|---|---|---|
| ElevenLabs v4 | — | 23–25 | 26–22 | 26–22 | 33–15 |
| Gemini 3.8 Flash TTS | 25–23 | — | 24–24 | 25–24 | 31–17 |
| Fish Audio Drama 3 | 22–26 | 24–24 | — | 29–19 | 26–22 |
| Cartesia Sonic 3 | 22–26 | 24–25 | 19–29 | — | 24–24 |
| Qwen 3.1 TTS Flash | 15–33 | 17–31 | 22–26 | 24–24 | — |
2.3 By passage
Table 3 gives each system's record on each passage. ElevenLabs had the best record on 7 of the sixteen passages, Gemini and Fish on 4 each and Sonic on 1; Qwen led none. A passage is only 30 judgments across the ten pairs, so a single passage's leader is a weak signal; the pattern that holds up is that no passage reversed the overall order at the bottom.
Table 3. Wins / pairings by passage. Each passage is one paragraph of 620–890 characters; "voice" is the gender of the narration role every system read it in.
| Passage | Voice | ElevenLabs | Gemini | Fish | Sonic | Qwen | Best record |
|---|---|---|---|---|---|---|---|
| Pride and Prejudice | female | 7/12 | 8/12 | 5/12 | 5/12 | 5/12 | Gemini |
| A Tale of Two Cities | female | 3/12 | 5/12 | 4/12 | 9/12 | 9/12 | Sonic |
| The Adventures of Sherlock Holmes | female | 5/12 | 5/13 | 8/12 | 6/13 | 7/12 | Fish |
| The Picture of Dorian Gray | female | 6/12 | 9/12 | 8/12 | 3/12 | 4/12 | Gemini |
| Moby Dick | female | 4/12 | 10/12 | 5/12 | 3/12 | 8/12 | Gemini |
| Frankenstein | female | 8/12 | 8/12 | 6/12 | 2/12 | 6/12 | ElevenLabs |
| Dracula | female | 7/12 | 5/12 | 6/12 | 6/12 | 6/12 | ElevenLabs |
| The Strange Case of Dr Jekyll | female | 9/12 | 7/12 | 8/12 | 3/12 | 3/12 | ElevenLabs |
| Wuthering Heights | male | 8/12 | 6/12 | 9/12 | 6/12 | 1/12 | Fish |
| Emma | male | 6/12 | 6/12 | 9/12 | 7/12 | 2/12 | Fish |
| Sense and Sensibility | male | 6/12 | 9/12 | 4/12 | 5/12 | 6/12 | Gemini |
| Great Expectations | male | 9/12 | 3/12 | 5/12 | 7/12 | 6/12 | ElevenLabs |
| Oliver Twist | male | 9/12 | 6/12 | 3/12 | 8/12 | 4/12 | ElevenLabs |
| David Copperfield | male | 6/12 | 6/12 | 9/12 | 7/12 | 2/12 | Fish |
| War and Peace | male | 7/12 | 7/12 | 5/12 | 5/12 | 6/12 | ElevenLabs |
| Anna Karenina | male | 8/12 | 5/12 | 7/12 | 7/12 | 3/12 | ElevenLabs |
2.4 By voice gender (exploratory)
Each system read eight passages in female voices and eight in male voices, two voices of each, so every system's score pools four voices and the comparison is between providers' voice catalogues as much as between models. Splitting the judgments by the gender of the role (Table 4) moves the order: in male-voiced passages ElevenLabs won 61% and Gemini 50%; in female-voiced passages Gemini won 59% and ElevenLabs 51%. Sonic's record was 54% in male voices and 38% in female; Qwen's 50% and 31%. These halves are about 96 pairings each with intervals of ±10 points, the split was examined after the ranking was known, and the study did not balance voices within gender; the sensible reading is that a buyer's result depends on which of a provider's voices they pick at least as much as on the provider, which is an argument for testing the specific voice rather than the brand.
Table 4. Win rate by the gender of the narration role, with Wilson 95% intervals on the raw counts.
| System | Female roles | Win rate | Male roles | Win rate |
|---|---|---|---|---|
| ElevenLabs v4 | 49/96 | 51% (41 – 61) | 59/96 | 61% (51 – 71) |
| Gemini 3.8 Flash TTS | 57/97 | 59% (49 – 68) | 48/96 | 50% (40 – 60) |
| Fish Audio Drama 3 | 50/96 | 52% (42 – 62) | 51/96 | 53% (43 – 63) |
| Cartesia Sonic 3 | 37/97 | 38% (29 – 48) | 52/96 | 54% (44 – 64) |
| Qwen 3.1 TTS Flash | 48/96 | 50% (40 – 60) | 30/96 | 31% (23 – 41) |
2.5 Level, bandwidth and pace as confounds
The five systems ship audio at different levels and sample rates. All clips were resampled to 44.1 kHz and the whole set shifted by one shared gain (-3.2 dB) so that no clip was normalised individually (§4.4), which leaves the providers' own level differences in place: after the shift, system means ranged from −26.6 dB RMS (ElevenLabs) to −21.0 dB (Sonic) (Table 5). If listeners simply preferred louder, ElevenLabs would have lost; the louder clip in a pair was chosen 45% of the time. Bandwidth is a real difference that listeners could hear: Qwen's output has energy only to about 10 kHz against 12–16 kHz for the others, and Gemini's to 12 kHz. Spectral tilt alone separates most pairs of systems (Table 6; a classifier on tilt distinguishes ElevenLabs from Qwen with AUC 0.98), so timbre is part of what was judged; unlike a detection study, that is legitimate here, since a listener choosing an audiobook hears the timbre too. Speaking rate was deliberately not equalised and differed from 13.7 characters per second (Fish) to 18.9 (Sonic); it is part of what each system does with a passage.
Table 5. Stimuli by system: model identifier as called, the four stock voices used, speaking rate over the sixteen renders, clip duration, level after the shared gain, effective bandwidth, and word error rate of a Whisper large-v3 transcript against the passage text (median / worst clip). A WER of 3% on this prose is mostly the transcriber on archaic names; no clip was short or over-long (§4.3).
| System | Model | Voices | chars/s | Seconds (range) | RMS dB | Bandwidth kHz | WER |
|---|---|---|---|---|---|---|---|
| ElevenLabs v4 | eleven_v4 | Alice, Daniel, George, Matilda | 15.4 | 50 (39–58) | -26.6 | 15.2 | 2.8% / 10% |
| Gemini 3.8 Flash TTS | gemini-3.8-flash-tts | Aoede, Charon, Fenrir, Kore | 16.2 | 47 (35–59) | -23.2 | 12.2 | 2.5% / 10% |
| Fish Audio Drama 3 | drama-3 | English Female Audiobook, Nigerian Storyteller, STOIC RANGER AUDIOBOOKS, calm storyteller male | 13.7 | 56 (40–71) | -21.7 | 14.8 | 3.4% / 10% |
| Cartesia Sonic 3 | sonic-3 | Clive, Jolene, Lauren, Quentin | 18.9 | 40 (32–47) | -21.0 | 15.6 | 3.3% / 12% |
| Qwen 3.1 TTS Flash | qwen-audio-3.1-tts-flash | Abby, Andy, Ava, Eric | 15.9 | 48 (35–58) | -22.9 | 9.9 | 2.2% / 10% |
Table 6. How separable two systems' clips are on spectral tilt alone (AUC of a one-feature classifier; 0.5 is indistinguishable, 1.0 perfectly separable).
| ElevenLabs | Gemini | Fish | Sonic | Qwen | |
|---|---|---|---|---|---|
| ElevenLabs v4 | — | 0.83 | 0.98 | 0.91 | 0.98 |
| Gemini 3.8 Flash TTS | 0.83 | — | 0.68 | 0.52 | 0.73 |
| Fish Audio Drama 3 | 0.98 | 0.68 | — | 0.73 | 0.53 |
| Cartesia Sonic 3 | 0.91 | 0.52 | 0.73 | — | 0.74 |
| Qwen 3.1 TTS Flash | 0.98 | 0.73 | 0.53 | 0.74 | — |
3Interpretation and limitations
The defensible findings are two. Qwen 3.1 TTS Flash was the system listeners least wanted to keep hearing, by a margin that survives every resampling, and it was the only one to lose every series; the stimulus measurements point at a plausible reason, the narrowest bandwidth, without proving it. Cartesia Sonic 3 sat below the leading group. Among ElevenLabs v4, Gemini 3.8 Flash TTS and Fish Audio Drama 3 the study cannot say which is best, and the design did not expect to: the preregistered sample of 48 judgments per pair was chosen, by simulation, to identify a top system that led by about 70 Elo points, and no system led by that much.
Two readings should be resisted. The first is that ElevenLabs "won": it had the highest observed rate and ranked first in about half of the replicates, which is the definition of not established. The second is that the systems are therefore interchangeable: the leading three differ in level, bandwidth, pace and voice catalogue, and Table 4 suggests their relative standing moves with the voice chosen; a buyer should test the voice they would actually use.
What this study does and does not show
It shows that, given identical text and matched voice gender, three of five current systems were preferred about equally by lay listeners, that Sonic 3 trailed them, and that Qwen 3.1 Flash was clearly last.
It does not show which of the three leaders is best for audiobooks. It does not test long-form narration, where drift, pacing monotony and chapter-scale consistency would matter; each clip was under a minute. It does not test voice cloning, and it does not test each provider's best voice, only four stock voices chosen before any audio was heard. It does not test the Plus tier of Qwen's TTS (§4.1). Listeners were untrained members of the public, not audiobook professionals.
Specific limitations:
- Sized for "which is best", not for the full order. Simulations before data collection (§4.6) showed 48 judgments per pair recovering a top system correctly 98% of the time when one led by 70 Elo, and the full ordering only 22% of the time. The observed spread among the leaders is about 20 Elo. After this result was known, the study was extended to 8 judgments per item (1,280 in total) with the per-listener cap lowered from 16 to 10; that extension is in progress. This report is the preregistered analysis at the preregistered sample. The extension was decided after inspecting these results, so the extended analysis will be reported as a separate version, with this one unchanged, and the reader should treat the extension's intervals as those of a data-dependent continuation.
- No tie option. The design document specified a tie option; the platform served the study as a forced choice and all 481 judgments are forced choices. A listener with no preference contributed a coin flip, which pulls every win rate toward 50% and widens nothing; it does not bias the order.
- Four voices per system, chosen by catalogue description. The voice is confounded with the system. Fish's catalogue is user-uploaded, so its four voices were chosen by searching audiobook terms and reading descriptions; the others are the providers' stock narrators. A different pick of voices could move a provider by more than the gaps at the top of Table 1.
- Qwen is the Flash tier. Qwen's higher Plus tier exposes only Mandarin-native voices that read English with an accent and could not fill the male roles, so the Flash tier with native-English voices was used and is labelled as such throughout. The result is for Flash, not for the model family.
- Short clips of nineteenth-century prose. Passages are 620–890 characters of public-domain literary fiction with long sentences and period vocabulary. Contemporary genre fiction, nonfiction or dialogue-heavy text could rank the systems differently.
- Listeners. 78 raters contributed a median of 5 judgments (range 1–16; 2 reached the cap of 16). Mobile devices accounted for 57% of the judgments with device data, so many listened on phone speakers or earbuds; bandwidth differences may be less audible there than on headphones. Rater nationality is verified at payout; a majority had not yet completed verification at the time of this snapshot.
- Interface signals were not captured for every judgment. Listening-time and device data were recorded for 429 of 481 judgments; the 52 earliest predate that instrumentation. Of the 429, 423 show both clips played to at least 90% of their length.
- Exploratory splits. The gender and passage analyses were run after the ranking was known and are reported with raw-count intervals as descriptions, not tests.
4Methods
4.1 Systems and voices
Five commercial systems, called on their providers' own APIs between 1 and 2 November 2026, each with the model identifier in Table 5. The field was chosen from a public text-to-speech arena's top ranks, taking one system per provider; a sixth system was dropped at design time to halve the number of pairs and double the judgments per pair at the same budget. For each system four stock voices were selected before any audio was rendered: two female and two male, described by the provider as narration or storytelling voices (for Fish, whose catalogue is user-uploaded, by searching audiobook terms and reading the description). Voices were assigned to the sixteen narration roles in a fixed cycle so that each voice read four passages and the gender of the voice matched the role. Where a voice produced a defective render on a passage the provider's other voice of that gender was used instead and the override recorded; this happened twice, both for Qwen, where one voice hallucinated continuation text on two passages that the other read cleanly. Qwen's Plus tier was evaluated and not used: its four voices are Mandarin-native and read English with an accent, and only one is male. ElevenLabs was called with its eleven_v4 model, Gemini with gemini-3.8-flash-tts, Fish with drama-3, Cartesia with sonic-3 and Qwen with qwen-audio-3.1-tts-flash, all at default settings and with no SSML, style or emotion controls.
4.2 Passages
Sixteen passages, one from each of sixteen novels on Project Gutenberg: Austen (Pride and Prejudice, Emma, Sense and Sensibility), Dickens (A Tale of Two Cities, Great Expectations, Oliver Twist, David Copperfield), Brontë (Wuthering Heights), Melville (Moby Dick), Shelley (Frankenstein), Stoker (Dracula), Stevenson (Jekyll and Hyde), Doyle (The Adventures of Sherlock Holmes), Wilde (Dorian Gray) and Tolstoy (War and Peace, Anna Karenina). One passage per book so that no book's register could dominate a system's score and no listener meets the same author twice. Each passage is a run of consecutive prose paragraphs of 620–890 characters, cut at sentence boundaries, with no verse, title matter, indexes or dense dialogue, chosen for a target of about 45 seconds of reading. The text was frozen before rendering and is identical for every system.
4.3 Rendering and verification
Each clip is one API call for the whole passage, never chunked. Every clip was transcribed with Whisper large-v3 (beam 5) and compared with the text it was asked to read on two measures: word error rate, and the ratio of transcript length to text length, which separates a model that kept going or stopped early from a transcriber struggling with period prose. Clips with a ratio outside 0.85–1.15 were rejected and re-rendered; the two Qwen hallucinations in §4.1 were caught this way (ratios 1.36 and 1.61). All 80 clips in the study have ratios within 0.98–1.02 and WER of at most 12.5% (median 2–3%), consistent with transcriber error on names.
4.4 Level and sample rate
Providers deliver 22.05 kHz (Qwen), 24 kHz (Gemini) and 44.1 kHz (the others). All clips were resampled to 44.1 kHz so that a difference in bandwidth is a property of the system and not of playback. Levels were handled with one shared gain of -3.2 dB applied to every clip, which brought the loudest system near −21 dB RMS with every peak at or below -3.2 dBFS; no clip was normalised individually, because per-clip normalisation flattens dynamic range, which is part of what makes narration good, and would favour whichever system is already most compressed. The resulting per-system levels, bandwidth and spectral tilt were measured and are reported (Tables 5 and 6) rather than corrected. Speaking rate and pause structure were not altered.
4.5 Items and assignment
Each item pairs two systems reading the same passage, so a trial varies only the system (and the voice that system uses for that role). Five systems give ten unordered pairs; across sixteen passages that is 160 items, each planned for three independent judgments. Which system was on the left alternated so that each system was on the left in half of its items. Assignment limited each listener to one judgment per passage (so at most 16) and to meeting any one system at most seven times, so no listener could learn a system's voices by repetition. Within those constraints a listener's next items were drawn from those still short of three judgments. One item received a fourth judgment when two listeners were issued its last slot concurrently; it is retained.
4.6 Sample size
Before data collection the design was simulated under a plausible truth (one system ahead by 70 Elo, the rest within 50 Elo) with listener-level noise, 40 runs per setting. At 48 judgments per pair (480 total) the top system was identified correctly in 98% of runs and the full order in 22%; at 128 per pair (1,280) the full order in 50%. The study was run at 480 because the question was which system is best. The extension to 1,280 (§3) uses the second setting.
4.7 Listeners and task
Recruitment, payment, blinding and assignment follow the Woodstitch platform methodology; only what is specific to this study is given here. 78 raters on the Woodstitch platform participated between 2 and 6 November 2026 (data snapshot 6 November 2026, 18:05 UTC), contributing a median of 5 judgments (range 1–16). Raters are paid per second of audio heard, about 35 cents per trial here, and payment does not depend on the answer. Each trial presented two readings labelled only A and B with the question "Which would you rather listen to for a long audiobook?" and a forced choice; both clips had to play before an answer could be given. Raters were not told which systems were involved or that all readings were synthetic. Median time on a trial was 114 s for a mean of 96 s of audio, and the median time from the second clip finishing to the answer was 2.3 s. Browser signals were captured for 429 judgments: 423 show both clips played to at least 90%, 10 reported a playback error, 123 show the tab losing focus at some point, and 243 (57%) were made on a mobile device. Judgments from the author's own rater account are excluded by rule; none were made on this study.
4.8 Statistical analysis
The primary statistic is each system's win rate over all pairings it appeared in, with a Wilson score interval4 on the raw count and, as the reported interval, a percentile interval from a non-parametric bootstrap that resamples listeners with replacement (3,000 replicates),3 because judgments from one listener are not independent. A second bootstrap resamples listeners and passages independently, weighting each judgment by the product of its listener's and passage's resampled multiplicities; independent resampling of crossed clusters is not guaranteed to be calibrated and these wider intervals are reported as indicative. System strengths are Bradley-Terry1 log-abilities fitted by minorisation-maximisation2 with a weak symmetric prior, centred to mean zero, refitted in every replicate; the share of replicates in which each system ranks first, in the top three, or last is reported. Head-to-head, passage and gender tables are raw counts. Level, bandwidth and tilt were measured on the clips as served. No significance tests are reported. No judgment was excluded and no listener was removed.
5Data and reproducibility
The per-judgment dataset (item, passage, the two systems, side, choice, time spent; listener identifiers replaced by anonymous indices), the sixteen passage texts with their Gutenberg sources, the voice assignments, the transcription report for all 80 clips and the render, verification, normalisation and analysis scripts are available on request from support@woodstitch.com. Twenty of the 80 clips (four passages, all five systems) are published on this page exactly as listeners heard them; the remaining 60 are in use in the running extension and will be released when it closes. The passage texts are public domain; the audio is published for evaluation and comparison under each provider's terms.
6Funding and competing interests
This study was funded by SquireWorks Audiobooks, an audiobook production company that uses ElevenLabs for some productions and operates its own fine-tuned open-weight narration model, and was conducted on Woodstitch, a human-evaluation platform operated by the author. SquireWorks is a paying customer of ElevenLabs, Cartesia, Fish Audio, Google and Alibaba for API access and has no other commercial relationship with any of them; no provider was informed of the study or had any involvement in its design, execution, analysis or reporting. The author uses Anthropic's Claude, which assisted in writing the analysis code and drafting this report; the judgments are entirely human and the analysis is deterministic from the data. The author declares no other competing interests.
7References
- Bradley RA, Terry ME. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika 1952;39(3/4):324–345.
- Hunter DR. MM algorithms for generalized Bradley-Terry models. Ann. Statist. 2004;32(1):384–406.
- Efron B, Tibshirani RJ. An Introduction to the Bootstrap. Chapman & Hall, 1993.
- Wilson EB. Probable inference, the law of succession, and statistical inference. J. Am. Stat. Assoc. 1927;22(158):209–212.
- Radford A, Kim JW, Xu T, Brockman G, McLeavey C, Sutskever I. Robust speech recognition via large-scale weak supervision. ICML 2023. (Whisper large-v3.)
- Project Gutenberg. https://www.gutenberg.org/.
8Version history
- 1.0 — 6 November 2026. Preregistered analysis at the preregistered sample (481 of 480 planned judgments). An extension to 1,280 judgments was started the same day after these results were inspected; it will be reported as version 1.1 without altering this version.