Woodstitch

The human benchmark for speech AI.

Woodstitch runs blinded listening studies on text-to-speech models with paid human raters, and publishes the protocol, the judgments and the limitations in full.

First published study

ElevenLabs v4 vs a fine-tuned open model, on audiobook narration

20 matched passages from 20 books · 6 paid listeners · 127 decisive judgments · October 2026

ElevenLabs v4
117
Fine-tuned model
10
No preference
11

All six listeners independently preferred ElevenLabs, ranging from 76% to 100% of their own decisive judgments. Every listener heard the same passage from both systems, with sides counterbalanced and audio loudness-matched so neither had a level advantage. We built the losing system. We funded this study, we published the result, and the protocol is below.

For AI teams

Testing a release candidate? We run blinded human evaluations on your models, voices and scripts. Private by default — paid work never changes a public result.

Evaluate your model

For listeners

Get paid to compare short audio clips and pick the one that sounds better. Work whenever you want.

Become a rater

How these studies are run

BlindedListeners never see which system produced which clip, and the page source reveals nothing.
CounterbalancedEach system appears on side A in exactly half the pairs, so position cannot favour either.
Loudness-matchedEvery clip normalised to −23 LUFS, because louder reliably sounds better.
Attention-checkedIdentical clips are served as a pair at intervals. Listeners who claim a difference are flagged.