Which AI model writes the best piano music?

Eight large language models composed short piano loops for the same twenty prompts. 62 paid listeners judged them blind, in pairs. Interim report: 514 of 1,638 planned judgments. Listen before you read.

1What we tested

Large language models are routinely asked to write music as symbolic notation, and leaderboards for the task are built from automated metrics or from the models judging each other. We wanted the simpler and harder measurement: put the output in front of people who do not know which model wrote it and ask which they like. To make the comparison about composition rather than sound, every model's notes were played by the same instrument at the same settings, so the only thing a listener could respond to was what the model chose to write.

Each of eight models, one flagship per lab, received the same twenty prompts: an identical technical brief (eight bars of 4/4 solo piano at 80–110 BPM, as a JSON list of notes with pitch, timing and velocity) with one line of musical direction that varied, five directions in each of four styles. Each model answered each prompt once, with no retries on bad output and no feedback. Valid answers were rendered to audio through a sampled grand piano. Listeners heard two loops written for the same prompt and chose one. The question is which model's loops people prefer, and secondarily, what the preferred loops have in common.

2Results

2.1 Two models lead, one trails, five cannot be ordered

Table 1 ranks the eight models by Bradley-Terry strength fitted to all 514 pairwise judgments; the win rate is the simpler statistic and gives the same order. Gemini 4 Argon won 66.1% of its pairings (listener-resampled 95% CI 57–75%; resampling listeners and prompts together, 51–79%) and Claude Opus 5.5 61.1% (51–70%; 46–75%). Both listener-resampled intervals exclude 50%; Claude’s listener-and-prompt interval does not. An aggregate win rate above 50% says a model beat its particular mix of opponents more often than not; it does not by itself establish that it beat any one of them, and the ordering between Gemini and Claude is not established: Gemini ranks first in 79% of replicates and Claude in 18%. Llama 4 Maverick won 26.8% (19–35%; 15–40%) and ranked last in every replicate. The five models between them won between 46% and 55%; their intervals all include 50% and each other, which means their order is unknown, not that they are equal.

Side of presentation did not matter: the left-hand loop was chosen on 48.4% of judgments (CI 44–53). Sides were balanced within every pair by design (§4.5).

Table 1. Ranking by Bradley-Terry strength. Win rate is wins over all pairings the model appeared in. The 95% interval resamples listeners (3,000 replicates); the second resamples listeners and prompts independently and is wider. Strength is on a natural-log scale, mean zero; a difference of 0.62 between two models predicts the stronger one wins 65% of their pairings. P(1st) is the share of listener-resampled replicates in which the model ranked first. Valid is loops that parsed and passed validation, out of 20 prompts.

2.2 Head-to-head

Table 2 gives every model's record against every other, pooled over prompts. Gemini has a winning record against all seven opponents. Claude lost only to Gemini (7–12) and, narrowly, Kimi (8–9), and beat Grok 10–2 and Llama 18–2. The two models with the same 46.2% overall rate split differently: GPT-6.1 lost 4–13 to Qwen but beat Llama 18–5, while DeepSeek went 8–8 with Qwen and only 12–9 against Llama. Cells hold 13 to 25 judgments each, so no individual series establishes superiority over that opponent. What the table shows is that Gemini’s observed record is positive against every opponent and Llama’s negative against every opponent; it does not show that either is better than any single named rival.

Table 2. Head-to-head records, wins of the row model over the column model, all prompts pooled. Each cell is one pair of models across up to 20 prompts at up to three judgments each (fewer for Qwen, which has 18 loops).

2.3 Results by style

The twenty prompts were five each in four styles: dark and emotional, uplifting and catchy, soulful and jazzy, minimal and hypnotic (§4.2). Table 3 gives each model's win rate within each style. Gemini is the only model without a weak style, winning 62–68% in all four. Claude was strongest on soulful prompts (70%) and weakest on minimal (55%). Qwen's overall 55% hides the widest spread in the table: 72% on dark prompts and 42% on uplifting. Grok fell to 32% on uplifting prompts and DeepSeek to 33% on soulful ones. Llama was below 40% in every style. Cells hold 19 to 45 judgments, so a 10-point difference between two cells is well inside noise; the within-model spread for Qwen, Grok and DeepSeek is large enough to be worth following as the sample grows, not to report as established.

Table 3. Win rate by style. Each cell is wins over pairings within that style's five prompts. Wilson 95% intervals on cells of this size are roughly ±15–20 points.

2.4 Taste is prompt-specific

Ranking the models within each prompt by observed win rate (ties counted once, for an arbitrary one of the tied models), Claude had the highest rate on six of the twenty prompts, Gemini on five, DeepSeek on three, Kimi and Qwen on two each, GPT-6.1 and Grok on one each. Llama had the lowest rate on six prompts and was shut out, with no wins at all, on four: 0–9 on the second dark prompt, 0–6 on two of the uplifting prompts and 0–5 on a third. Several mid-table models have both a prompt they dominated and one they lost badly: Grok went 9–1 on one minimal prompt and 1–8 on a soulful one; Kimi 8–1 on a soulful prompt and 0–3 on a minimal one; DeepSeek 1–10 on one soulful prompt. The per-prompt cells hold 5 to 10 judgments against an uneven mix of opponents, so different leaders would appear across prompts by chance alone; whether the models’ relative quality truly depends on the brief is a hypothesis this sample cannot test.

2.5 What the preferred loops look like

Because every loop exists as a note list, we can describe the compositions directly (Table 4) and ask, for each judgment, whether the preferred loop had more or less of each property than the rejected one (Table 5). Both analyses were specified after the rankings were known and are exploratory.

What the models wrote. The models differ most in how much they write and how far across the keyboard. Claude, Kimi, Gemini and GPT-6.1 average 105–127 notes per eight bars spanning four octaves (44–49 semitones); DeepSeek averages 71 notes over two octaves; Llama 52 notes over 17 semitones, with 1% of its notes below C3 against 13–17% for the leaders. Llama's loops are, in effect, a right hand alone. Gemini writes with the flattest dynamics (velocity standard deviation 6.7) and the longest notes (1.7 beats); Claude, GPT-6.1, Grok and Llama use the widest velocity range. Tempos are similar, 89–93 BPM on average, from a permitted 80–110. Rendered level, which the fixed pipeline leaves free to vary with what was written, ranges from a mean of −33.3 dBFS RMS (Gemini) to −37.3 (DeepSeek), a 4 dB spread; peaks are below −5 dBFS everywhere and nothing clipped.

Table 4. Composition statistics, mean per loop over each model's valid loops (20 each; Qwen 18). Notes is the count in eight bars; onsets is distinct note-start times; notes per onset is a polyphony measure; velocity is MIDI 1–127; pitch range is highest minus lowest pitch in semitones; bass share is the fraction of notes below C3 (MIDI 48); note length is mean duration in beats; level is the RMS of the rendered loop in dBFS.

What listeners preferred. Across the 514 judgments, the preferred loop had the wider pitch range 62% of the time (listener-resampled 95% CI 57–67%; listeners and prompts 54–69%), more notes 59% (55–64%), a larger bass share 58% (54–63%), and more onsets and more notes per onset 56% each (Table 5). It had the slower tempo 57% of the time among pairs whose tempos differed. It was also the louder of the two 56% of the time (51–60%), though not the one with the higher peak (50%). Dynamics went the other way: the preferred loop had the larger velocity standard deviation only 46% of the time (42–50%) and the wider velocity range 48%. Descriptively, preference went with fuller writing, a used left hand and a somewhat louder rendering, and not with velocity variation. The design cannot separate these: a loop with more notes in more registers renders louder through a fixed pipeline, so a preference for density and a preference for level are confounded here, and velocity standard deviation is a crude proxy for expressive dynamics in any case. These are marginal associations on non-independent judgments; the properties are correlated with each other and with model identity, and no causal reading is intended.

Table 5. Share of judgments in which the preferred loop had the higher value of each property than the rejected loop. Judgments where the two loops tied on the property are excluded, so denominators vary. Intervals are 95% percentile intervals from a bootstrap resampling listeners (3,000 replicates); intervals that also resample prompts are about 3 points wider at each end. Twelve properties were examined and are correlated with one another; no significance threshold is claimed.

2.6 Sensitivity: general-preference judgments only

The question shown to listeners changed after the first 96 judgments (§4.6): those asked which loop the listener would pick for a hip-hop song, the remaining 418 which loop they preferred. The pooled figures above therefore mix two questions. Table 6 repeats the main analysis on the 418 general-preference judgments alone (53 listeners). The top two and the last place are the same, with slightly larger margins: Gemini 68.7% (59–77%), Claude 65.3% (54–75%), Llama 27.7% (19–37%). GPT-6.1 falls to 41.0% (32–50%) and the middle reorders, which is what a 96-judgment difference between small subsets can do; it is not evidence of a wording effect, since wording changed together with time and listener cohort.

Table 6. Main analysis restricted to judgments collected under “Which piano loop do you prefer?” (418 judgments, 53 listeners). Intervals resample listeners (3,000 replicates).

3Interpretation and limitations

The defensible findings are three. Gemini 4 Argon and Claude Opus 5.5 wrote loops that lay listeners preferred to the field, with Gemini nominally ahead but not separated from Claude. Llama 4 Maverick wrote loops listeners preferred least, by a wide margin; the composition statistics are consistent with that, since it wrote about half as many notes as the leaders over a third of the pitch range with no left hand, but they do not establish why it lost. The five models in between cannot be ordered yet. The descriptive result that preference went with range, density, bass and level rather than with velocity variation is consistent with the ranking but is an association among correlated properties, not an explanation.

Two readings should be resisted. The first is that this measures "musical ability": it measures what 62 untrained listeners preferred in a 20-second solo piano loop played by one sample library, which rewards writing that is immediately full and legible and may penalise restraint. The second is that it measures the models' best work: each model answered each prompt once, with no system prompt, no examples and no chance to revise, so a model that is excellent on a second attempt scores exactly as well as its first attempt.

Specific limitations:

• Interim snapshot. The study is planned at 1,638 judgments (546 pairs × 3). This report uses the 514 available on 5 November 2026, 31% of the plan. The headline ranking was inspected at about 100 and 500 judgments before this writing. The intervals are not adjusted for data-dependent inspection. A versioned update will be issued at the planned sample size whatever it shows.
• Question wording changed early. The first 96 judgments (14 listeners) were collected under the question "Which piano loop would you pick for a hip-hop song?"; the remaining 418 under "Which piano loop do you prefer?" (§4.6). The headline figures pool both, so they estimate a mixture of two questions; §2.6 gives the general-preference-only analysis, which is the cleaner answer to the question in the title and gives the same top two and last place. Under the first wording alone (96 judgments) GPT-6.1 won 63% of 30 pairings and Qwen 67% of 21, against 41% and 53% afterwards, while Claude went from 44% of 25 to 65% of 101; those subsets are too small, and confounded with time and listener cohort, to attribute to the wording.
• One answer per prompt. Sampling variance within a model is not measured. A model's twenty loops are twenty draws, and a different draw could have ranked differently. Models were called with the vendors' default sampling settings.
• Listeners are not musicians. Woodstitch raters are paid members of the public with no screening for musical training. Trained listeners could rank differently, and might weigh harmony, voice-leading or dynamics more than this group did.
• One renderer, one piano. Every loop was played by the Salamander Grand sample library through FluidSynth at fixed settings (§4.4). The library was chosen for its velocity layering and open licence; a physically modelled piano could render dynamics more audibly. Loops were not level-normalised, by design: a model that wrote quietly was heard quietly.
• Serving and precision. Three arms (Gemini, Claude, Grok) were called on their labs’ own APIs. GPT-6.1 was called through OpenRouter, which serves closed-weight models only from the lab or its cloud partner. Qwen, Kimi, DeepSeek and Llama were called through OpenRouter pinned to a single named host; Llama’s host is a third-party reseller serving at FP8 precision, the only explicitly reduced-precision endpoint. Whether FP8 affects symbolic music generation is unknown; it is disclosed, not controlled for.
• Qwen has 18 loops. On two prompts Qwen 3.8 Max spent its entire 24,000-token output budget on reasoning and returned no answer; those pairs do not exist, so Qwen appears in 114 pairings against 126–142 for the others. Its win rate is therefore conditional on having produced a valid loop; a reader who counts a non-answer as a loss should read its validity column alongside its win rate.
• Uneven interim coverage. At this snapshot 352 of the 546 items have at least one judgment (219 have one, 104 two, 29 three) and 194 have none; prompts have between 18 and 31 judgments and models between 114 and 142 appearances. Win rates are pooled over the judgments that exist and so weight prompts and opponents by their current coverage, not equally; the planned sample gives every pair of models three judgments on every prompt.
• Loops, not songs. Eight bars of solo piano is a narrow test of composition. Structure, development over minutes, melody against accompaniment, and arrangement are not tested.
• Brief not shown to listeners. The models were asked for a hip-hop foundation; listeners were asked only which loop they preferred. The result is unconditioned preference, not fitness for the brief.
• Exploratory analysis. The style, prompt and composition-property analyses were run after the ranking was known. Cells are small, ten properties were examined on non-independent judgments, and no significance claim is made.

4Methods

4.1 Models and access

One flagship model per lab, as of 2 November 2026 (Table 7). The rule was to use the lab's own API where we hold a key, otherwise OpenRouter pinned to a single named host with fallbacks disabled, and never a reduced-precision endpoint when a better one was offered. Two errors were caught before generation and are recorded because either would have shipped silently: DeepSeek was initially configured as its budget model rather than its flagship, and DeepSeek's first test outputs through OpenRouter came from seven different resellers, which is why hosts are pinned. GPT-6.1 Terra Pro was accessed through OpenRouter because our own OpenAI key did not have the Pro tier; closed-weight models on OpenRouter are served only by the lab or its cloud partner. Alibaba's own endpoint accepted the prompt but never returned, so Qwen was called through OpenRouter pinned to Alibaba's endpoint, which serves the same weights. Each call was a single user message with no system prompt, a 24,000-token output limit and the provider's default sampling settings.

Table 7. Arms. Route is the lab's own API ("direct") or OpenRouter pinned to the named host. Precision is as declared by the endpoint; "full / undeclared" means no quantisation was reported.

4.2 Prompts

Twenty prompts, identical except for one line of musical direction. The shared body asked for an original instrumental piano loop "that could serve as the musical foundation for a professional hip-hop song": exactly eight bars of 4/4 at 80–110 BPM, piano only, with a distinctive motif, harmonic and rhythmic interest, development rather than repetition, a final bar that loops back to the first, memorability over complexity, and no reproduction of an existing melody. Output was to be JSON only, in a fixed schema: bpm, time_signature, and a list of notes with MIDI pitch (21–108), start in beats from 0, duration in beats and velocity (1–127), with the instruction to use velocity to shape expression. The direction line took one of twenty values, five in each of four styles: dark / emotional (e.g. "Moody and haunting, with a sense of unease"), uplifting / catchy ("Playful and bouncy, with a hook you could hum back"), soulful / jazzy ("Smoky and late-night, with loose, behind-the-beat phrasing") and minimal / hypnotic ("Hypnotic and repetitive, built on one insistent figure"). Style is a blocking factor so that a model with a talent for one idiom is not reported as better at music in general. A script verified that all twenty prompts share one body and differ only in the direction line.

4.3 Generation and validation

Every model received every prompt exactly once, on 2 November 2026. There were no retries on a bad answer, no feedback and no self-critique; the only retries were for transport failures (rate limits, timeouts, server errors), with up to five attempts. A markdown code fence around the JSON was stripped if present and recorded; nothing else was repaired. Each answer was then validated and rejected, never fixed, if it failed any of: parseable JSON; BPM a number in 80–110; time signature 4/4; every pitch on the keyboard (21–108); every velocity in 1–127; at most 400 notes; and music extending to at least the last bar (an answer that fills only the first half is not eight bars). The validity rate is reported as a column in Table 1 because it separates technical reliability from musical quality. 158 of 160 calls produced a valid loop. The two failures were both Qwen 3.8 Max, which on two prompts exhausted its 24,000-token output budget while reasoning and returned no text. Every other model was valid on all twenty prompts.

4.4 Rendering

Valid note lists were written to MIDI (pretty_midi) and rendered to 48 kHz audio with FluidSynth using the Salamander Grand Piano V3 sample library (SF2 conversion by the FreePats project), at a fixed gain of 0.60, reverb room size 0.50 and level 0.30, chorus off. These settings never varied by model or clip. Salamander was chosen over the FluidSynth default bank because the default has no velocity layers for the tested note (brightness identical from velocity 20 to 127, 32 dB dynamic range), which would have rendered an expressive crescendo and a flat loop identically; Salamander keeps 16 velocity layers and spans 53 dB. A better-sounding physically modelled piano was considered and not used because it cannot be redistributed, and the study is meant to be re-runnable by anyone with the published sample library. No loudness normalisation was applied, per clip or shared: the pipeline is identical for every loop, so level can move only through what the model wrote, which includes velocity but also note density, register, overlap and sustain. Measured on the rendered loops, per-model mean RMS ranged from −33.3 to −37.3 dBFS, every peak was below −5 dBFS, and nothing clipped (Table 4). That spread is treated as part of the composition and its consequences are discussed in §2.5. Rendered loops ran 19.5–22.9 s (eight bars at the model's chosen tempo, plus reverb tail).

4.5 Items and assignment

Each item pairs two models' loops for the same prompt, so a trial varies only the model. Eight models give 28 unordered pairs; across 20 prompts that is 560 items, less the 14 involving Qwen on its two missing prompts, for 546 items. Which model was on the left alternated by prompt within each pair from a randomly chosen start, so every model is on the left in half its items. Each item is planned for three independent judgments (1,638 in total). Assignment rules limited each listener to one judgment per prompt (so at most 20 per listener) and to meeting any one model at most seven times, so no listener could learn a model's style by repetition. Listeners never heard the same prompt twice. Within those constraints a listener’s next items were drawn from the items still short of three judgments; the draw was not stratified by prompt or model, which is why interim coverage is uneven (§3).

4.6 Listeners and task

Recruitment, payment, blinding and assignment follow the Woodstitch platform methodology; only what is specific to this study is given here. 62 raters on the Woodstitch platform participated between 3 and 5 November 2026 (data snapshot 5 November 2026, 23:35 UTC), contributing a median of 6 judgments (range 1–20; three completed all 20). Raters are paid per second of audio heard, about 19 cents per trial here, and payment does not depend on the answer given. Each trial presented two loops, labelled only A and B, and a forced choice; there was no tie option. The player permits one playback per clip and both clips had to play through before an answer could be given. Raters were not told that the loops were machine-composed, which models were involved, or the brief the models had been given. Clip order within a sitting was randomised. No feedback was given. The question shown changed once: the first 96 judgments (ratings up to 3 November 16:30 UTC) were collected under "Which piano loop would you pick for a hip-hop song?", and the remaining 418 under "Which piano loop do you prefer?". The change was made to ask for unconditioned preference; all judgments are retained and the split is reported in §3. Nine judgments submitted from the author's own rater account during interface testing are excluded. Of 523 trials answered, none were skipped. Browser interaction signals were captured for all 514 judgments: in 508 the recorded listening time reached at least 90% of both clips, none reported an audio error, and 55 showed the tab losing focus at some point. 255 judgments (50%) were made on a mobile device. Median time on a trial was 52 s and the median time from the second clip finishing to the answer was 1.9 s.

4.7 Statistical analysis

The primary statistic is each model's win rate over all pairings it appeared in, with a Wilson score interval⁴ on the raw count and, as the reported interval, a percentile interval from a non-parametric bootstrap that resamples listeners with replacement (3,000 replicates),³ because judgments from one listener are not independent. A second bootstrap resamples listeners and prompts independently (3,000 replicates), weighting each judgment by the product of its listener's and prompt's resampled multiplicities, to indicate uncertainty from the choice of prompts; independent resampling of crossed clusters is not guaranteed to be calibrated and these wider intervals are reported as indicative. Model strengths are Bradley-Terry¹ log-abilities fitted by the minorisation-maximisation algorithm² with a weak symmetric prior, centred to mean zero; they are refitted in every bootstrap replicate, and the share of replicates in which each model ranks first, in the top two, or last is reported. Head-to-head, style and prompt tables are raw counts. Composition properties were computed from the models' JSON: note count, distinct onset times (rounded to a thousandth of a beat), notes per onset, mean and standard deviation and range of velocity, pitch range in semitones, share of notes below MIDI 48, mean note duration in beats and BPM. Table 5 counts, per judgment, whether the preferred loop’s value exceeded the rejected loop’s, excluding ties, with intervals from the same listener bootstrap. Rendered level and peak were measured on the audio files as RMS and peak in dBFS. No significance tests are reported. Nine judgments from the author's test account were excluded; no other judgments were excluded and no listener was removed. No adjustment is made for this being an interim snapshot.

5Data and reproducibility

The per-judgment dataset (trial, prompt, the two models, side, choice, time spent; listener identifiers replaced by anonymous indices), all 160 raw model responses with the serving metadata recorded per call, the 158 validated note lists, the twenty prompts, and the generation, validation, rendering and analysis scripts are available on request from support@woodstitch.com. Thirty-two of the 158 rendered loops (four briefs, one per style, all eight models) are published on this page, exactly as raters heard them; the rest remain in use in the running study and will be released when it closes. The piano sample library is the Salamander Grand Piano V3 (version v3+20200602 SF2, FreePats), which anyone may download; rendering the published note lists through it with the settings in §4.4 reproduces the stimuli.

Salamander Grand Piano by Alexander Holm, CC-BY 3.0. SF2 conversion by roberto@zenvoid.org for the FreePats project.

6Funding and competing interests

This study was funded by SquireWorks Audiobooks and was conducted on Woodstitch, a human-evaluation platform operated by the author. Neither has a commercial relationship with any of the eight labs beyond paying for API access, and no lab was informed of the study or had any involvement in its design, execution, analysis or reporting. The author uses products from several of these labs in daily work, including Anthropic's Claude, which assisted in writing the analysis code and drafting this report; the judgments are entirely human and the analysis is deterministic from the data. The author declares no other competing interests.

7References

1. Bradley RA, Terry ME. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika 1952;39(3/4):324–345.
2. Hunter DR. MM algorithms for generalized Bradley-Terry models. Ann. Statist. 2004;32(1):384–406.
3. Efron B, Tibshirani RJ. An Introduction to the Bootstrap. Chapman & Hall, 1993.
4. Wilson EB. Probable inference, the law of succession, and statistical inference. J. Am. Stat. Assoc. 1927;22(158):209–212.
5. Holm A. Salamander Grand Piano V3. CC BY 3.0. SF2 conversion, FreePats project. freepats.zenvoid.org.
6. FluidSynth. A SoundFont synthesizer. https://www.fluidsynth.org/.
7. Raffel C, Ellis DPW. Intuitive analysis, creation and manipulation of MIDI data with pretty_midi. ISMIR Late-Breaking 2014.

8Version history

• 1.0 (interim) — 5 November 2026; revised 6 November 2026 after external methodological review: rank-separation language softened throughout, Table 5 intervals changed from Wilson to listener-resampled, rendered level measured and added as a confound (Table 4, §2.5, §4.4), general-preference-only sensitivity analysis added (§2.6, Table 6), serving-route summary corrected (three direct arms, five via OpenRouter), interim item coverage added; a listening section with 32 of the study loops added at the top. Numerical results otherwise unchanged. First release at 514 of 1,638 planned judgments. A versioned update will be issued at the planned sample size whatever it shows.