Riffusion vs MusicGen

Mostly a historical comparison now. Riffusion was the clever proof that image diffusion could make music; MusicGen is what you would actually use.

Side by side

Where they actually differ

The spectrogram trick was the point

Riffusion fine-tuned Stable Diffusion on spectrogram images and converted the results back to audio. It is a genuinely elegant idea and worth understanding. It also means every output passes through spectrogram inversion, which is where its characteristic smeared quality comes from.

MusicGen models audio directly

Generating EnCodec tokens and decoding them avoids the inversion step entirely, which is most of why it sounds better. It also supports melody conditioning, which Riffusion has no equivalent for.

Both have licensing problems, of different kinds

MusicGen’s weights are CC-BY-NC — non-commercial. Riffusion’s are OpenRAIL-M, which permits commercial use but attaches behavioural restrictions you must pass downstream. Neither is simply "MIT" whatever the repository badge suggests.

Name confusion is worth flagging

The hosted product now called Riffusion is a different, closed commercial service. The open project here is the original riffusion-hobby code.

Which should you choose?

Choose Riffusion when…

• You are studying how spectrogram diffusion works
• You want the historical reference implementation

Choose MusicGen when…

• You want usable audio quality
• You need melody conditioning
• You want something still receiving attention

Full MusicGen tutorial → · Licensing matrix →

← All comparisons