Music recommender with audio auto-tagging

Problem

A music streaming platform has to recommend tracks and playlists, including new tracks that nobody has played yet. Listening behavior alone cannot place those tracks.

Approach

I built a music recommender that combines collaborative signals from listening behavior with content signals extracted from the audio itself.

• Auto-tagging from waveforms. A pipeline classifies tracks into genre, mood, and instrument tags. It extracts embeddings with self-supervised music foundation models (MuQ, MERT), fine-tunes the Audio Spectrogram Transformer (AST) and PANNs CNN14 on mel-spectrograms, and uses CLAP audio-text embeddings for zero-shot tagging of new labels.
• Audio embeddings as track representations. The same embeddings drive similar-song retrieval and recommending new tracks with no listening history.
• Sessions as sequences. Listening sessions are modeled with SASRec and BERT4Rec to predict the next tracks a user is likely to play. I also experimented with Meta's generative recommender HSTU.

Architecture

Raw audio becomes mel-spectrograms and foundation-model embeddings. Tagging models label each track, the embeddings become track representations for content-based retrieval, and sequential models rank the next tracks from each user's session history, blended with collaborative signals.

Results

The news and music recommenders together contributed to an approximately 40% increase in customer retention.