How Convolutional Neural Networks Learn Musical Similarity

audio embeddings for music suggestion?

Streaming platforms (Spotify, Apple Music, and so on.) have to have the power to suggest new songs to their customers. The higher the suggestions, the higher the listening expertise.

There are a lot of methods these platforms can construct their suggestion techniques. Trendy techniques will mix completely different suggestion strategies collectively right into a hybrid construction. 

Take into consideration whenever you first joined Spotify, you’ll have been requested what genres you want. Primarily based on the genres you choose, Spotify will suggest some songs. Suggestions based mostly on music metadata like this are known as content-based filtering. Collaborative filtering can be used, which teams collectively prospects that behave equally, after which recommendations are transferred between them.

The 2 strategies above lean closely on consumer behaviour. One other methodology, which is more and more being utilized by giant streaming companies, is utilizing Deep Studying to characterize songs in discovered embedding areas. This permits songs to be represented in a excessive dimensional embedding house which captures rhythm, timbre, texture, and manufacturing fashion. Similarity between songs can then be computed simply, which scales higher than utilizing classical collaborative filtering approaches when contemplating a whole lot of hundreds of thousands of customers and tens of hundreds of thousands of tracks.

By way of the rise of LLMs, phrase and phrase embeddings have turn into mainstream and are comparatively properly understood. However how does embedding work for songs and what downside are they fixing? The rest of this submit focuses on how audio turns into a mannequin enter, what architectural decisions encode music options, how contrastive studying shapes the geometry of the embedding house and the way a music recommender system utilizing an embedding may work in observe.

How does Audio turn into an enter right into a neural community?

Uncooked audio information like MP3 are essentially a waveform – a quickly various time collection. Studying from these information is feasible, however is usually data-hungry and computationally costly. We will convert .mp3 information into mel-spectrograms, that are way more suited as inputs to a neural community.

Mel-spectrograms are a approach of representing audio file’s frequency content material over time, tailored to how people understand sound. It’s a 2D illustration the place the x-axis corresponds to time, the y-axis corresponds to mel-scaled frequency bands, and every worth represents the log-scaled vitality in that band at the moment.

The colors and shapes we see on a mel-spectrogram can inform us significant musical info. Brighter colors point out increased vitality at that frequency and time and darker colors point out decrease vitality. Skinny horizontal bands point out sustained pitches and sometimes correspond to sustained notes (vocals, strings, synth pads). Tall, vertical streaks point out vitality throughout many frequencies directly, concentrated in time. These can characterize drum snares and claps.

Now we will begin to consider how convolutional neural networks can study to recognise options of those audio representations. At this level, the important thing problem turns into: how can we prepare a mannequin to recognise that two brief audio excerpts belong to the identical music with out labels?

Chunking and Contrastive Studying

Earlier than we leap into the structure of the CNN that we’ve got used, we’ll take a while to speak about how we load the spectrogram knowledge into the community, and the way we arrange the loss operate of the community with out labels.

At a really excessive stage, we feed the spectrograms into the CNN, a lot of matrix multiplication occurs inside, after which we’re left with a 128-dimensional vector which is a latent illustration of bodily options of that audio file. However how can we arrange the batching and loss for the community to have the ability to consider comparable songs.

Let’s begin with the batching. Now we have a dataset of songs (from the FMA small dataset) that we’ve got transformed into spectrograms. We make use of the tensorflow.keras.utils.Sequence class to randomly choose 8 songs from the dataset. We then randomly “chunk” every spectrogram to pick out a 128 x 129 rectangle which represents a small portion of every music, as depicted beneath.

Because of this each batch we feed into the community is of the form (8, 128, 129, 1) (batch dimension, mel frequency dimension, time chunk, channel dimension). By feeding chunks of songs as an alternative of complete songs, the mannequin will see completely different elements of the identical songs throughout coaching epochs. This prevents the mannequin from overfitting to a particular second in every monitor. Utilizing brief samples from every music encourages the community to study native musical texture (timbre, rhythmic density) somewhat than long-range construction.

Subsequent, we make use of a contrastive studying goal. Contrastive loss was launched in 2005 by Chopra et al. to study an embedding house the place comparable pairs (constructive pairs) have a low Euclidean distance, and dissimilar pairs (adverse pairs) are separated by no less than a sure margin. We’re utilizing the same idea by making use of InfoNCE loss.

We create two stochastic “views” of every batch. What this actually means is that we create two augmentations of the batch, every with random, usually distributed noise added. That is finished merely, with the next operate:

@tf.operate
def increase(x):
    """Tiny time-frequency noise."""
    noise = tf.random.regular(form=tf.form(x), imply=0.0, stddev=0.05)
    return tf.clip_by_value(x + noise, -80.0, 0.0)  
# mel dB vary normally -80–0

Embeddings of the identical audio pattern must be extra comparable to one another than to embeddings of some other pattern within the batch.

So for a batch of dimension 8, we compute the similarity of each embedding from the primary view and each embedding from the second view, leading to an 8×8 similarity matrix.

 We outline the 2 L2-normalised augmented batches as [z_i, z_j in mathbb{R}^{N times d} ]

Every row (a 128-D embedding, in our case) of the 2 batches are L2-normalised, that’s,

[ Vert z_i^{(k)} Vert_2 = 1 ]

We will then compute the similarity of each embedding from the primary view and each embedding from the second view, leading to an NxN similarity matrix. This matrix is outlined as:

[ S = frac{1}{tau} z_i z_j^T ]

The place each ingredient of S is the similarity between the embedding of music okay and embedding of music l throughout each augmentations. This may be outlined element-wise as:

[
S_{kl} = frac{1}{tau} langle z_i^{(k)}, z_j^{(l)} rangle
= frac{1}{tau} cos(z_i^{(k)}, z_j^{(l)})
]

The place tau is a temperature parameter. Because of this the diagonal entries (the similarity between chunks from the identical music) would be the constructive pairs, and the off-diagonal entries are the adverse pairs.

Then for every row okay of the similarity matrix, we compute:

[
ell_k =log
frac{exp(S_{kk})}{sum_{l=1}^{N} exp(S_{kl})}
]

This can be a softmax cross-entropy loss the place the numerator is similarity between the constructive chunks, and the denominator is the sum of all of the similarities throughout the row.

Lastly we common the loss over the batch, giving us the complete loss goal:

[
L =
frac{1}{N}
sum_{k=1}^{N}
left( log
frac{
expleft(
frac{1}{tau}
langle z_i^{(k)}, z_j^{(k)} rangle
right)
}{
sum_{l=1}^{N}
expleft(
frac{1}{tau}
langle z_i^{(k)}, z_j^{(l)} rangle
right)
}
right)
]

Minimising the contrastive loss encourages the mannequin to assign the very best similarity to matching augmented views whereas suppressing similarity to all different samples within the batch. This concurrently pulls representations of the identical audio nearer collectively and pushes representations of various audio additional aside, shaping a structured embedding house with out requiring express labels.

This loss operate is neatly described by the next python operate:

def contrastive_loss(z_i, z_j, temperature=0.1):
    """
    Compute InfoNCE loss between two batches of embeddings.
    z_i, z_j: (batch_size, embedding_dim)
    """
    z_i = tf.math.l2_normalize(z_i, axis=1)
    z_j = tf.math.l2_normalize(z_j, axis=1)


    logits = tf.matmul(z_i, z_j, transpose_b=True) / temperature
    labels = tf.vary(tf.form(logits)[0])
    loss = tf.keras.losses.sparse_categorical_crossentropy(labels, logits, from_logits=True)
    return tf.reduce_mean(loss)

Now we’ve got constructed some instinct of how we load batches into the mannequin and the way minimising our loss operate clusters comparable sounds collectively, we will dive into the construction of the CNN.

A easy CNN structure

Now we have chosen a reasonably easy convolutional neural community structure for this process. CNNs first originated with Yann LeCun and crew once they created LeNet for handwritten digit recognition. CNNs are nice at studying to know photos, and we’ve got transformed every music into an image-like format that works with CNNs.

The primary convolution layer applies 32 small filters throughout the spectrogram. At this level, the community is generally studying very native patterns: issues like brief bursts of vitality, harmonic traces, or sudden modifications that usually correspond to notice onsets or percussion. Batch normalization retains the activations well-behaved throughout coaching, and max pooling reduces the decision barely so the mannequin doesn’t overreact to tiny shifts in time or frequency.

The second block will increase the variety of filters to 64 and begins combining these low-level patterns into extra significant constructions. Right here, the community begins to select up on broader textures, repeating rhythmic patterns, and constant timbral options. Pooling once more compresses the illustration whereas protecting crucial activations.

By the third convolution layer, the mannequin is working with 128 channels. These characteristic maps are likely to mirror higher-level elements of the sound, reminiscent of total spectral steadiness or instrument-like textures. At this stage, the precise place of a characteristic issues lower than whether or not it seems in any respect.

World common pooling removes the remaining time–frequency construction by averaging every characteristic map all the way down to a single worth. This forces the community to summarize what patterns are current within the chunk, somewhat than the place they happen, and produces a fixed-size vector no matter enter size.

A dense layer then maps this abstract right into a 128-dimensional embedding. That is the house the place similarity is discovered: chunks that sound alike ought to find yourself shut collectively, whereas dissimilar sounds are pushed aside.

Lastly, the embedding is L2-normalized so that each one vectors lie on the unit sphere. This makes cosine similarity straightforward to compute and retains distances within the embedding house constant throughout contrastive coaching.

At a excessive stage, this mannequin learns about music in a lot the identical approach {that a} convolutional neural community learns about photos. As a substitute of pixels organized by peak and width, the enter here’s a mel-spectrogram organized by frequency and time. 

How do we all know the mannequin is any good?

Every little thing we’ve talked about to date has been fairly summary. How can we really know that the mel-spectrogram representations, the mannequin structure and the contrastive studying have finished an honest job at creating significant embeddings?

One widespread approach of understanding the embedding house we’ve got created is to visualise the house in a lower-dimensional one, one which people can really visualise. This system is known as dimensionality discount, and is beneficial when attempting to know excessive dimensionality knowledge.

Two methods we will use are PCA (Principal Element Evaluation) and t-SNE (t-distributed Stochastic Neighbor Embedding). PCA is a linear methodology that preserves world construction, making it helpful for understanding the general form and main instructions of variation in an embedding house. t-SNE is a non-linear methodology that prioritises native neighbourhood relationships, which makes it higher for revealing small clusters of comparable factors however much less dependable for deciphering world distances. Because of this, PCA is best for assessing whether or not an embedding house is coherent total, whereas t-SNE is best for checking whether or not comparable gadgets are likely to group collectively regionally.

As talked about above, I skilled this CNN utilizing the FMA small dataset, which accommodates style labels for every music. Once we visualise the embedding house, we will group genres collectively which helps us make some statements concerning the high quality of the embedding house.

The 2-dimensional projections give completely different however complementary views of the discovered embedding house. Neither plot exhibits completely separated style clusters, which is predicted and really fascinating for a music similarity mannequin.

Within the PCA projection, genres are closely combined and kind a easy, steady form somewhat than distinct teams. This means that the embeddings seize gradual variations in musical traits reminiscent of timbre and rhythm, somewhat than memorising style labels. As a result of PCA preserves world construction, this means that the embedding house is coherent and organised in a significant approach.

The t-SNE projection focuses on native relationships. Right here, tracks from the identical style usually tend to seem close to one another, forming small, unfastened clusters. On the similar time, there’s nonetheless important overlap between genres, reflecting the truth that many songs share traits throughout style boundaries.

General, these visualisations recommend that the embeddings work properly for similarity-based duties. PCA exhibits that the house is globally well-structured, whereas t-SNE exhibits that regionally comparable songs are likely to group collectively — each of that are essential properties for a music suggestion system. To additional consider the standard of the embeddings we might additionally have a look at recommendation-related analysis  metrics, like NDCG and recall@okay.

Turning the venture right into a usable music suggestion ap

Lastly we’ll spend a while speaking about how we will really flip this skilled mannequin into one thing usable. For example how a CNN like this could be utilized in observe, I’ve created a quite simple music recommender internet app. This app takes an uploaded MP3 file, computes its embedding and returns a listing of probably the most comparable tracks based mostly on cosine similarity. Quite than treating the mannequin in isolation, I designed the pipeline end-to-end: audio preprocessing, spectrogram era, embedding inference, similarity search, and consequence presentation. This mirrors how such a system can be utilized in observe, the place fashions should function reliably on unseen inputs somewhat than curated datasets.

The embeddings from the FMA small dataset are precomputed and saved offline, permitting suggestions to be generated rapidly utilizing cosine similarity somewhat than working the mannequin repeatedly. Chunk-level embeddings are aggregated right into a single song-level illustration, making certain constant behaviour for tracks of various lengths.

The ultimate result’s a light-weight internet utility that demonstrates how a discovered illustration may be built-in into an actual suggestion workflow. 

This can be a quite simple illustration of how embeddings might be utilized in an precise suggestion system, however it doesn’t seize the entire image. Trendy suggestion techniques will mix each audio embeddings and collaborative filtering, as talked about firstly of this text. 

Audio embeddings seize what issues sound like and collaborative filtering captures who likes what. A mix of the 2, together with extra rating fashions can mix to create a hybrid system that balances acoustic similarity and private style.

Information sources and Photographs

This venture makes use of the FMA Small dataset, a publicly accessible subset of the Free Music Archive (FMA) dataset launched by Defferrard et al. The dataset consists of brief music clips launched underneath Inventive Commons licenses and is extensively used for tutorial analysis in music info retrieval.

All schematic diagrams on this article had been generated by the creator utilizing AI-assisted picture era instruments and are utilized in accordance with the instrument’s phrases, which enable business use. The pictures had been created from authentic prompts and don’t reference copyrighted works, fictional characters, or actual people.