- [Narrator 1] With 500 million monthly users, Spotify is the world's largest music streaming service. - [Narrator 2] Spotify is the home of audio. - [Narrator 1] It's known for its personalized playlists, made with its recommendation algorithm.
- Think about users as this raw material, and then on top of the data layer, we're able to build shared models. - [Narrator 1] But relying so much on artificial intelligence has also drawn criticism from some industry experts worried about algorithmic bias. Here's how Spotify uses AI to personalize users experiences on the platform.
This is the tech behind Spotify. In the early two thousands, many people found music recommendations through top charts and early streaming platforms like Pandora and Last. fm.
- [Narrator 3] With the Last. fm app from the App Store, you can listen to great bands. - [Narrator 1] So, when Spotify entered the scene in 2008.
- It's not so much that they the first people to start using analytics to recommend music, but it was the way in which they combined various computational techniques in order to make their recommendations feel more lifelike. - [Narrator 1] Thomas Hodgson studies algorithms and artificial intelligence with a focus on how new technology from music streaming companies impact artists. - Fans who have listened to Discover Weekly and Daily Mix.
The way that they talk about them is in very human-like terms. - [Narrator 4] Discover Weekly, you magnificent. You've done it again.
- [Narrator 1] In 2014, Spotify acquired music analytics firm The Echo Nest, which blended machine learning and natural language processing to build a database of songs and artists. Spotify says this technology marked an important step in the evolution of its recommendation system. So, how does that system work?
It starts with a process called collaborative filtering. - Collaborative filtering looks at the pattern, across all of this data and tries to understand when do tracks happen to be playlisted together very often? You can think of it as building a map of music and podcast.
- [Narrator 1] That map looks something like this. Each point represents a different track in Spotify's catalog and the location of each point is determined by collaborative filtering. - Which means that these tracks go together, according to the way users have playlisted them and listened to them.
- [Narrator 1] So, if these two songs, are frequently playlisted together they will be close to each other in this map. Whereas if these songs are never playlisted together they will be farther apart in the map. But recommendations based purely on collaborative filtering aren't perfect.
For example, during the holidays, Mariah Carey's "All I want for Christmas is You," might get playlisted more frequently with "Silent Night," Even though this sounds like a pop song. ♪ Baby all I want for Christmas is you ♪ And this sounds like a Christmas Carol. ♪ Silent night ♪ If Spotify only generated recommendations, based on proximity, then users who like Mariah Carey, might get recommended "Silent Night" when they aren't interested in Christmas Carols.
To prevent this, Spotify adds another layer of analysis, called content-based filtering. This algorithm gathers metadata like the release date and label and executes a raw audio analysis. It uses metrics like dance ability and loudness to describe the sonic characteristics of the track.
These are the results for "Uptown Funk" which sounds like this. ♪ This hit, that ice cold ♪ ♪ Michelle Pfeiffer, that white gold ♪ And has the Dance Ability score of 0. 856 on a scale of zero to one.
The algorithm also dissects each track's temporal structure. Here's the visual representation of that for "Anti-Hero" by Taylor Swift. These are the beats, the bars and the sections.
Content-based filtering also takes into account attracts cultural context which means studying the lyrics and analyzing the adjectives used to describe the track in articles and blogs. These filtering techniques are not unique to Spotify, but industry experts say what sets the platform apart is the amount of user data it has and the products it creates from it. Spotify says its content based filtering technology has evolved over the years and now includes more advanced proprietary facing features.
(upbeat music) But Hodgson says the danger with algorithms is that they could reinforce existing biases. - This could mean that a particular catalog of music has more male artists than female artists. One of the dangers with machine learning is that as listeners start to engage with that catalog those biases become magnified and that this creates what's called a kind of feedback loop.
- [Narrator 1] Spotify says it's research teams evaluate and mitigate against potential algorithmic inequities and harms and strive for transparency about its impact. Another criticism is that the algorithm isn't optimized for new artists because there's no user data. This is known as the cold start problem.
Sultan says this is where human editors, play a significant role in delivering recommendations. - They're possibly some of the best people in the world at trying to understand new releases and culture and what's relevant. - [Narrator 1] But Hodgson says, the bigger concern is that certain metrics used in the platform's audio analysis might be culturally biased.
- In other parts of the world, they have musical systems and musical cultures that are entirely different. - Like this North Indian classical track for example. (gentle music) Spotify's algorithm labels its key signature as E minor which Hodgson says is inappropriate for this musical tradition.
- However, it's still the case that the music that is emerging from South Asia is being understood algorithmically, under the Western equal temperament scale. - [Narrator 1] Spotify says the audio analysis is one small part of the overall system, which takes into account many factors, before making a recommendation. Some industry experts also point to issues with how the system understands metadata for classical music.
For example, the metadata for a Tchaikovsky track, can include not just the name of the work and the artist, but also the movement, opus number and conductor. Spotify's algorithm isn't optimized for that. Apple Music, which has emerged in recent years as a competitor to Spotify, released a new app in March that the company says is designed to solve this problem.
(gentle orchestral music) Spotify says it doesn't comment on a competitor's marketing campaigns. In February, the streaming service joined the recent buzz, around generative AI. - [X] I'm X, and from this moment on I'm gonna be your own personal AI DJ on Spotify.
- [Narrator 1] The DJ gives the algorithm a human voice and offers listeners additional context, around a recommendation. - [X] Up next, I know you've been on a summer song kick lately. - [Narrator 1] Sultan says, the company is also exploring reinforcement learning, a technique that would allow the recommendation system to learn automatically based on feedback.
- It will help with the diversity of their recommendation, it will help with the longer term retention and we are trying to push the state-of-the-art in each of those, introducing new technologies, new capabilities and bringing new experiences.
