The Evolution of Billboard’s “Hot 100” and its Relation to US Popular Sentiment
Through analyzing lyrics from top charting songs on the Billboard Hot 100 since the 1950’s, I will try to understand the changes in the music that people listened to in the US.
Introduction
Since 1958, Billboard Magazine has published a weekly list called the Billboard Hot 100 that lists most popular songs in the US based on sales, radio play, and (more recently) online steaming. This ranking has not only seen many different genres of songs grace its list, but has also been around long enough to be present to see significant shifts in popular sentiment within America. In this project, I aim to see if one can observe these shifts in the popular American consciousness through the lyrics of these popular songs by analyzing them in R.
Data Gathering and Preparation
The Data
For this project, I have used three main sources of data. First, was this dataset that contains information (song title, artist, ranking, weeks on chart, etc.) about all of the songs that have been on the Top 100 list from 1958–2019 by week. Second, was the geniusr package (documentation here) which I used to obtain the lyrics of the songs in the first dataset. Lastly, to understand US popular sentiment by year, I used Gallup polls regarding topics such as Religion, Government, and the Economy with the intent to compare them to the average sentiment that I garner from the songs popular in that respective year.
Cleaning and Wrangling Data for Use
The dataset containing all of the songs that had ever been on the Hot 100 was large (roughly 325,000 rows) so using the Genius API to secure the lyrics of all of the songs was intractable since it roughly took ~1.4 seconds per song to obtain its lyrics. Because of this, I took some steps to make the data smaller and only kept the most significant songs within the data.
First, there were multiple occurrences of many songs within the data since they could have been on the list for multiple weeks. Therefore, I only kept the occurrence of each song in the last week it appeared on the charts in that year. Then, for each year, I kept the top 100 songs with the longest duration on the charts. This shortened the dataset from 325,000 rows to only around 6,200 rows. Finally, with this data, I was able to use the Genius API to obtain the lyrics for the songs which took roughly one hour and fifteen minutes.
Due to the limitations of the Genius API, not all of the songs I inputted were able to have their lyrics generated. But, in playing around with some regex to make the artist and song names understandable to the API, I was able to get lyrics for around 75% of the songs inputted, leaving me with around ~4,700 songs with lyrics to do some text analysis on.
Data Analysis
Wordclouds
To understand how the vocabulary of popular music changed throughout the years, I first created a wordcloud for each decade (1960’s all the way to 2010’s) using the eponymously named package in R. To keep things simple, I included the two extraneous years of 1958 and 1959 into the 1960’s .
First, lets look at the wordclouds from the 60’s and the 70's.
In these two wordclouds, we can see the most dominant theme from popular music at this time, which was romance. The two most prominent words in these decades are “love” and “baby” (seen as there as babi due to the word stemming which the lyrics were put through). In the 60’s, one can also see words such as “heart” and “girl” being pretty common, with other warm words like “summertime”, “sweet”, “kiss”, and “dream” being seen in the periphery. In the 70’s, that theme of music stayed with an added emphasis on dancing and having fun — words like “dance”, “tonight”, and “good” started to gain prevalence with dated slang such as “boogie” making appearances as well.
In these subsequent wordclouds, the words that were gaining prevalence in the 1970’s (such as “feel”, “night”, and “time”) became more central to the vocabulary used in popular music in the 80’s and 90’s. Interestingly, more colors can be seen in these two decades wordclouds as opposed to the previous two decades, indicating that the variety of words being used within popular music was increasing. One last thing to note is in 90’s is the first occurrence of the n-word within the wordcloud, which coincides with the entrance of hip-hop and rap into the mainstream in the “Golden Age of Hip Hop”. This influence of hip-hop and rap will be seen to increase in the following decades.
These final wordclouds have the most variety of words within charted music as evidenced by them having the most variety of colors in the wordclouds so far. The romance words such as “baby”, “love”, “heart”, and “girl” are still relevant while words like “night” and “good” have taken more of a backseat. The most stark contrast between these decades and all other ones from before, however, is the appearance and sharp rise of profanities within popular music. Words such as “shit”, “bitch”, and “fuck” appeared in the 2000’s and only became more central to the vernacular of popular music in the 2010's. This uptick corresponds to the period of time in which rap music became deeply embedded within popular music in the US, and became one of the most (if not the most) dominant genre of music listened to in the US.
Topic Generation
After seeing the vocabulary of popular music in flux over time, I wanted to see if a closer grouping of words can be made within each decade’s music to better understand subject matter and genre change over the years. So, rather than rely on the hand-wavy analysis I alluded to in the previous section, I wanted to see groupings of words that were commonly seen together in popular music within each decade. To do this, I used a LDA model from the r package topicmodels to try and see these topic groupings and their change over time. I split the popular music of each decade into five topics, and listed the fifteen most common words within each topic.
As we did in the above section, we will first examine the decades of the 60’s and 70's.
First, for the 1960’s, we can see that each topic generated was a different flavor of straightforward romance — from Topic 2 encompassing sad romantic songs with the word “cri” (cry) to Topic 1’s grouping which reflects some sort of romantic chase / longing with the words “man”, “wait”, “girl”, “run” and “love”. Although dancing was alluded to in some topics in the 60’s, it was not until the 70’s where that got more pronounced with songs in Topic 4 which had top words of “dance”, “boogie”, “night”, and “music”. Something that comes up here that didn’t come up in the wordclouds however is the appearance of some religious related words — “angel” in Topic 3 of 1960’s and “heaven” in Topic 5 of 1970’s, which is an observation which will become more important later in the analysis.
Now, for the topic models of the 80’s, we mostly see a continuation of the topics from the 1970’s — topics consisting of love, but some having a dance twist to them like in Topics 1 and 4. In the 90’s, however, we see a clear dance genre with Topic 3 (interestingly enough, with the song “Macarena” having a strong influence). Topics 2, 4 and 5 in the 90’s still have an pretty strong emphasis on love, but Topic 1 and 3 seem to have started to divert from that image, showing the formation of a greater variety of topics talked about in mainstream music.
Finally, for the last topic models examined, we get to the final state of popular music today. What appears to have merged are three main subsets of popular music — Dance / Club Music , Hip-Hop and Rap Music, and General Romance Music. There are some overlap like in the 2000’s Topic 4, but by the 2010’s it seems like clear topics have emerged (Topic 1 for Dance Music, Topic 2, 3, and 4 for Romance, and Topic 5 for rap music). This shows a great change from the original popular music landscape of the 1960’s and 1970’s, which was much more homogenous.
Music Sentiment over Time
Now, after gaining an understanding how the subject matter and topics of popular music have shifted over time, we can see if this change in sentiment has any similarities with the shift of public opinion on different social issues in the US. To see what the general public thought about these issues, I gathered data from Gallup polls discussing three topics — The economy, the government, and religion. Then, to quantify sentiment within the lyrics of the music, I used the syuzhet package to generate an average sentiment of songs per year observed.
The first comparison is the average sentiment of songs versus the average American’s economic outlook per year. These two trends are the least alike out of the ones examined — where the sentiment (on the whole) gradually decreases over time, the economic outlook for Americans is very volatile, varying basically from year to year.
I believe this discrepancy is due to two factors — the subject matter of popular music, and the speed at which sentiment changes. Music is pretty slow to release on average, and wouldn’t really capture the ever changing sentiment people have about the economy. On top of that, the subject matter of the economy is never covered in music, leading less reason to believe it would reflect/effect people’s outlook on the matter.
Next, is the comparison between song sentiment and Americans’ view on how their government is doing. Although there is less data for American government satisfaction, I believe we can see some commonalities in the period of time after 2000. After 2000, not only is there an increasingly steep decline in the sentiment of music, but there also exists a steep decline in the percent of American’s satisfied with the current government at the time. However, there exist some discrepancies with the trend, such as the low points in the 1970’s for government satisfaction, whereas the sentiment of music then was pretty high.
Some possible explanations for this similar trend can lie in the genre shift in music in the 2000’s. With the rise of the commonly anti-establishment and liberally explicit rap genre in the 2000’s and 2010's, sentiment within popular music definitely decreased. The cause of this rise in popularity of rap could have been in part of things such as the many conflicts that the US got involved in during the early 2000’s, as well as issues regarding discrimination within the criminal justice system which lowered Americans’ approval rating of the government. So, although government sentiment seems to be able to shift more quickly than the subject matter of music, sentiment of popular music (especially after 2000) can provide a relatively close proxy to sentiment towards the US government.
Finally, we move to the trends that I believe to have the closest correlation — average lyric sentiment of popular music and the percent of US that identifies as part of a religion. Both graphs presented have a gradual decline from the 1960’s that increases faster past the year 2000.
In this case, I believe that the decrease in the sentiment of popular music is directly caused by the decreasing amount of religious people within the US. This can be supported by all of the analysis we saw above — in the 1960’s and 1970’s, religious words such as “angel” and “heaven” were appearing in popular music. Over time, as people have become less and less religious, the mainstream has loosened up allowing 2000’s and 2010’s era rap to dominate the mainstream, whose excessive usage of profanities is one of the core aspects of the genre — something that is stereotypically seen as very anti-religious. So, in observing the change in music lyrics over time, we can see how religion has loosened its hold over the American consciousness since the 1960's.
Conclusion
Music is an artist’s form of expression, and popular music is the music that the public like and can relate to. So, its not surprising that in some cases the content of popular music reflects the public’s common sentiment at the time. However, music was not always used to address social issues so openly — from moving from only comprising of themes such love and romance to branching out to other genres, later music became a tool to reflect anti-establishment views and rejection of the status quo. Therefore, in some cases, we can use music as almost a litmus test of popular sentiment in prevalent issues that confront the American public.
