The main focus of this project is algorithm and metadata: covert systems which influence and guide our interaction with the digital world. It is through the use of these systems that the line between audience, music and technology has become increasingly blurred, and where the themes and interest at the heart of this project lie.
One of the most prevalent uses of metadata and algorithms in the world of popular music is by the streaming service Spotify, who create endless playlists and suggestions based on the listening habits of the user. The technology used to enable this process is the Echo Nest, a music intelligence platform originally created by Tristan Jehan and Brian Whitman that was purchased by Spotify in 2014 and merged into their API in 2016. The Echo Nest enables in-depth analysis of songs in the creation of metadata used by Spotify in their bespoke user-orientated service, analysing and recording information in relation to a variety of parameters including tempo, danceability, energy, loudness and valence, as well as full detailed audio analysis (see Appendix A).
This automatised process of analysing and recording data, and particularly data based on quite subjective and humanly abstract parameters such as ‘danceability’, raises interesting questions about the current way that we interact with music through these data based systems. Where is the line between audience, technology and music? Is the audience informing the data that instructs the systems, or is the data influencing the audience? If a digital system can analyse music into data, can a digital system recreate that data into music?
It is the latter question, informed by the themes of the preceding, that forms the main line of enquiry within this piece. Can musical metadata be re-sonified into something meaningful? This project aims to explore this line of enquiry using Max to build a sonification instrument and composition using Spotify’s metadata.
One of the most influential works in the creation of this project is R. Luke Dubois’ Timelapse (2006). On this album, Dubois analysed popular music listening habits by taking a “spectral average” of each US Billboard Number One between 1958-2000, condensing this into a one second clip, and playing these back in chronological order (Dubois, 2015). This results in an experience that not only can help us analyse trends in the sonic summation of popular music over this period, but also enables us to look at the audience in a decontextualised way.
The period of time covered by Dubois ends around the time that music consumption began moving ever increasingly towards digital download and streaming, with the release of Timelapse coinciding with the year that Spotify was founded. Where Dubois’ work dissected the relationship between audience and music aided by a breakdown of the sonic data provided by the audio itself, this work attempts to analyse the same area with the additional lens of metadata, breaking down and reconstructing this into a new sonic composition to analyse the connection/disconnection between data and source.
As the project solidified into a work of sonification, parameters had to be set to enable it to fit within its own scope and boundaries. The framework provided by Timelapse offered a lot of help, both within its construct of chronological weekly number ones and its allocation of one second per week. Due to the intent of examining a period in which algorithms and metadata became more widely used by digital platforms, the time period of sonification was chosen to be approximately the last decade. The location of the audience was chosen to be the UK, due to that being the location the project was being created, so the data being analysed and sonified would be from the UK Charts (n.d.).
Another decision in the preliminary process was to add a personal and political touch to the project parameters, aiming the lens at a period that had preexisting personal and societal events and outcomes. The resulting chosen parameters of the project were every UK No.1 during the period of Conservative Party leadership in the 21st Century. This period was chosen as it covers the period in which Spotify became one of the hegemonic music listening platforms and algorithms became widely used in music consumption, but also due to the period of time containing many ‘interesting’ political and societal events, such as Brexit, Covid-19, and austerity. This enabled the project to explore questions such as ‘what effect did the introduction of Universal Credit have on the danceability of pop music’ (and many others)? Questions such as these, as well as those asked throughout this documentation are key thematic considerations in the creation of this project.
A final consideration was made regarding the desired outcome of this project: although some trends and analytical insights may be ascertained from the final result, this was not intended to be a scientific experiment. Its intentions are that of aesthetic sonification and societal commentary.
The possible technical processes and parameters that could be used to accomplish the goal of this work were wide-ranging and potentially limitless, with most being chosen either for their accessibility within the confines of the project's timeframe, and others being the result of subjective creative whim.
The most crucial element was to find a way of accessing Spotify’s metadata through its API, which was done through the use of Node.js (2014) and the node.script object in Max (see Appendix B). This enabled communication between the Max and Spotify APIs for the transferring of messages and data between the two. The initial hope was to get the sonification instrument requesting and using the data in realtime, however due to the scope of the project (654 weeks of Number Ones within as many seconds), and some issues within the Spotify API in regards to advanced search options this was not currently seen as viable. Instead, an automated system was created to communicate via Python between the UK No.1 chart data (Official Charts Company, n.d.) and the Spotify API, outputting all of the required metadata into a spreadsheet. This was achieved through a collaboration with programmer Craig Tinney (Lo-Fi Games).
Having gathered the core data, the rest of the technical process involved the creation of the instrument and systems within Max to enable its re-sonification. A full insight and breakdown into the choices and techniques used is available in the supplementary material Main Project Patch Breakdown (Weallans, 2022c). Guidance was taken from The Sonification Handbook (Hermann et al., 2011) and specifically the chapters on Parameter Mapping Sonification (Grond & Berger, 2011) and Model-Based Sonification (Hermann, 2011), as to what processes were optimum. The musical nature of the data allowed for a re-musicalisation, which was a unique position to be in. Parameter mapping was therefore used as the primary model, with keys and mode mapped to note playback, and other data fields mapped across parameters such as octave, volume, dynamic processing, effects, drum probability and segment tempo.
In conclusion, can audio metadata be re-sonified back into music? The answer is yes, and the results are surprisingly coherent and often beautiful. This is in part due to the processes that the piece allowed itself as a work of subjective art. This composition is not meant to be scientific, and therefore adopts aesthetic choices based on personal opinion. This process allows for similar systems to be created by other people with alternative choices and interpretations, but perhaps obscures a definitive answer or an objective truth about the sonification of metadata.
One key example of this clouding is within the parameter setting. The audio features defined by Spotify’s API (see Appendix A) are varied in what they represent, with some being more abstract than others. There are brief explanations of how this data is analysed and interpreted by the Echo Nest, with varying degrees of specificity. For this project, the concepts that the metadata explores (such as danceability and valence) were taken and interpreted by a human mind to distribute across values and parameters within the instrument and composition. A more scientific approach that could and should be explored would be to try to reproduce these exact systems in the sonification process to try to eliminate the human element.
Where this piece explored a time period of popular music, creating a new composition from condensed snippets of information, the themes and systems could also be explored through the recreation of entire songs. Where this project used the metadata within Spotify API’s ‘audio features’ (Spotify, n.d.-a) within another section of the API there is data on the audio track analysis where a song is broken down into ‘sections’ (Spotify, n.d.-b). These sections are similar to samples, and have individual metadata reads for each section, which would allow a song to be reconstructed from a sonification of its individual parts. The data kept within these sections is slightly different from the audio features, and doesn’t contain danceability, valence or energy, which were integral to the exploration of this project. However, this concept could be explored in the future in interesting ways.
A further aspect that should be considered within the possible progression of the project is the incomplete nature of the data collection period. The UK still finds itself under Conservative Leadership, and the UK Charts continue to crown a number 1 every week. Whilst a real-time data solution was not found within the timeframe of this iteration of the project, this does not mean it cannot be achieved (a similar system and thematic area is used with James Bulley and Daniel Jones’ installation Maelstrom (2011). Achieving this autonomously updating system would allow the project to better emulate the digital world and system that informs and inspires it, grounding it deeper within its core themes.
Bulley, J., & Jones, D. (2011). Maelstrom [Sound Installation]. Barbican. https://www.jones-bulley.com/maelstrom/
Dubois, R. L. (2006, May 30). Timelapse. Cantaloupe Music.
Dubois, R. L. (2015, July 6). Timelapse, by R. Luke Dubois. Timelapse Bandcamp. https://rlukedubois.bandcamp.com/album/timelapse
Grond, F., & Berger, J. (2011). Parameter Mapping Sonification. In A. Hunt, T. Hermann, & J. G. Neuhoff (Eds.), The Sonification Handbook (pp. 363–398). Logos Publishing House.
Hansen, M. B. N. (2015). Feed-forward : on the future of twenty-first-century media. University Of Chicago Press.
Hermann, T., Hunt, A., & Neuhoff, J. G. (Eds.). (2011). The sonification handbook. Logos Publishing House.
Hermann, T. (2011). Model-Based Sonification. In A. Hunt, T. Hermann, & J. G. Neuhoff (Eds.), The Sonification Handbook (pp. 399–427). Logos Publishing House.
Hinde, K. (2011). About. Kathy Hinde. https://kathyhinde.co.uk/about/
Kubisch, C. (2007), Five Electrical Walks [Album], Important Records.
Marclay, C. (2000). Guitar Drag [Video]. MoMA.
Node.js Foundation. (2014). About | Node.js. Node.js. https://nodejs.org/en/about/
Official Charts Company. (n.d.). Official Singles Chart Top 40. Officialcharts.com. Retrieved November 10, 2022, from https://www.officialcharts.com/charts/uk-top-40-singles-chart/
Spotify. (n.d.-a). Web API Reference | Spotify for Developers. Developer.spotify.com. https://developer.spotify.com/documentation/web-api/reference/#/operations/get-audio-features
Spotify. (n.d.-b). Web API Reference | Spotify for Developers. Developer.spotify.com. https://developer.spotify.com/documentation/web-api/reference/#/operations/get-audio-analysis
Weallans, J. (2022c). Main Project Patch Breakdown [Video]. Innovation in Sound, DBS Institute of Sound and Digital Technologies.
Spotify API: Audio Features
The below are taken from the official developer documentation for the Spotify Web API. https://developer.spotify.com/documentation/web-api/reference/#/operations/get-audio-features
Acousticness (number<float>)
A confidence measure from 0.0 to 1.0 of whether the track is acoustic. 1.0 represents high confidence the track is acoustic.
Danceability (number<float>)
Danceability describes how suitable a track is for dancing based on a combination of musical elements including tempo, rhythm stability, beat strength, and overall regularity. A value of 0.0 is least danceable and 1.0 is most danceable.
Duration_ms (integer)
The duration of the track in milliseconds.
Energy (number<float>)
Energy is a measure from 0.0 to 1.0 and represents a perceptual measure of intensity and activity. Typically, energetic tracks feel fast, loud, and noisy. For example, death metal has high energy, while a Bach prelude scores low on the scale. Perceptual features contributing to this attribute include dynamic range, perceived loudness, timbre, onset rate, and general entropy.
Instrumentalness (number<float>)
Predicts whether a track contains no vocals. "Ooh" and "aah" sounds are treated as instrumental in this context. Rap or spoken word tracks are clearly "vocal". The closer the instrumentalness value is to 1.0, the greater likelihood the track contains no vocal content. Values above 0.5 are intended to represent instrumental tracks, but confidence is higher as the value approaches 1.0.
Key (integer)
The key the track is in. Integers map to pitches using standard Pitch Class notation. E.g. 0 = C, 1 = C♯/D♭, 2 = D, and so on. If no key was detected, the value is -1.
>= -1<= 11
Liveness (number<float>)
Detects the presence of an audience in the recording. Higher liveness values represent an increased probability that the track was performed live. A value above 0.8 provides strong likelihood that the track is live.
Loudness (number<float>)
The overall loudness of a track in decibels (dB). Loudness values are averaged across the entire track and are useful for comparing relative loudness of tracks. Loudness is the quality of a sound that is the primary psychological correlate of physical strength (amplitude). Values typically range between -60 and 0 db.
Mode (integer)
Mode indicates the modality (major or minor) of a track, the type of scale from which its melodic content is derived. Major is represented by 1 and minor is 0.
Speechiness (number<float>)
Speechiness detects the presence of spoken words in a track. The more exclusively speech-like the recording (e.g. talk show, audio book, poetry), the closer to 1.0 the attribute value. Values above 0.66 describe tracks that are probably made entirely of spoken words. Values between 0.33 and 0.66 describe tracks that may contain both music and speech, either in sections or layered, including such cases as rap music. Values below 0.33 most likely represent music and other non-speech-like tracks.
Tempo (number<float>)
The overall estimated tempo of a track in beats per minute (BPM). In musical terminology, tempo is the speed or pace of a given piece and derives directly from the average beat duration.
Time_signature (integer)
An estimated time signature. The time signature (meter) is a notational convention to specify how many beats are in each bar (or measure). The time signature ranges from 3 to 7 indicating time signatures of "3/4", to "7/4".
Valence (number<float>)
A measure from 0.0 to 1.0 describing the musical positiveness conveyed by a track. Tracks with high valence sound more positive (e.g. happy, cheerful, euphoric), while tracks with low valence sound more negative (e.g. sad, depressed, angry).
Node.script
The below is taken from the reference notes on the Cycling ‘74 website for the node.script object.
https://docs.cycling74.com/max8/refpages/node.script
Description
Control a local Node.js process from Max. Messages preceded by script control the lifecycle of the Node script (start to start, stop to stop). Other messages will be sent to the process directly.
Discussion
Node is a JavaScript framework for writing applications. To learn more about Node, visit their website.
Some Max patchers will want to do things that are difficult to do from Max. For example, it might be desirable to pull some data from a web address, then parse and analyze that data to generate synthesis parameters. Or, one might want to watch a local folder for files, triggering some action inside Max when the contents of that folder change. For these kinds of long-running tasks, which may require some kind of complex, persistent state, Node can be a very useful tool.
Another reason to use Node is to take advantage of the wealth of open source modules registered with the Node Package Manager (NPM). These packages contain libraries of JavaScript code for doing everything from parsing dates, to training neural nets, to running a web server, to writing chord progressions and much more. Node for Max helps make these resources available to Max programmers.
To communicate with Max, the Node script uses the "max-api" Node module. This set of functions simplifies the process of sending data between the Node application and Max. This API can attach handlers to Max messages; parse Max dictionaries into JSON objects; and read, update and set named dictionaries within Max. For examples on how to use this API, see the code examples included with the Node for Max package, as well as the API documentation at https://docs.cycling74.com/maxnode.docs/max-node-api/
The node.script object refers to a JavaScript source file, so it has some overlap with the js object. However, there are some important differences. Unlike js, node.script runs in a separate process. This has some benefits when it comes to parallelization, since Max and the Node script can run on separate cores. The Node script is a full application, which means that once it's started it has its own run loop, and has total freedom with respect to what code it can run when. It responds to events just like any other application, and can even have its own Graphical User Interface (GUI). Messages sent to the Node script will be executed asynchronously, since Node and Max are running in separate processes. Finally, scripts running with node.script have access to the full Node library for network communication, file I/O and more.