This compilation thesis explores AI techniques in three areas related to music making: neural instrument synthesis, multi-modal representations, and symbolic music generation. In neural instrument synthesis, we explore architectural changes and transfer learning techniques to apply neural synthesis methods to instruments where little data is available. We then move to zero-shot audio applications of multi-modal representations, including text-guided audio equalization, visualization of instrument sounds, and text-driven synthesizer programming. In the domain of symbolic music, we propose superposed language modelling, a generalisation of masked language modelling that enables controllable generation and editing of music using event-attribute domain constraints. We then experiment with text-driven music generation and editing with LLMs augmented with a retrieval system to fetch relevant few-shot examples, showing early signs that LLMs could challenge domain specific approaches to symbolic music generation. We then bridge the symbolic and audio domains by using an audio-domain model of human preferences as a reward to tune a symbolic music generation model, producing music which according to the preference model is better than Mozart. Reflecting on our work, we focus on data availability as the key factor in determining Music AI capabilities and that much of our work in this thesis can be seen as capability arbitrage: redirecting capabilities from data-rich domains towards data-poor domains. We conclude by speculating on music making capabilities of future AI considering the massive iceberg of data that remains unused.

Denna avhandling utforskar AI-tekniker inom tre områden relaterade till musikskapande: neural instrumentsyntes, multimodala representationer och symbolisk musikgenerering. Inom neural instrumentsyntes utforskar vi arkitekturförändringar och överföringsinlärning för att tillämpa neurala syntesmetoder på instrument där lite data finns tillgänglig. Vi övergår sedan till zero-shot-ljudtillämpningar av multimodala representationer, inklusive textguidad ljudekvalisering, visualisering av instrumentljud och textdriven synthesizerprogrammering. Inom symbolisk musik föreslår vi superponerade språkmodeller, en generalisering av maskerade språkmodeller för kontrollerbar generering och redigering av musik med event-attribut-domänbegränsningar. Vi experimenterar sedan med textdriven musikgenerering och redigering med LLM:er förstärkta med ett retrieval-system för att hämta relevanta few-shot-exempel, ett tidigt tecken på att LLM:er kan utmana domänspecifika metoder för symbolisk musikgenerering. Vi överbryggar sedan de symboliska och ljuddomänerna genom att använda en ljuddomänmodell av mänskliga preferenser som belöningssignal för att finjustera en symbolisk musikgenereringsmodell, och producerar musik som enligt preferensmodellen är bättre än Mozart. I en reflektion kring vårt arbete lyfter vi datatillgänglighet som den avgörande faktorn för musik-AI:s förmågor, och att mycket av vårt arbete kan ses som capability arbitrage: en omdirigering av förmågor från datarika domäner mot datafattiga domäner. Vi avslutar med att spekulera kring framtida AI-förmågor för musikskapande med hänsyn till det massiva isberg av data som fortfarande inte nyttjas.

Knowing which sounds can be produced by a simulated vocal model and how they are connected to its articulatory behavior is not trivial. Being able to map this out can be interesting for applications that make use of the extended capabilities of a voice, e.g., singing or vocal imitations. We present a method that achieves this for a state-of-the-art articulatory vocal model (VocalTractLab) by combining it with a recent Quality-Diversity algorithm (CMA-MAE) and audio embeddings obtained through a multi-modal pretrained model (CLAP). The text-capabilities of CLAP make it possible to steer the exploration through a text prompt. We show that the method explores more efficiently than a random sampling baseline, covering more of the measure space and achieving higher objective scores. We provide several listening examples and the source code for a scalable implementation.

Recent work has proposed training machine learning models to predict aesthetic ratings for music audio. Our work explores whether such models can be used to finetune a symbolic music generation system with reinforcement learning, and what effect this has on the system outputs. To test this, we use group relative policy optimization to finetune a piano MIDI model with Meta Audiobox Aesthetics ratings of audio-rendered outputs as the reward. We find that this optimization affects multiple low-level features of the generated outputs, and improves the average subjective ratings in a preliminary listening study with 14 participants. We also find that over-optimization dramatically reduces diversity of model outputs. Code and listening examples can be found here: https://github.com/erl-j/SMART.

A text-to-music (TTM) model should synthesize audio that reflects the concepts in a given prompt as long as it has been trained on those concepts. If a prompt references concepts that the TTM model has not been trained on then the audio it synthesizes will likely not match. This paper investigates the application of a simple gradient-based approach called textual inversion (TI) to expand the concept vocabulary of a trained TTM model without compromising the fidelity of concepts on which it has already been trained. We apply this technique to MusicGen and measure its reconstruction and editability quality, as well as its subjective quality. We see TI can expand the concept vocabulary of a pretrained TTM model, thus making it personalized and more controllable without having to finetune the entire model. 

We explore the use of neural synthesis for acoustic guitar from string-wise MIDI input. We propose four different systems and compare them with both objective metrics and subjective evaluation against natural audio and a sample-based baseline. We aiteratively develop these four systems by making various considerations on the architecture and intermediate tasks, such as predicting pitch and loudness control features. We find that formulating the control feature prediction task as a classification task rather than a regression task yields better results. Furthermore, we find that our simplest proposed system, which directly predicts synthesis parameters from MIDI input performs the best out of the four proposed systems. Audio examples and code are available.

The dominating approach for modeling sequences (e.g. text, music) with deep learning is the causal approach, which consists in learning to predict tokens sequentially given those preceding it. Another paradigm is masked language modeling, which consists of learning to predict the masked tokens of a sequence in no specific order, given all non-masked tokens. Both approaches can be used for generation, but the latter is more flexible for editing, e.g. changing the middle of a sequence. This paper investigates the viability of masked language modeling applied to Irish traditional music represented in the text-based format abc-notation. Our model, called abcMLM, enables a user to edit tunes in arbitrary ways while retaining similar generation capabilities to causal models. We find that generation using masked language modeling is more challenging, but leveraging additional information from a dataset, e.g., imputing musical structure, can generate sequences that are on par with previous models.

In the paper Sparks of Artificial General Intelligence, the authors show how OpenAI’s GPT-4 is able do well in variety of tasks that be represented with text and claim it to have “a more general intelligence than previous AI models.” One of the tasks they explore is symbolic music generation. In this paper we critically analyze their results and extend the discourse around the capabilities of LLMs for music by exploring additional musical tasks and LLMs. Furthermore, we will investigate the viability of smaller models when used in conjunction with Retrieval Augmented Generation, as well as finetuning on programmatically written prompts using Quantized Low Rank Adapters. Finally, we discuss some critical aspects of LLMs as a tool for music generation.

With the goal of building a system capable of controllablesymbolic music loop generation and editing, this paper explores a gen-eralisation of Masked Language Modelling we call Superposed LanguageModelling. Rather than input tokens being known or unknown, a Super-posed Language Model takes priors over the sequence as input, enablingus to apply various constraints to the generation at inference time. Afterdetailing our approach, we demonstrate our model across various editingtasks in the domain of multi-instrument MIDI loops. We end by high-lighting some limitations of the approach and avenues for future work.

We explore the use of large language models (LLMs) for music generation using a retrieval system to select relevant few-shot examples. We find promising initial results for music generation in a dialogue with the user, especially considering the ease with which such a system can be implemented. The code is available online.

We explore the generation of visualisations of audio latent spaces using an audio-to-image generation pipeline. We believe this can help with the interpretability of audio latent spaces. We demonstrate a variety of results on the NSynth dataset. A web demo is available.