Online commercial artificial intelligence (AI) platforms for generating music from text prompts (AI music) are now being used by many users to create millions of music audio recordings daily. Some AI music is appearing in advertising, music playlists of restaurants and gyms, and even hit music charts, in many countries. How are users engaging with these text‑to‑music AI platforms, where text is a principal mode of interaction to specify prompts (e.g., free terms), lyrics (e.g., sung terms), and tags (e.g., high‑level stylistic terms)? What languages appear? What characterizes prompts, lyrics, and tags? How are mentions of real artists used? What kind of additional instructions (metatags) are used? To address these questions, we assemble and analyze a collection of 101, 953 songs generated from May to October 2024 by 60, 342 users of Suno and Udio. Using a combination of state‑of‑the‑art text‑embedding models, dimensionality reduction, and clustering methods, we analyze the prompts, tags, and lyrics and automatically annotate and display the processed data in interactive plots. Our results reveal prominent themes in lyrics, language preferences, and prompting strategies, as well as peculiar attempts at steering models through the use of metatags. We share our code and data resources to promote further musicological study of AI music.

Recent work has proposed training machine learning models to predict aesthetic ratings for music audio. Our work explores whether such models can be used to finetune a symbolic music generation system with reinforcement learning, and what effect this has on the system outputs. To test this, we use group relative policy optimization to finetune a piano MIDI model with Meta Audiobox Aesthetics ratings of audio-rendered outputs as the reward. We find that this optimization affects multiple low-level features of the generated outputs, and improves the average subjective ratings in a preliminary listening study with 14 participants. We also find that over-optimization dramatically reduces diversity of model outputs. Code and listening examples can be found here: https://github.com/erl-j/SMART.

Several companies now offer platforms for users to create music at unprecedented scales by textual prompting. As the quality of this music rises, concern grows about how to differentiate AI-generated music from human-made music, with implications for content identification, copyright enforcement, and music recommendation systems. This article explores the detection of AI-generated music by assembling and studying a large dataset of music audio recordings (30,000 full tracks totaling 1,770 h, 33 m, and 31 s in duration), of which 10,000 are from the Million Song Dataset (Bertin-Mahieux et al., 2011) and 20,000 are generated and released by users of two popular AI music platforms: Suno and Udio. We build and evaluate several AI music detectors operating on Contrastive Language Audio Pretraining embeddings of the music audio, then compare them to a commercial baseline system as well as an open-source one. We applied various audio transformations to see their impacts on detector performance and found that the commercial baseline system is easily fooled by simply resampling audio to 22.05 kHz. We argue that careful consideration needs to be given to the experimental design underlying work in this area, as well as the very definition of "AI music." We release all our code at https://github.com/lcrosvila/ai-music-detection.

As music generated using artificial intelligence (AI music) becomes more prevalent — originating not only from individuals but also commercial services — the need to study it and its impacts becomes important. How can this material and its sources be meaningfully studied and critically engaged with, especially considering the unprecedented scales possible with generative AI? The paper begins to answer this question by considering AI music along seven aspects: 1) the company providing an AI music service; 2) its founders and employees; 3) the use of the service; 4) the users; 5) the algorithms; 6) the music; and 7) the sustainability. We make our discussion more concrete by considering the contemporary AI music service Boomy. While our investigations are preliminary and focused on a single AI music service, we argue that they open several interesting avenues of exploration for many disciplines and their intersections to help prepare for the coming flood of AI music. This paper asks many more questions than it answers, which is a feature (not a bug) of it advocating for a new domain of study: AI Music Studies.

The dominating approach for modeling sequences (e.g. text, music) with deep learning is the causal approach, which consists in learning to predict tokens sequentially given those preceding it. Another paradigm is masked language modeling, which consists of learning to predict the masked tokens of a sequence in no specific order, given all non-masked tokens. Both approaches can be used for generation, but the latter is more flexible for editing, e.g. changing the middle of a sequence. This paper investigates the viability of masked language modeling applied to Irish traditional music represented in the text-based format abc-notation. Our model, called abcMLM, enables a user to edit tunes in arbitrary ways while retaining similar generation capabilities to causal models. We find that generation using masked language modeling is more challenging, but leveraging additional information from a dataset, e.g., imputing musical structure, can generate sequences that are on par with previous models.

In the paper Sparks of Artificial General Intelligence, the authors show how OpenAI’s GPT-4 is able do well in variety of tasks that be represented with text and claim it to have “a more general intelligence than previous AI models.” One of the tasks they explore is symbolic music generation. In this paper we critically analyze their results and extend the discourse around the capabilities of LLMs for music by exploring additional musical tasks and LLMs. Furthermore, we will investigate the viability of smaller models when used in conjunction with Retrieval Augmented Generation, as well as finetuning on programmatically written prompts using Quantized Low Rank Adapters. Finally, we discuss some critical aspects of LLMs as a tool for music generation.

With the goal of building a system capable of controllablesymbolic music loop generation and editing, this paper explores a gen-eralisation of Masked Language Modelling we call Superposed LanguageModelling. Rather than input tokens being known or unknown, a Super-posed Language Model takes priors over the sequence as input, enablingus to apply various constraints to the generation at inference time. Afterdetailing our approach, we demonstrate our model across various editingtasks in the domain of multi-instrument MIDI loops. We end by high-lighting some limitations of the approach and avenues for future work.

This paper (a product of artistic research) engages with the following challenge: combine publicly available generative AI tools to simulate a commercial radio station, complete with dialogue, news and advertisements, and music programming. Our five success criteria for the “station” are: 1) it runs autonomously; 2) it features diverse content; 3) its content is generated and assembled in faster than real-time; 4) it sounds like commercial radio; and 5) it is engaging for longer than its novelty factor. We consider a variety of generative AI systems for text and dialogue, synthesizing expressive speech, and generating music audio. We describe our engineered pipeline and illustrate its components with several audio examples. We compare our results to other “endless” streams of content. Our resulting stream — “Stochastic Pirate Radio (KSPR)” — can be heard here: https://www.youtube.com/@KSPRStochasticPirateRadio.

We explore the use of large language models (LLMs) for music generation using a retrieval system to select relevant few-shot examples. We find promising initial results for music generation in a dialogue with the user, especially considering the ease with which such a system can be implemented. The code is available online.

We explore the transformer neural network architecture for modeling music, specifically Irish and Swedish traditional dance music.Given the repetitive structures of these kinds of music, the transformer should be as successful with fewer parameters and complexity as the hitherto most successful model, a vanilla long short-term memory network.We find that achieving good performance with the transformer is not straightforward,and careful consideration is needed for the sampling strategy, evaluating intermediate outputs in relation to engineering choices, and finally analyzing what the model learns.We discuss these points with several illustrations, providing reusable insights for engineering other music generation systems. We also report the high performance of our final transformer model in a competition of music generation systems focused on a type of Swedish dance.