In the paper Sparks of Artificial General Intelligence, the authors show how OpenAI’s GPT-4 is able do well in variety of tasks that be represented with text and claim it to have “a more general intelligence than previous AI models.” One of the tasks they explore is symbolic music generation. In this paper we critically analyze their results and extend the discourse around the capabilities of LLMs for music by exploring additional musical tasks and LLMs. Furthermore, we will investigate the viability of smaller models when used in conjunction with Retrieval Augmented Generation, as well as finetuning on programmatically written prompts using Quantized Low Rank Adapters. Finally, we discuss some critical aspects of LLMs as a tool for music generation.