Generating Music using an LSTM Network E4040.2017Fall.TONE.report Nikhil Kotecha nsk2147, Paul Young py2227 Columbia University Abstract A model of music needs to have the ability to recall past details and have a clear, coherent understanding of musical structure. Detailed in the paper is a neural network architecture that predicts and generates polyphonic music aligned with musical rules. The probabilistic model presented is a Bi-axial LSTM trained with a “kernel” reminiscent of a convolutional kernel. When analyzed quantitatively and qualitatively, this approach performs well in composing polyphonic music. Link to the code is provided[1]. 1. Introduction This paper describes an algorithmic approach to the generation of music. The key goal is to model and learn musical styles, then generate new musical content. This is challenging to model because it requires the function to be able to recall past information to project in the future. Further, the model has to learn the original subject and transform it. This is a non-trivial task. The next challenge is to understand the underlying substructure of the piece so that it performs the piece cohesively. It is easier to create small, non-connected subunits that do not contribute to sense of a coherent piece. One method is to train a probabilistic model of music. Model the music as a probability distribution, mapping measures, or sequences of notes based on likelihood of appearance in the corpus of training music. These probabilities are learnt from the input data without prior specification of particular musical rules. The algorithm uncovers patterns from the music alone. After the model is trained, new music is generated. This generated music comes from a sampling of the learned probability distribution. This approach is complicated by the structure of music. Structurally, most music contains a melody, or a key sequence of notes with a single instrument or vocal theme. This melody can be monodic, meaning at most one note per time step. The melody can also be polyphonic, meaning greater than one note per time step[2]. In the case of Bach's chorales, they have a polyphony, or multiple voices producing a polyphonic melody. These melodies can also have an accompaniment. This can be counterpoint, composed of one or more melodies or voices[3]. A form of accompaniment can also be a sequence of chords that provide an associated term called a harmony. The input has great bearing on the nature of the output generated. These musical details are relevant because training a probabilistic model is complicated by the multidimensionality of polyphonic music. For instance, within a single time step multiple notes can occur creating harmonic intervals. These notes can also be patterns across multiple time steps in sequence. Further, musical notes are expressed by octave, or by interval between musical pitches. Pitches one or more octaves apart are by assumption musically equivalent, creating the idea of pitch circularity. Pitch is therefore viewed as having two dimensions: height, which refers to the absolute physical frequency of the note (e.g. 440 Hz); and pitch class, which refers to relative position within the octave. Therefore, when music is moved up or down a key the absolute frequency of the note is different but the fundamental linkages between notes is preserved. This is a necessary feature of a model. Chen et al[4] offered an early paper on deep learning generated music with a limited macro structure to the entire piece. The model created small, non-connected subunits that did not contribute to a sense of a coherent composition. To effectively model music, attention needs to be paid to the structure of the music. A model of music needs to have the ability to recall past details and understand the underlying sub-structure to create a coherent piece in line with musical structure. Recurrent neural networks (RNN), and in particular long short-term memory networks (LSTM), are successful in capturing patterns occurring over time. To capture the complexity of musical structure vis a vis harmonic and melodic structure, notes at each time step should be modeled as a joint probability distribution. To account for octaves and pitch circularity, greater context is needed. Following the convolutional neural network architecture, a solution is to employ a kernel or a window of notes and sliding that kernel or convolving across surrounding notes. Inspired by Daniel Johnson’s[5] Bi-axial LSTM model, we describe a neural network architecture that generates music. The probabilistic model described is a stacked recurrent network with a structure employing a convolution-esque kernel. Presented is model of the original paper, our changes to the model, our approach to training, and generation. Our code is available [1]. 2. Methodology In this section, presented is Daniel’s Johnson’s original model followed by our extensions to the model. In the original paper there are a few models attempted to
