Metis Final Project: Music Composition with LSTMs
For my final project at Metis Data Science, I designed a recurrent neural network utilizing Long Short-Term Memory nodes (LSTMs) to learn patterns in the Six Cello Suites by J.S. Bach, and subsequently generate its own musical fragments. I learned a ton about deep learning and feature engineering, and was inspired to continue exploring the intersection between data and art - watch this space for further developments!
Bach and neural networks...why?
So why would you want to leverage machine learning for the purpose of generating bad music? I have to admit, this does sound like a ill-advised and totally impractical idea. After all, some of the talented folks in my cohort focused on seemingly more substantial topics, such as predicting seizures in hospital patients, or identifying patterns of insider trading.
The short answer is that I really wanted to make the capstone a personal project by making full use of my musical background and interest, having as much fun as possible and developing new questions about music and data in the process. Also, I can't resist injecting my personal projects with a bit of silliness, as seen in similar endeavors like TED-RNN.
I set out with modest goals:
- Create a model which generates "interesting" musical fragments derived from material in the Bach Suites (this is different from realistically imitating Bach)
- Develop a better understanding of the end-to-end process involved in using a deep learning model
In this context, building JohaNN was highly fruitful. Enough with the leadup, on with the implementation!
Sourcing the data and general model strategy
First, I had to find a good data representation of the Cello Suites preferably in a format parseable by the wonderful music21 library, since I wasn't about to go into transcribing from raw audio recordings, nor did I have time to do automated scanning of PDF scores.
Text-based notation formats like ABC have been used to great effect in projects like this stunning Irish folk tune bot, but data availability is patchy when it comes to classical works and they seem less well-suited for representing polyphonic structures. Also, the general idea of feeding music into a NN model as plaintext - with disregard for its inherent musical features - didn't sound as interesting to me, despite the apparent effectiveness of domain-agnostic char-RNN structures in a number of projects. Ultimately, I went with a solid MIDI rendition of the Suites found here.
Much has been written about the features of LSTMs, so I'll skip over the in-depth explanation here - in short, the robustness of the basic memory cell unit's combination of input/forget/output gates makes LSTMs rather suitable to learning long patterns in sequential data. For the purpose of this model, I chose a simple two-layer LSTM network, applying dropout after each layer.
Parsing the Bach corpus
The imported Bach MIDI files were treated as a stream of notes (and rests) using the rich set of feature extractors in music21. Roughly, this is how each note in the music is represented:
- Each note or rest is represented as a tuple:
(midi_pitch_number, beat_strength, duration_in_quarters), wheremidi_pitch_number = 0for rests, and thebeat_strengthis calculated based on metrical accent - The corpus is represented as a
listof these tuple-ized notes, and thesetversion (essentially a dictionary of the note tokens in the Bach suites) forms ann-dimensional space, wherenis the number of distinct notes and rests found in the works
Some code to accompany this step:
from music21 import converter, clef, stream, pitch, note, meter, midi
import numpy as np
KEY_SIG_OFFSET = 0
def parse_notes(midi_stream):
melody_corpus = []
last_pitch = 1
chord_buffer = []
prev_offset = 0.0
for m in midi_stream.measures(1, None):
time_sig = m.timeSignature
for nr in m.flat.notesAndRests:
offset_loc = nr.offset
# pitch = nr.pitch.pitchClass + 1 if isinstance(nr, note.Note) else 0
pitch = nr.pitch.midi if isinstance(nr, note.Note) else 0
beat_strength = round(nr.beatStrength * 4.0, 0)
duration = float(nr.quarterLength)
note_repr = (pitch, beat_strength, duration)
# note_repr = (pitch, duration)
# Handle chords
if nr.offset == prev_offset:
if note_repr[0] > 0:
chord_buffer.append(note_repr)
else:
if chord_buffer: # Choose tone from chord buffer closest to current note
chord_melody_tone = sorted(chord_buffer, key=lambda x: abs(x[0] - pitch))[0]
melody_corpus.append(chord_melody_tone)
melody_corpus.append(note_repr)
chord_buffer = []
prev_offset = nr.offset
return melody_corpus
def build_corpus(midi_files):
melody_corpus = []
for file in midi_files:
midi_stream = converter.parse(file)
midi_stream = midi_stream[0]
if '1008' in file or '1011' in file:
midi_stream.keySignature = midi_stream.keySignature.relative
key_sig = midi_stream.keySignature
print('Input file: {} ({})'.format(file, str(key_sig)))
midi_stream.transpose(KEY_SIG_OFFSET - key_sig.tonic.pitchClass, inPlace=True)
melody_corpus.extend(parse_notes(midi_stream))
# map indices for constructing matrix representations
melody_set = set(melody_corpus)
notes_indices = {note: i for i, note in enumerate(melody_set)}
indices_notes = {i: note for i, note in enumerate(melody_set)}
return melody_corpus, melody_set, notes_indices, indices_notes
Training the model
The model was built using the popular Keras framework on a Theano backend. I trained several incarnations of the model, using different sequence lengths, i.e. the number of notes in a given melodic fragment used to make note predictions. Amazon EC2 g2.2xlarge GPU-equipped instances came in handy here.
import numpy as np
from keras.models import Sequential, load_model
from keras.layers.core import Dense, Activation, Dropout
from keras.layers.recurrent import LSTM
from keras.callbacks import History, ModelCheckpoint
from keras.optimizers import RMSprop
from corpus import build_corpus
def train_model(midi_files, save_path, model_path=None, step_size=3, phrase_len=20, layer_size=128, batch_size=128, nb_epoch=1):
melody_corpus, melody_set, notes_indices, indices_notes = build_corpus(midi_files)
corpus_size = len(melody_set)
# cut the corpus into semi-redundant sequences of max_len values
# step_size = 3
# phrase_len = 20
phrases = []
next_notes = []
for i in range(0, len(melody_corpus) - phrase_len, step_size):
phrases.append(melody_corpus[i: i + phrase_len])
next_notes.append(melody_corpus[i + phrase_len])
print('nb sequences:', len(phrases))
# transform data into binary matrices
X = np.zeros((len(phrases), phrase_len, corpus_size), dtype=np.bool)
y = np.zeros((len(phrases), corpus_size), dtype=np.bool)
for i, phrase in enumerate(phrases):
for j, note in enumerate(phrase):
X[i, j, notes_indices[note]] = 1
y[i, notes_indices[next_notes[i]]] = 1
if model_path is None:
model = Sequential()
model.add(LSTM(layer_size, return_sequences=True, input_shape=(phrase_len, corpus_size)))
model.add(Dropout(0.2))
model.add(LSTM(layer_size, return_sequences=False))
model.add(Dropout(0.2))
model.add(Dense(corpus_size))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy', optimizer=RMSprop())
else:
model = load_model(model_path)
checkpoint = ModelCheckpoint(filepath=save_path,
verbose=1, save_best_only=False)
history = History()
model.fit(X, y, batch_size=batch_size, nb_epoch=nb_epoch, callbacks=[checkpoint, history])
return model, melody_corpus, melody_set, notes_indices, indices_notes
Generating fresh melodies
After training the models overnight, I hacked together a simple Flask app where you can generate new quasi-Baroque jingles in the browser!
See code for music generation below (borrows heavily from an LSTM text generation example shipped with Keras). The temperature parameter for sampling from the probability vector produced by the final softmax output is very important here - too low a value, and the predictions quickly converge to a single pitch repeated over and over again, whereas too high a value results in basically random outputs with no semblance of melodic contour or rhythmic cohesion. Generally, temperature values between 1.0 ~ 2.0 seemed to work best, which is to say that slightly smoothing out the respective class probabilities predicted by the model produced interesting, yet structured melodic sequences.
import numpy as np
from music21 import midi, stream, pitch, note, clef, instrument
def __sample(preds, temperature=1.0):
# helper function to sample an index from a probability array
preds = np.asarray(preds).astype('float64')
preds = np.log(preds) / temperature
exp_preds = np.exp(preds)
preds = exp_preds / np.sum(exp_preds)
probas = np.random.multinomial(1, preds, 1)
return np.argmax(probas)
def __predict(model, x, indices_notes, temperature):
preds = model.predict(x, verbose=0)[0]
next_index = __sample(preds, temperature)
next_val = indices_notes[next_index]
return next_val
def generate_sequence(model, seq_len, melody_corpus, melody_set, phrase_len, notes_indices, indices_notes, temperature):
gen_melody_indices = np.zeros((1, phrase_len, len(melody_set)))
start_pos = np.random.randint(0, len(melody_corpus) - phrase_len)
seed_phrase = melody_corpus[start_pos : start_pos + phrase_len]
gen_melody = seed_phrase
for _ in range(seq_len):
seed_phrase = gen_melody[-phrase_len:]
for i, note in enumerate(seed_phrase):
gen_melody_indices[0, i, notes_indices[note]] = 1
x = gen_melody_indices
next_note = __predict(model, x, indices_notes, temperature)
# seed_phrase.append(next_note)
gen_melody.append(next_note)
# seed_phrase = seed_phrase[1:]
# gen_melody = [indices_notes[i] for i in gen_melody_indices]
return gen_melody
def play_melody(gen_melody):
v = stream.Voice()
last_note_duration = 0
for n in gen_melody:
if n[0] == 0:
new_note = note.Rest()
else:
new_pitch = pitch.Pitch()
# new_pitch.midi = 59.0 + n[0] - 24
new_pitch.midi = n[0]
new_note = note.Note(new_pitch)
new_note.offset = v.highestOffset + last_note_duration
new_note.duration.quarterLength = n[2]
last_note_duration = new_note.duration.quarterLength
v.insert(new_note)
s = stream.Stream()
part = stream.Part()
part.clef = clef.BassClef()
part.append(instrument.Harpsichord())
part.insert(v)
s.insert(part)
return s
Lessons learned
As expected, this simple 2-layer LSTM model doesn't come close to approaching a real composition model. A more sophisticated network topology, trained on a larger corpus of compositions would likely perform better; for the purposes of this project, I imposed the restriction of supplying the model with only the six Cello Suites (36 movements in total). This limitation is compounded by the fact that the harmonic realizations in these works are largely implied rather than actually realized, meaning that most of the chord tones for each harmony are not played, and the listener's ear is left to fill in the gaps based on their experience with Baroque harmonic progressions.
So in a sense, this is a special sort of cold start problem, where not all of the required information (harmony) is available to the model to begin with! It would be interesting to redo this exercise on a model pretrained with Baroque harmonic progressions and counterpoint, to see if the output becomes more plausible. Others in the space have created novel network structures like "biaxial" networks or Clockwork RNNs, which present inspiration for future projects.