<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:media="http://search.yahoo.com/mrss/"><channel><title><![CDATA[banalities.]]></title><description><![CDATA[disorganized musings on data, music, sports, politics and more.]]></description><link>https://blog.naoya.io/</link><image><url>https://blog.naoya.io/favicon.png</url><title>banalities.</title><link>https://blog.naoya.io/</link></image><generator>Ghost 1.22</generator><lastBuildDate>Fri, 21 Aug 2026 18:50:21 GMT</lastBuildDate><atom:link href="https://blog.naoya.io/rss/" rel="self" type="application/rss+xml"/><ttl>60</ttl><item><title><![CDATA[Metis Final Project: Music Composition with LSTMs]]></title><description><![CDATA[<div class="kg-card-markdown"><p>For my final project at <a href="http://www.thisismetis.com">Metis Data Science</a>, I designed a recurrent neural network utilizing <a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/">Long Short-Term Memory</a> nodes (LSTMs) to learn patterns in the <a href="https://en.wikipedia.org/wiki/Cello_Suites_(Bach)">Six Cello Suites</a> by J.S. Bach, and subsequently generate its own musical fragments. I learned a ton about deep learning and feature engineering, and</p></div>]]></description><link>https://blog.naoya.io/metis-final-project-music-composition-with-lstms/</link><guid isPermaLink="false">5ac9ac9e5a0642089f4803da</guid><dc:creator><![CDATA[Naoya Kanai]]></dc:creator><pubDate>Tue, 27 Sep 2016 09:43:56 GMT</pubDate><media:content url="https://blog.naoya.io/content/images/2018/04/blog-shot-001.jpeg" medium="image"/><content:encoded><![CDATA[<div class="kg-card-markdown"><img src="https://blog.naoya.io/content/images/2018/04/blog-shot-001.jpeg" alt="Metis Final Project: Music Composition with LSTMs"><p>For my final project at <a href="http://www.thisismetis.com">Metis Data Science</a>, I designed a recurrent neural network utilizing <a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/">Long Short-Term Memory</a> nodes (LSTMs) to learn patterns in the <a href="https://en.wikipedia.org/wiki/Cello_Suites_(Bach)">Six Cello Suites</a> by J.S. Bach, and subsequently generate its own musical fragments. I learned a ton about deep learning and feature engineering, and was inspired to continue exploring the intersection between data and art - watch this space for further developments!</p>
<h3 id="bachandneuralnetworkswhy">Bach and neural networks...why?</h3>
<p>So why would you want to leverage machine learning for the purpose of generating bad music? I have to admit, this does sound like a ill-advised and totally impractical idea. After all, some of the talented folks in my cohort focused on seemingly more substantial topics, such as predicting seizures in hospital patients, or identifying patterns of insider trading.</p>
<p>The short answer is that I really wanted to make the capstone a personal project by making full use of my musical background and interest, having as much fun as possible and developing new questions about music and data in the process. Also, I can't resist injecting my personal projects with a bit of silliness, as seen in similar endeavors like <a href="https://medium.com/@samim/ted-rnn-machine-generated-ted-talks-3dd682b894c0">TED-RNN</a>.</p>
<p>I set out with modest goals:</p>
<ol>
<li>Create a model which generates &quot;interesting&quot; musical fragments derived from material in the Bach Suites (this is different from realistically <strong>imitating</strong> Bach)</li>
<li>Develop a better understanding of the end-to-end process involved in using a deep learning model</li>
</ol>
<p>In this context, building <a href="https://github.com/naoyak/JohaNN">JohaNN</a> was highly fruitful. Enough with the leadup, on with the implementation!</p>
<h3 id="sourcingthedataandgeneralmodelstrategy">Sourcing the data and general model strategy</h3>
<p>First, I had to find a good data representation of the Cello Suites preferably in a format parseable by the wonderful <a href="http://web.mit.edu/music21/">music21 library</a>, since I wasn't about to go into <a href="https://archive.org/details/EuroPython_2016_qQpm0AW9">transcribing from raw audio recordings</a>, nor did I have time to do <a href="http://www.peachnote.com/">automated scanning of PDF scores</a>.</p>
<p>Text-based notation formats like <a href="http://abcnotation.com/">ABC</a> have been used to great effect in projects like this <a href="http://www.eecs.qmul.ac.uk/~sturm/research/RNNIrishTrad/index.html">stunning Irish folk tune bot</a>, but data availability is patchy when it comes to classical works and they seem less well-suited for representing polyphonic structures. Also, the general idea of feeding music into a NN model as plaintext - with disregard for its inherent <em>musical</em> features - didn't sound as interesting to me, despite the apparent effectiveness of <a href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/">domain-agnostic char-RNN structures</a> in a number of projects. Ultimately, I went with a solid MIDI rendition of the Suites found <a href="http://www.kunstderfuge.com/bach/chamber.htm#Cello">here</a>.</p>
<p>Much has been written about the features of LSTMs, so I'll skip over the in-depth explanation here - in short, the robustness of the basic memory cell unit's combination of input/forget/output gates makes LSTMs rather suitable to learning long patterns in sequential data. For the purpose of this model, I chose a simple two-layer LSTM network, applying dropout after each layer.</p>
<h3 id="parsingthebachcorpus">Parsing the Bach corpus</h3>
<p>The imported Bach MIDI files were treated as a stream of notes (and rests) using the rich set of feature extractors in <code>music21</code>. Roughly, this is how each note in the music is represented:</p>
<ol>
<li>Each note or rest is represented as a tuple: <code>(midi_pitch_number, beat_strength, duration_in_quarters)</code>, where <code>midi_pitch_number = 0</code> for rests, and the <code>beat_strength</code> is <a href="http://web.mit.edu/music21/doc/moduleReference/moduleBase.html#music21.base.Music21Object.beatStrength">calculated based on metrical accent</a></li>
<li>The corpus is represented as a <code>list</code> of these tuple-ized notes, and the <code>set</code> version (essentially a dictionary of the note tokens in the Bach suites) forms an <code>n</code>-dimensional space, where <code>n</code> is the number of distinct notes and rests found in the works</li>
</ol>
<p>Some code to accompany this step:</p>
<pre><code class="language-python">from music21 import converter, clef, stream, pitch, note, meter, midi
import numpy as np


KEY_SIG_OFFSET = 0

def parse_notes(midi_stream):
    melody_corpus = []

    last_pitch = 1
    chord_buffer = []
    prev_offset = 0.0
    for m in midi_stream.measures(1, None):
        time_sig = m.timeSignature
        for nr in m.flat.notesAndRests:
            offset_loc = nr.offset
            # pitch = nr.pitch.pitchClass + 1  if isinstance(nr, note.Note) else 0
            pitch = nr.pitch.midi  if isinstance(nr, note.Note) else 0
            beat_strength = round(nr.beatStrength * 4.0, 0)
            duration = float(nr.quarterLength)

            note_repr = (pitch, beat_strength, duration)
            # note_repr = (pitch, duration)
            # Handle chords
            if nr.offset == prev_offset:
                if note_repr[0] &gt; 0:
                    chord_buffer.append(note_repr)
            else:
                if chord_buffer: # Choose tone from chord buffer closest to current note
                    chord_melody_tone = sorted(chord_buffer, key=lambda x: abs(x[0] - pitch))[0]
                    melody_corpus.append(chord_melody_tone)
                melody_corpus.append(note_repr)
                chord_buffer = []
            prev_offset = nr.offset

    return melody_corpus


def build_corpus(midi_files):
    melody_corpus = []
    for file in midi_files:
        midi_stream = converter.parse(file)
        midi_stream = midi_stream[0]
        if '1008' in file or '1011' in file:
            midi_stream.keySignature = midi_stream.keySignature.relative
        key_sig = midi_stream.keySignature
        print('Input file: {} ({})'.format(file, str(key_sig)))
        midi_stream.transpose(KEY_SIG_OFFSET - key_sig.tonic.pitchClass, inPlace=True)
        melody_corpus.extend(parse_notes(midi_stream))
    # map indices for constructing matrix representations
    melody_set = set(melody_corpus)
    notes_indices = {note: i for i, note in enumerate(melody_set)}
    indices_notes = {i: note for i, note in enumerate(melody_set)}

    return melody_corpus, melody_set, notes_indices, indices_notes

</code></pre>
<h3 id="trainingthemodel">Training the model</h3>
<p>The model was built using the popular <a href="https://keras.io/">Keras</a> framework on a <a href="http://deeplearning.net/software/theano/">Theano</a> backend. I trained several incarnations of the model, using different sequence lengths, i.e. the number of notes in a given melodic fragment used to make note predictions. Amazon EC2 g2.2xlarge GPU-equipped instances came in handy here.</p>
<pre><code class="language-python">import numpy as np
from keras.models import Sequential, load_model
from keras.layers.core import Dense, Activation, Dropout
from keras.layers.recurrent import LSTM
from keras.callbacks import History, ModelCheckpoint
from keras.optimizers import RMSprop

from corpus import build_corpus



def train_model(midi_files, save_path, model_path=None, step_size=3, phrase_len=20, layer_size=128, batch_size=128, nb_epoch=1):

    melody_corpus, melody_set, notes_indices, indices_notes = build_corpus(midi_files)

    corpus_size = len(melody_set)

    # cut the corpus into semi-redundant sequences of max_len values
    # step_size = 3
    # phrase_len = 20
    phrases = []
    next_notes = []
    for i in range(0, len(melody_corpus) - phrase_len, step_size):
        phrases.append(melody_corpus[i: i + phrase_len])
        next_notes.append(melody_corpus[i + phrase_len])
    print('nb sequences:', len(phrases))

    # transform data into binary matrices
    X = np.zeros((len(phrases), phrase_len, corpus_size), dtype=np.bool)
    y = np.zeros((len(phrases), corpus_size), dtype=np.bool)
    for i, phrase in enumerate(phrases):
        for j, note in enumerate(phrase):
            X[i, j, notes_indices[note]] = 1
        y[i, notes_indices[next_notes[i]]] = 1
    if model_path is None:
        model = Sequential()
        model.add(LSTM(layer_size, return_sequences=True, input_shape=(phrase_len, corpus_size)))
        model.add(Dropout(0.2))
        model.add(LSTM(layer_size, return_sequences=False))
        model.add(Dropout(0.2))
        model.add(Dense(corpus_size))
        model.add(Activation('softmax'))

        model.compile(loss='categorical_crossentropy', optimizer=RMSprop())

    else:
        model = load_model(model_path)

    checkpoint = ModelCheckpoint(filepath=save_path,
        verbose=1, save_best_only=False)
    history = History()
    model.fit(X, y, batch_size=batch_size, nb_epoch=nb_epoch, callbacks=[checkpoint, history])

    return model, melody_corpus, melody_set, notes_indices, indices_notes
</code></pre>
<h3 id="generatingfreshmelodies">Generating fresh melodies</h3>
<p>After training the models overnight, I hacked together a simple Flask app where you can generate new quasi-Baroque jingles in the browser!</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/MD4ySjFfsH4" frameborder="0" allowfullscreen></iframe>
<p>See code for music generation below (borrows heavily from an LSTM text generation example shipped with Keras). The temperature parameter for sampling from the probability vector produced by the final softmax output is very important here - too low a value, and the predictions quickly converge to a single pitch repeated over and over again, whereas too high a value results in basically random outputs with no semblance of melodic contour or rhythmic cohesion. Generally, temperature values between 1.0 ~ 2.0 seemed to work best, which is to say that slightly smoothing out the respective class probabilities predicted by the model produced interesting, yet structured melodic sequences.</p>
<pre><code class="language-python">
import numpy as np
from music21 import midi, stream, pitch, note, clef, instrument

def __sample(preds, temperature=1.0):
    # helper function to sample an index from a probability array
    preds = np.asarray(preds).astype('float64')
    preds = np.log(preds) / temperature
    exp_preds = np.exp(preds)
    preds = exp_preds / np.sum(exp_preds)
    probas = np.random.multinomial(1, preds, 1)
    return np.argmax(probas)

def __predict(model, x, indices_notes, temperature):
    preds = model.predict(x, verbose=0)[0]
    next_index = __sample(preds, temperature)
    next_val = indices_notes[next_index]

    return next_val

def generate_sequence(model, seq_len, melody_corpus, melody_set, phrase_len, notes_indices, indices_notes, temperature):
    gen_melody_indices = np.zeros((1, phrase_len, len(melody_set)))
    start_pos = np.random.randint(0, len(melody_corpus) - phrase_len)
    seed_phrase = melody_corpus[start_pos : start_pos + phrase_len]
    gen_melody = seed_phrase


    for _ in range(seq_len):
        seed_phrase = gen_melody[-phrase_len:]
        for i, note in enumerate(seed_phrase):
            gen_melody_indices[0, i, notes_indices[note]] = 1
        x = gen_melody_indices
        next_note = __predict(model, x, indices_notes, temperature)
        # seed_phrase.append(next_note)
        gen_melody.append(next_note)
        # seed_phrase = seed_phrase[1:]

#     gen_melody = [indices_notes[i] for i in gen_melody_indices]
    return gen_melody

def play_melody(gen_melody):
    v = stream.Voice()
    last_note_duration = 0
    for n in gen_melody:
        if n[0] == 0:
            new_note = note.Rest()
        else:
            new_pitch = pitch.Pitch()
            # new_pitch.midi = 59.0 + n[0] - 24
            new_pitch.midi = n[0]
            new_note = note.Note(new_pitch)
        new_note.offset = v.highestOffset + last_note_duration
        new_note.duration.quarterLength = n[2]
        last_note_duration = new_note.duration.quarterLength
        v.insert(new_note)
    s = stream.Stream()
    part = stream.Part()
    part.clef = clef.BassClef()
    part.append(instrument.Harpsichord())
    part.insert(v)
    s.insert(part)

    return s
</code></pre>
<h3 id="lessonslearned">Lessons learned</h3>
<p>As expected, this simple 2-layer LSTM model doesn't come close to approaching a real composition model. A more sophisticated network topology, trained on a larger corpus of compositions would likely perform better; for the purposes of this project, I imposed the restriction of supplying the model with only the six Cello Suites (36 movements in total). This limitation is compounded by the fact that the harmonic realizations in these works are largely <strong>implied</strong> rather than actually <strong>realized</strong>, meaning that most of the chord tones for each harmony are not played, and the listener's ear is left to fill in the gaps based on their experience with Baroque harmonic progressions.</p>
<p>So in a sense, this is a special sort of cold start problem, where not all of the required information (harmony) is available to the model to begin with! It would be interesting to redo this exercise on a model pretrained with Baroque harmonic progressions and counterpoint, to see if the output becomes more plausible. Others in the space have created novel network structures like <a href="http://www.hexahedria.com/2015/08/03/composing-music-with-recurrent-neural-networks/">&quot;biaxial&quot; networks</a> or <a href="https://cm-gitlab.stanford.edu/tsob/musicNet/">Clockwork RNNs</a>, which present inspiration for future projects.</p>
</div>]]></content:encoded></item><item><title><![CDATA[Metis Weeks 1-3: Scraping, Modeling and Visualization]]></title><description><![CDATA[Recapping the first 3 weeks of Metis Data Science Bootcamp.]]></description><link>https://blog.naoya.io/metis-weeks-1-3/</link><guid isPermaLink="false">5ac9ac9e5a0642089f4803d9</guid><category><![CDATA[bootcamp]]></category><category><![CDATA[data-science]]></category><category><![CDATA[learning]]></category><dc:creator><![CDATA[Naoya Kanai]]></dc:creator><pubDate>Sun, 31 Jul 2016 05:15:23 GMT</pubDate><media:content url="https://blog.naoya.io/content/images/2018/04/download--3--1.png" medium="image"/><content:encoded><![CDATA[<div class="kg-card-markdown"><img src="https://blog.naoya.io/content/images/2018/04/download--3--1.png" alt="Metis Weeks 1-3: Scraping, Modeling and Visualization"><p>I just finished my 4th week (of 12) at the <a href="https://blog.naoya.io/metis-weeks-1-3/www.thisismetis.com">Metis Data Science Bootcamp</a> in SoMa, where I will be holed up until the end of September training to become a data scientist. I'll be using this space to document my progress and offer up my thoughts, both related to data science and my other interests.</p>
<p>To kick things off, I'll do a quick recap of the 2nd project assigned as part of the program, dubbed <strong>Benson</strong>. The assignment was to generate a regression model predicting box office success for movies using data acquired from the web. Skills used included web scraping (using <a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/">Beautiful Soup</a> / <a href="http://www.seleniumhq.org/">Selenium</a>), data manipulation with <a href="http://pandas.pydata.org/">pandas</a>, and our first taste of regression and machine learning with the ubiquitous <a href="https://blog.naoya.io/metis-weeks-1-3/scikit-learn.org">scikit-learn</a> library.</p>
<h3 id="scrapingfromboxofficemojo">Scraping from <a href="https://blog.naoya.io/metis-weeks-1-3/www.boxofficemojo.com">Box Office Mojo</a></h3>
<p>The first step was to acquire a large data set of movie characteristics (genre, actors, rating, etc.) and box office revenue numbers from the web. I chose to gather the entire set of movies profiled on <a href="https://blog.naoya.io/metis-weeks-1-3/www.boxofficemojo.com">Box Office Mojo</a>, a total of 16,000+ films. This consisted of 4 main steps:</p>
<ol>
<li>Building a list of movie page URLs from the alphabetical index pages</li>
<li>Visiting movie URLs with <code>BeautifulSoup</code> and retrieving raw HTML</li>
<li>Extracting relevant data (revenue figures, actors/directors, release dates)</li>
<li>Storing data in pandas <code>DataFrames</code> and CSV files for analysis</li>
</ol>
<p>Here is an example of a Box Office Mojo page for the new <a href="http://www.boxofficemojo.com/movies/?id=ghostbusters2016.htm">Ghostbusters</a> release:<br>
<img src="https://blog.naoya.io/content/images/2016/07/ex_movie_metadata-3.png" alt="Metis Weeks 1-3: Scraping, Modeling and Visualization"></p>
<p>Code for scraping the movie URL list:</p>
<pre><code class="language-python">from bs4 import BeautifulSoup as bs
import requests
import re
import csv

url = 'http://www.boxofficemojo.com/movies/alphabetical.htm?letter=NUM'
response = requests.get(url)
soup = bs(response.text, 'lxml')

letters = soup.findAll('a', href=re.compile('letter='))
letters_index = ['http://www.boxofficemojo.com{}'.format(letter['href'])
                for letter in letters][:int(len(letters)/2)]
letter_subpages = []

for letter_page in letters_index:
    response = requests.get(letter_page)
    soup = bs(response.text, 'lxml')
    subpage_links = soup.findAll('a', href=re.compile('page='))
    subpage_urls = ['http://www.boxofficemojo.com{}'.format(subpage['href'])
        for subpage in subpage_links]
    subpage_urls = subpage_urls[:int(len(subpage_urls)/2)]
    letter_subpages.extend(subpage_urls)

letters_index.extend(letter_subpages)
letters_index = sorted(letters_index, lambda x: x)
print('Ready to scrape {} subpages!'.format(len(letters_index)))

header = ['Movie', 'URL']

movie_link_data = []

for letter_subpage_url in letters_index:
    response = requests.get(letter_subpage_url)
    soup = bs(response.text, 'lxml').find(id='body')
    movie_hrefs = soup.findAll('a', href=re.compile('id='))
    for movie_href in movie_hrefs:
        row_dict = {}
        row_dict['id'] = movie_href['href'].replace('/movies/?id=', '').replace('.htm', '')
        row_dict['title'] = movie_href.text
        movie_link_data.append(row_dict)

movie_page_df = pd.DataFrame(movie_link_data)
movie_page_df.to_csv('movie_page_urls.csv', index=False)

print('Collected titles and ids for {} movies!'.format(len(movie_page_df.index)))
</code></pre>
<p><code>Collected titles and ids for 16553 movies!</code></p>
<h4 id="extractingdatafrommoviepages">Extracting data from movie pages</h4>
<p>BOM pages are nearly devoid of <code>id</code> and <code>class</code> tags, perhaps to deter users from scraping the site and gobbling up server bandwidth. This made the BeautifulSoup code rather ugly with lots of messy case handling:</p>
<pre><code class="language-python">from time import sleep
from random import randint

def scrape_movie(movie_id):
    url = 'http://www.boxofficemojo.com/movies/?id={}.htm'.format(movie_id)
    headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.0; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0'}
    response = requests.get(url, headers=headers)
    if response.status_code == 200:
        break
    else:
        print('{}: Received status code {}.'.format(movie_id, response.status_code))
        return None
    soup = bs(response.text, 'lxml').find(id='body')
    return soup

def get_revenue(movie_body):
    domestic_revenue_tag = movie_body.find(text='Domestic:')
    foreign_revenue_tag = movie_body.find(text='Foreign:')
    if not domestic_revenue_tag:
        domestic_revenue = None
    else:
        domestic_revenue = parse_currency(domestic_revenue_tag.parent.parent.next_sibling.next_sibling.text)
    if not foreign_revenue_tag:
        foreign_revenue = None
    else:
        foreign_revenue = parse_currency(foreign_revenue_tag.parent.parent.next_sibling.next_sibling.text.strip())
    return {'domestic_revenue': domestic_revenue, 'foreign_revenue': foreign_revenue}

def get_players(movie_body):
    players_table = movie_body.find(class_='mp_box_tab', text='The Players')
    if not players_table:
        return None
    else:
        players_table = players_table.next_sibling.next_sibling

    director_table = players_table.findAll('a', href=re.compile('view=Director&amp;id'))
    if not director_table:
        directors = None
    else:
        directors = [director['href'].replace('/people/chart/?view=Director&amp;id=','').replace('.htm', '') for director in director_table]
    writer_table = players_table.findAll('a', href=re.compile('view=Writer&amp;id'))
    if not writer_table:
        writers = None
    else:
        writers = [writer['href'].replace('/people/chart/?view=writer&amp;id=','').replace('.htm', '') for writer in writer_table]
    composer_table = players_table.findAll('a', href=re.compile('view=Composer&amp;id'))
    if not composer_table:
        composers = None
    else:
        composers = [composer['href'].replace('/people/chart/?view=Composer&amp;id=','').replace('.htm', '') for composer in composer_table]
    actor_table = players_table.findAll('a', href=re.compile('view=Actor&amp;id'))
    if not actor_table:
        actors = None
    else:
        actors = [actor['href'].replace('/people/chart/?view=Actor&amp;id=','').replace('.htm', '') for actor in actor_table]

    players = {
        'directors': directors,
        'actors': actors,
        'composers': composers,
    }
    return players

movie_data = []

with open('movie_data_scraped_2.csv', 'a') as csvfile:
    field_names = ['actors',
                   'budget',
                   'composers',
                   'directors',
                   'distributor',
                   'opening_domestic',
                   'domestic_revenue',
                   'foreign_revenue',
                   'genres',
                   'genre_primary',
                   'id',
                   'mpaa_rating',
                   'release_date',
                   'runtime',
                   'opening_theaters',
                   'title']
    
    writer = csv.DictWriter(csvfile, fieldnames=field_names)
#     writer.writeheader()
    for movie in movie_page_df[movie_page_df['scraped'] == False].index:
        sleep(randint(1, 5) * 0.1 + 1)
        try:
            scrape_status = {}
            movie_body = scrape_movie(movie)
            if movie_body is None:
                continue
            movie_entry = parse_movie_data(movie_body)
            movie_entry.update({'id': movie, 'title': movie_page_df.loc[movie].title})
            print(movie + ': OK') 
            movie_data.append(movie_entry)
            if len(movie_data) &gt;= 50:
                print('Writing to DataFrame!')
                movie_data_df.append(pd.DataFrame.from_records(movie_data, index='id'))
                print('Writing to CSV!')
                for row_dict in movie_data:
                    writer.writerow(row_dict)
                movie_data = []

            scrape_status.update({'id': movie, 'status': 'OK'})


        except Exception as e:
            error_log[movie] = str(e)
            print(str(e))
            scrape_status.update({'id': movie, 'status': 'error'})

        movie_scrape_status.append(scrape_status)
</code></pre>
<p>After some trial and error, my scraper was ready to grab 16k+ pages from the BOM website. Several hours later...</p>
<p><img src="https://blog.naoya.io/content/images/2016/07/Screen-Shot-2016-07-30-at-10-26-06-PM.png" alt="Metis Weeks 1-3: Scraping, Modeling and Visualization"></p>
<h3 id="examining16000rowsofmoviedata">Examining 16,000 rows of movie data</h3>
<p>After a few runs of cleaning and conversion, the data was more or less ready for analysis. Some of the steps involved:</p>
<ul>
<li>Standardizing currency variables (<code>$5 million</code> -&gt; <code>5000000</code>, <code>$500,000</code> -&gt; <code>500000</code>)</li>
<li>Converting fields with multiple categorical values to <code>dict</code>s (<code>['chloemoretz', 'minkakelly']</code> -&gt; <code>{'chloemoritz</code>: 1, 'minkakelly': 1}`)</li>
<li>Standardizing revenues and budgets to June 2016 to adjust for inflation</li>
</ul>
<h3 id="howdowetreatcategoricalvariables">How do we treat categorical variables?</h3>
<p>One topic covered in class was vectorization of categorical variables into binary columns. Behold!</p>
<pre><code class="language-python"># Vectorize categorical variables
def vectorize(df, col_name, sparse=False, prefix=None):    
    vec = DictVectorizer()
    dict_col = df.loc[:, col_name]
    vec_sparse = vec.fit_transform(dict_col)
    if sparse:
        return vec_sparse
    vec_array = vec_sparse.toarray()
    columns = ['{}_{}'.format(prefix, feature_name) for feature_name in vec.get_feature_names()] if prefix else vec.get_feature_names()
    vec_df = pd.DataFrame(data=vec_array, index=df.index, columns=columns)
    return vec_df
</code></pre>
<p><img src="https://blog.naoya.io/content/images/2016/07/Screen-Shot-2016-07-30-at-11-43-30-PM-3.png" alt="Metis Weeks 1-3: Scraping, Modeling and Visualization"></p>
<p>Now we have 800 additional variables in the DataFrame, and can visualize the top actors by number of appearances. Are we really surprised?</p>
<pre><code class="language-python">actor_appearances = sorted([(actor_col, actor_df[actor_col].sum()) for actor_col in actor_df.columns.tolist()], key=lambda x: x[1], reverse=True)
actors = [actor_count[0] for actor_count in actor_appearances]
appearances = [actor_count[1] for actor_count in actor_appearances]

n_actors = 50
plt.figure(figsize=(10, 10))
# fig, ax = plt.subplots()
# ax.set_xticklabels(actors[:50], fontsize='small', rotation='vertical')
plt.xlabel('# of film appearances')
plt.ylabel('Top 50 actors in Box Office Mojo database')
plt.barh(np.arange(n_actors), appearances[:n_actors], 0.4, tick_label=[actor.replace('actor_', '') for actor in actors][:n_actors])
# plt.ticks(np.arange(n_actors), [actor.replace('actor_', '') for actor in actors][:n_actors], fontsize='x-small', rotation='vertical')
plt.gca().invert_yaxis()
plt.tick_params(axis='both', which='major', labelsize=12)
plt.title('Number of appearances by actor: top 50')
plt.show()
</code></pre>
<p><img src="https://blog.naoya.io/content/images/2016/07/download.png" alt="Metis Weeks 1-3: Scraping, Modeling and Visualization"></p>
<p>For the time being, I did some simple feature engineering to compress the actor appearance data into 2 columns:</p>
<ul>
<li><code>actor_prev_count</code>: the # of <em>previous</em> films made by actors featured in a film</li>
<li><code>actor_prev_revenue</code>: the total revenue made by said previous films</li>
</ul>
<pre><code class="language-python">def actors_prev_exp(row, actor_df):
    actors_prev_movie_count = {}
    for actor, _ in row['actors'].items():
        appearances = actor_df.loc[(actor_df['actor_' + actor] &gt; 0) &amp; (actor_df['release_date'] &lt; row['release_date'])]
        actors_prev_movie_count[actor] = {'count': appearances.shape[0],
                                          'revenue': appearances['revenue'].sum()}
    row['actors_prev'] = actors_prev_movie_count
    return row

df = df.apply(actors_prev_exp, axis=1, actor_df=actor_df)
df['actor_prev_count'] = df['actors_prev'].apply(lambda x: sum([v['count'] for v in list(x.values())]))
df['actor_prev_rev'] = df['actors_prev'].apply(lambda x: sum([v['revenue'] for v in list(x.values())]))
</code></pre>
<h4 id="exploringvariablerelationships">Exploring variable relationships</h4>
<p><a href="https://stanford.edu/~mwaskom/software/seaborn/index.html">Seaborn</a> comes in handy for visualizing variable-pair relationships and generating ideas for additional features.</p>
<pre><code class="language-python">import seaborn as sns
sns.pairplot(df_vars[[&quot;budget_parsed_adj&quot;, &quot;opening_theaters&quot;, &quot;actor_prev_rev&quot;, &quot;domestic_revenue_adj&quot;]])
</code></pre>
<p><img src="https://blog.naoya.io/content/images/2016/07/download--1-.png" alt="Metis Weeks 1-3: Scraping, Modeling and Visualization"></p>
<p>I tried transforming the <code>budget_parsed_adj</code>, and <code>domestic_revenue_adj</code>, and <code>actor_prev_rev</code> variables to log values:</p>
<pre><code class="language-python">df_vars.loc[:, &quot;domestic_revenue_adj_log&quot;] = np.log(df_vars.domestic_revenue_adj)
df_vars.loc[:, &quot;budget_parsed_adj_log&quot;] = np.log(df_vars.budget_parsed_adj)
df_vars.loc[:, 'actor_prev_rev_log'] = np.log(df_vars.loc[:, 'actor_prev_rev'])
sns.pairplot(df_vars[[&quot;budget_parsed_adj_log&quot;, &quot;opening_theaters&quot;, &quot;actor_prev_rev_log&quot;, &quot;domestic_revenue_adj_log&quot;]])
</code></pre>
<p>Starting to see some cleaner relationships!<br>
<img src="https://blog.naoya.io/content/images/2016/07/download--3-.png" alt="Metis Weeks 1-3: Scraping, Modeling and Visualization"></p>
<h3 id="runningmodels">Running models</h3>
<p>Time to become the (automated) movie success whisperer! Running linear and ridge regressions produced the following results:</p>
<pre><code class="language-python"># Scaling variables
scaler = preprocessing.StandardScaler().fit(X)
X_scaled = scaler.transform(X)
y = y / y.median()

from sklearn.linear_model import LinearRegression, RidgeCV

models = {'linreg': LinearRegression(), 'ridge': RidgeCV(alphas=[0.1, 1.0, 10.0])}
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y)
for name, model in models.items():
    model.fit(X_train, y_train)
    print(name)
    print(model.coef_)
#     print(' Coefs:' + model.coef_)
    print('Training Score: {}'.format(model.score(X_train, y_train)))
    print('Test Score: {}'.format(model.score(X_test, y_test)))
</code></pre>
<pre><code>linreg
[ 0.03938255  0.04354528 -0.00163238]
Training Score: 0.5068223807389381
Test Score: 0.3734715765598262
ridge
[ 0.03923569  0.04336833 -0.00150752]
Training Score: 0.5068157184060109
Test Score: 0.37462124220183046
In [232]:
</code></pre>
<p>Somewhat respectable fit scores:<br>
<img src="https://blog.naoya.io/content/images/2016/07/plot.png" alt="Metis Weeks 1-3: Scraping, Modeling and Visualization"></p>
<h4 id="anotherapproachgradientboostingwithgenredataadded">Another approach: gradient boosting with genre data added</h4>
<p>In search of a better model, I tried applying a gradient boosting technique, this time with movie genre data (encoded as one-hot variables):</p>
<pre><code class="language-python">from sklearn.ensemble import GradientBoostingRegressor
from sklearn.grid_search import GridSearchCV

X_onehot = df_vars_onehot
scaler_1 = preprocessing.StandardScaler().fit(X_onehot)
X_scaled_1 = scaler_1.transform(X_onehot)

X_train_1, X_test_1, y_train_1, y_test_1 = train_test_split(X_scaled_1, y)

gb = GradientBoostingRegressor()
param_grid = {'learning_rate': [0.1, 0.05, 0.02, 0.01],
              'max_depth': [3, 4, 5],
              'min_samples_leaf': [3, 5, 10, 20],
              # 'max_features': [1.0, 0.3, 0.1] ## not possible in our example (only 1 fx)
              }
gs = GridSearchCV(gb, param_grid).fit(X_train_1, y_train_1)
gb_model = gs.best_estimator_
print('Training score: ' + str(gs.score(X_train_1, y_train_1)))
print('Test score: ' + str(gb_model.score(X_test_1, y_test_1)))
</code></pre>
<pre><code>0.6382110233431133
0.54062209368206093
</code></pre>
<p>Visualizing shows a better fit as well:</p>
<pre><code class="language-python">y_predict_1 = gb_model.predict(X_test_1)
plot_line = np.linspace(0.6, 1.2, 100)
plt.xlabel('log(domestic_revenue) - actual')
plt.ylabel('log(domestic_revenue) - predicted')
plt.title('Revenue prediction - Gradient Boosting with actor data')
plt.plot(plot_line, plot_line, 'g')
plt.scatter(y_test_1, y_predict_1)
plt.show()
</code></pre>
<p><img src="https://blog.naoya.io/content/images/2016/07/download--4-.png" alt="Metis Weeks 1-3: Scraping, Modeling and Visualization"></p>
<h3 id="keytakeaways">Key takeaways</h3>
<ul>
<li>Acquiring data is....a PITA</li>
<li>Models are black boxes until you take the time to understand the math and implementations</li>
<li>Even a trivial blog setup can take a while to get up and running</li>
</ul>
</div>]]></content:encoded></item></channel></rss>