In my classes, I’ve been talking about and showing off the kinds of visual analysis one can accomplish by extracting frames from video and comparing them in a montage format. This approach has been pioneered by Lev Manovich, Jeremy Douglass and others under the heading of Software Studies, and Manovich’s website discusses this technique and provides some tools for these and other cultural analytics. ImagePlot (a macro for ImageJ) is one such tool that many have had success with.

I haven’t used it much myself, though, because I found other tools a little bit more convenient for what I’ve been doing. For example, I used ffmpeg (now called avconv) and ImageMagick to produce this visualization of episodes 1 – 11 of the TV show, “Sleepy Hollow:”

episodes

In this view, you can see how colors define certain rhythms for the show, and someone familiar with a few episodes (like me) can look for color correspondence to known plot events to speculate about color corresponds in plot events for episodes I haven’t yet watched. Episode 9, for example, includes several short but sharply contrasting bands, conclusing with an ominous red streak. In episode 11, something major seems to happen about halfway through because the color palette shifts completely and irrevocably into a darker register.

Students have asked how I did this, so here are the basic commands I used. This isn’t a full-fledged tutorial; just some notes for future reference.

I’m using Ubuntu 14.10, where ffmpeg has recently been replaced by avconv, so for each episode I ran a command like this from my terminal:

$ avconv -i s01e01.mkv -r 1 s01e01/img-%05d.png

What that says is, “For input (-i), take a file called s01e01.mkv and using a rate (-r) of 1 frame per second, output a sequentially-named png image into subfolder called s01e01/.” This process took a while, and for each episode, I ended up with a folder of about 3000 PNGs, each 720 pixels wide.

That’s a lot of disk space, and looking at them at that size isn’t going to be particularly enlightening since I’ll only be able to see a few at a time. After some experimentation, I found that shrinking them down to 20 pixels wide left enough visual information to be recognizable but let the files be small enough to actually work with them. I did this with ImageMagick’s mogrify command, which lets you modify a bunch of images all in one place. It overwrites the original file instead of generating a new one, like so, for each folder of episodic PNGs:

$ mogrify *.png -resize 20

The *.png says to select every file that ends in “.png” and for that file, -resize it proportionally so that it the new version is 20 pixels wide. Again, this step took some time to complete.

Finally, a folder full of tiny PNGs isn’t very interesting, so this is where ImageMagick’s montage command comes in handy. This one took a lot of trial and error, both to get the output working at all and then to get the proportions looking the way I wanted them to. Here’s what I ended up with:

$ montage *.png -tile 25x113 -geometry 20x11+0+0 ../s01e01m.png

The important parts here is that -tile 25x113 setting which arranges the images into a canvas of 25 columns and 113 rows. The -geometry 20x11+0+0 just sets these images (which are 20 pixels wide, 11 pixels tall) into that canvas with 0 offset. This all outputs to a new file called, in this case, s01e01m.png. I imported all of those montage PNGs into GIMP to compare and arrange them as above.

And that’s it! I’m sure there are other ways to accomplish all this depending on your OS and whatever tools you have available. It’s apparently possible to extract frames with VLC, for instance, and as I mentioned, ImagePlot does work well for many people.