music animation machine

One of the people who attended the Experimental Two-Day Course by Edward Tufte last month was Stephen Malinowski, the creator of the Music Animation Machine.



Malinowski's work was shown in Tufte's course as an example of a multivariate information display. A key to interpreting the dimensions in the display:

• Musical themes are represented by color.
• The pitch of the notes are represented by placement on the vertical axis.
• The duration of the notes are represented by the horizontal length of the lines.
• The combination of different notes being played at the same time is shown by vertical overlap.
• And all of the above is being plotted over time (via scrolling).

That's five dimensions of data being mapped onto two dimensions, horizontal and vertical.

The point of this display is to help listeners who cannot read sheet music to visualize the music as if they did know how to read sheet music. By focusing on certain elements in the visualization, one can listen critically for details and hear things and anticipate things that would otherwise be difficult without any sort of visualization.

It's sort of interesting how one more clearly hears the notes that one is seeing in the display. Plus since we humans are so good at pattern recognition (relative to any computer), the visualization makes it easier for us to spot and hear little repeated themes in the music. This is the world that sheet music readers live in -- they can follow the score and see the music while they are listening to it. The great thing about Malinowski's animation is that it's intuitive enough that non-sheet readers, and terribly slow sheet readers who never learned how to read the bass clef (like me) can use it in place of sheet music. It's not meant to be a replacement of sheet music -- just an alternate presentation of it.

So what's the point? Is this just something novel and fun? Well yes, the Music Animation Machine is fun to watch and listen to, but there's a greater lesson to be told here. Visualization helps us draw out (in the figurative and literal sense) our understanding what is happening, not only in a piece of music, but in anything we are interested in studying in the world. The world is much more interesting than a line graph because it's so multivariate. However, the challenge that remains is how to effectively present multiple dimensions of information to help us understand the system at hand and tell the story of what's really happening.