Almost a year ago we organized the "Long night of research" where we opened up our labs, along with the University and other research centers, to show to people what we do. We wanted to talk about innovation and our audience was made of ordinary people and kids, so we had to talk simple and sexy, they are not necessarily somebody who gets excited by the concept of distributability and parallel computing.

For this reason I decided to create a little side project for the event, something that could relate to the concept of innovation as the result of a process. I wanted to show how you can take something interesting but old, re-implement it with new technology and get something more out if it as a result. So I thought about making a theremin, the first electronic music instrument, with devices we have at home, like a laptop and a WII controller. Ok, I know, it's been done before, the Web is full of many different attempts to make a theremin practically out of anything (my favorite is this attempt to make it out of three AM radios). My point though, was not only to implement a theremin out of a laptop+WII but, as I was using newer technology compared to 1920, to take advantage of what the technology I was using allowed me to do.

What I particularly like about the theremin is its human-machine interface, this idea of controlling the music with two hands without actually touching anything. That was the concept I wanted to keep. But the theremin controllers, two antennas, work on just one dimension each, while we move our hands on a three dimensional space. That was something I wanted to work on. Also, the theremin has just one sound while, as I wanted to digitalize the process, I could generate any sound. So, in the end, the idea was: why not take the original theremin interaction, enhance it by adding a 2D/3D experience and use it to command any kind of instrument. That was how the Therminal project was born.

Capturing the hands position is straight-forward with the built-in camera of the WII controller. It works pretty smoothly, it can capture up to four Infrared points in space and report their cartesian position, plus there's lots of code out there that allows you to collect this data with any platform (I used python and the beautiful python-cwiid lib). Once you have the position, you have to convert it to sound. This is interesting because with just one point you can already have the theremin interaction. With the theremin you have two controls: the pitch and the volume. You can map the volume to the Y axis and the pitch to the X axis and you're set. You move this Infrared emitter in this two-dimensional space, with one hand instead of two, and you generate notes. We're innovating: we don't HAVE TO copy the theremin, just get inspiration.

To generate sounds I used a pretty naive solution: a linear function to map a set from a dimension to another. Basically the position of the point on the X axis could go from 0 to 128 and I had 12 notes for 7 octaves (84 notes in total). In practice is like having a 7-octaves virtual piano in front of you: wherever you put your hand it will be on a key. Then I did the same with the volume. I mapped the position on the Y axis between 0 and 128 onto a 0-100 dimension, so that by moving my hand up and down I could control the volume (think of it from 0 to 100%). I tried different functions and in the end I chose a non-linear one which allowed me to have a more fine-grained control on the high volume range. These are two good starting points to generate a note and in particular a MIDI note. You can go deeper and map the velocity of the point to simulate the pressure on the key, for the Aftertouch. Once you have a MIDI note, you can map it to an instrument and you're done. Check out the video if my words aren't clear, it should explain everything.

So, to summarize, what the therminal code does is: poll the WII camera, collect each point position, map the point coordinates to MIDI notes, send the MIDI notes to a MIDI synthesizer (jack/alsa does this) and the software converts this note into sounds according to the instrument you've chosen. It's pretty easy and simple. You can play with it and extend the interaction. Another nice option, as I mentioned 3D experience, is to map the third dimension. You can collect 3D coordinates with two WII controllers and this software, the rest of the approach stays as-is. What to map this third dimension to is left to you :)

Here's the code on my github, please drop me a line here if you need any help or manage to do something out of it.