January 9, 2011

Kinect Theremin - Sneak Peek



Not surprisingly I've contracted Kinect fever :) From the moment I heard about the Kinect I knew it would be perfect for a Theremin simulator. Well, it turns out it's not perfect, but it is super fun.

How does the interface work?
When you step in front of the Kinect controller you see a contour map of yourself in gray. Once you position your hands before you they are detected and mapped to corresponding pitch (right hand z-axis), volume (left hand y-axis) and modulation (left hand x-axis). The color hue represents the pitch where C is red, D is orange, etc. The brightness of the colors represents the volume, so soft sounds are very faint where loud sounds are bright.

Describe the gear you're using?
Aside from a Kinect controller and a PC, no special hardware is needed, unlike the Wii Theremin.
I wrote the code in C++ for Windows using the C driver package written by Stijn Kuipers / "Zephod". The program uses only the 3D depth information to detect where the performer's hands are in space, then sends corresponding MIDI messages for pitch bend, volume and modulation. I use Maple Virtual MIDI Cable to route the messages to Absynth 5.0 software synthesizer by Native Instruments. Rather than sending distinct Note On / Note Off messages, the program sends a single Note On at startup and subsequently sends only Pitch Bend messages based on your hand position. Most software synthesizers don't support fluid multi-octave pitch bend, but Absynth handles this with grace.

So what's wrong with Kinect?
Don't get me wrong, the Kinect is a great device - it enables applications like this where the user doesn't need to hold anything or don special gloves to control the system. However, unlike the Wii Theremin, there are some limitations to the Kinect hardware that make it more difficult to play musically. First off, the Kinect controller has significant latency (time between the user's action and the delivery of the data). I haven't done any precise measurements but it's probably 200 to 300 milliseconds, which is really tricky for an instrument that depends so heavily on player feedback to target a specific pitch. Even if you can position your hands precisely by muscle memory you need to hit the note a fraction of a second before you want the note to sound, which makes even moderately fast musical passages difficult to play.

Secondly, the Kinect delivers data frames just 30 times per second so when playing a smooth glissando (slide) between two notes -- a common element in Theremin performance -- the gaps between pitches are fairly audible. The Wii Theremin, on the other hand, samples data 100 times per second so glissandos are very smooth.

When can I download and play with it?
Currently I have no idea when or if I'll share the code. I'm investigating product and performance opportunities so I'm not sharing the source at this point, sorry :(

Where have you been since creating the Wii Theremin two years ago?
Working on my other big work in progress.

8 comments:

Anonymous said...

Very cool stuff! The video effects also look really cool.

I wonder if the 30 frames/sec sampling is a hardware issue or software-defined. Seems like the hardware should be able to support a higher sampling rate.

PS About the source code: check with the licenses of the open-source Kinect drivers you used. You may be required to release your source-code under the same license.

Gareth said...

Fantastic project. I love the use of colour to indicate pitch. Do you think this would be a good tool to teach new players, or would the lag be too much of a problem to overcome?

Ken Moore said...

@Gareth, the lag is truly a pain -- even if they could get the lag down to 100ms it would be a huge impediment for beginners since hitting a single pitch is a feedback loop between the hand and ears, and if you overshoot a pitch you realize that 100ms late, then each subsequent adjustment takes 100ms... not effective!

Mojo said...

Great work, as usual! Makes me want to see what I can do with the PS Move that I just bougt my son for Christmas.

Glen Murphy said...

WE WANT MORE!

Anonymous said...

Hi.
first of all, it's a wonderful idea and a great produced product you got there.
may i ask why you had to re-write the whole thing to the kinect ? what was wrong with your first version to the wii ?
i'm about to write a motion-based software, and I want to know what will be my best choise - the wii or the kinect.

thank you very much in advance for any response.

Ken Moore said...

@Anonymous, my reason for trying the Kinect was simply the promise of not having to wear special gloves to simulate a Theremin. Some day when the Kinect's resolution and latency are much finer this should be a reasonable goal.

Randal said...

Great!