January 9, 2011

Kinect Theremin - Sneak Peek



Not surprisingly I've contracted Kinect fever :) From the moment I heard about the Kinect I knew it would be perfect for a Theremin simulator. Well, it turns out it's not perfect, but it is super fun.

How does the interface work?
When you step in front of the Kinect controller you see a contour map of yourself in gray. Once you position your hands before you they are detected and mapped to corresponding pitch (right hand z-axis), volume (left hand y-axis) and modulation (left hand x-axis). The color hue represents the pitch where C is red, D is orange, etc. The brightness of the colors represents the volume, so soft sounds are very faint where loud sounds are bright.

Describe the gear you're using?
Aside from a Kinect controller and a PC, no special hardware is needed, unlike the Wii Theremin.
I wrote the code in C++ for Windows using the C driver package written by Stijn Kuipers / "Zephod". The program uses only the 3D depth information to detect where the performer's hands are in space, then sends corresponding MIDI messages for pitch bend, volume and modulation. I use Maple Virtual MIDI Cable to route the messages to Absynth 5.0 software synthesizer by Native Instruments. Rather than sending distinct Note On / Note Off messages, the program sends a single Note On at startup and subsequently sends only Pitch Bend messages based on your hand position. Most software synthesizers don't support fluid multi-octave pitch bend, but Absynth handles this with grace.

So what's wrong with Kinect?
Don't get me wrong, the Kinect is a great device - it enables applications like this where the user doesn't need to hold anything or don special gloves to control the system. However, unlike the Wii Theremin, there are some limitations to the Kinect hardware that make it more difficult to play musically. First off, the Kinect controller has significant latency (time between the user's action and the delivery of the data). I haven't done any precise measurements but it's probably 200 to 300 milliseconds, which is really tricky for an instrument that depends so heavily on player feedback to target a specific pitch. Even if you can position your hands precisely by muscle memory you need to hit the note a fraction of a second before you want the note to sound, which makes even moderately fast musical passages difficult to play.

Secondly, the Kinect delivers data frames just 30 times per second so when playing a smooth glissando (slide) between two notes -- a common element in Theremin performance -- the gaps between pitches are fairly audible. The Wii Theremin, on the other hand, samples data 100 times per second so glissandos are very smooth.

When can I download and play with it?
Currently I have no idea when or if I'll share the code. I'm investigating product and performance opportunities so I'm not sharing the source at this point, sorry :(

Where have you been since creating the Wii Theremin two years ago?
Working on my other big work in progress.

December 17, 2008

Wii Theremin - Santa Claus is Coming to Town

This is was my submission for NPR's National Caroling Party. The gist is that people submit their own recordings of a common Christmas carol, and NPR blends selections together into the most diverse set of voices, instruments and interpretations you could imagine. When I heard about this I knew the Wii Theremin would be a perfect 'voice' to add to the choir.



[Wondering what the heck is going on here? See my earlier post Wii Theremin - How It Works]

I'm sure they're inundated with very qualified entries, but my fingers are crossed that they'll pick mine to share with their listeners.

UPDATE: I heard back from the producer - I made the cut! So, if playing this video over and over just isn't enough Santa Claus is Coming to Town on the Wii Theremin for you, then be sure to tune in to All Things Considered on Sunday, December 21st to hear it again!

Click here for the mp3 of the Theremin solo.

Click here for the mp3 of the Theremin with piano.

Enjoy!
Ken

PS - hugs and kisses to my wife, Kali, who lit, filmed and tolerated the many attempts it took to get it (this close to) right.

PPS - I've just gotten a request for Rudolph the Infrared-Nosed Reindeer! However, Kali's eager for me to bust out Tiptoe Through the Tulips next. Should be a fun 2009 :)

December 14, 2008

Wii Theremin - Doctor Who Theme

A holiday gift for those of you who requested the Doctor Who theme played on my Wii Theremin -- my own original arrangement!

Despite my mad Google Search skillz, I couldn't find a "karaoke" version of the theme without the lead synth, so I decided to create my own.

Over the years the Doctor Who theme has become more and more modern, but to me the triplet rhythm is a little tired and limits the music's edginess -- I broke the mold and adopted a driving sixteenths rhythm reminiscent of Heart's Barracuda. I hope you like it, and I hope the ghosts of Ron Grainer and Delia Derbyshire (who composed the original theme in 1963) don't haunt me too much!




Click here to Download the MP3. It sounds much more pleasant since I recorded the lead with a keyboard, not the tricky Wii Theremin.

Happy Holidays!
Ken

November 16, 2008

More Wii Theremin Videos

Videos of me playing my homemade Wii Theremin.

Star Trek Theme:




Sound Play:


Credit and many thanks to my wife, Kali, who lit and recorded all my shots and tolerated days of miserable noisemaking!

Wii Theremin - How It Works

How I constructed a Wii Theremin with a computer, synthesizer and a Wiimote.



Léon Theremin

The Theremin, invented in 1920 by Russian physicist Léon Theremin, was one of the world's first electronic instruments. It's the only instrument that is played without being touched. The proximity of the hands to the two antennae determines the pitch and volume of the sounds produced.

You've likely heard the eerie Theremin before - they became popular in those 1950's science fiction movie soundtracks that sound so cheesy today.

While Theremins are always a popular attraction, they remain pretty rare (I've only once seen a Theremin -- at a shi-shi company event during the dot com boom). A new Theremin costs about $450, so I'm not likely to get one as a stocking stuffer, but luckily thanks to the Wii and the inspiration of Wii hacking pioneer Johnny Lee I've been able to create my own.

To be precise, I've built a Theremin simulator using a computer, a Roland JV-1080 synthesizer, and a Wiimote (remote controller from the Wii game console).

At just $35, the Wiimote is an AMAZING piece of technology. It has an infrared camera in it which tracks the position up to 4 infrared light sources. So I bought a pair of leather gloves, wired up a couple infrared LEDs to 1.5 volt batteries, and poked an LED through the tip of the index finger of each glove.

Wii Theremin glove with infrared LED fingertip


Then, I connected my Wiimote to my computer (the Wiimote also supports Bluetooth connections): building on top of Brian Peek's Wiimote hacking software library, I wrote a program which detects the two infrared gloves and converts the vertical position of the left hand to volume, and converts the horizontal position of the right hand to pitch. That information is then sent via MIDI to the synthesizer which creates the actual sound.

One awesome thing about this design is that I'm not restricted to the sine wave sound of a traditional Theremin so the sonic possibilities are endless. My father-in-law, a Vancouver Washington dentist, suggested I should make it sound like a dental drill :)

I've had a lot of fun creating my Theremin, and I've learned a lot. I think my biggest lesson, though, is that while playing the Theremin is simple in concept, it's VERY difficult to play well.

November 12, 2008

How to make an infrared LED light pen

Simple instructions and shopping list for a great LED light pen.

If you want to experiment with Johnny Lee's Wiimote Whiteboard or other Wiimote hacking projects, I recommend starting by making your own IR LED light pen. My design shown here is sturdy, self-contained, cheap and easy to make.

Infrared LED pen for Wiimote Whiteboard



Click here for printable shopping list and instructions (PDF).

Here's your shopping list (makes 2 pens):

ITEMWHEREPART #COST
EXPO markersOffice Max21671870$6.00
IR LED x 2Radio Shack276-143$4.00
LED holdersRadio Shack276-080$1.50
SPST momentary switchesRadio Shack275-1571$3.00
N-size batteriesRadio Shack23-023$5.00
N-size battery holder x 2Radio Shack270-405$2.00
Soldering kitRadio Shack64-2802$8.00

TOTAL: $29.50

Directions:

  1. Over a trash-can remove the marker tip and core with a pair of pliers (be careful as the ink can spray a bit).
  2. Wipe out the inside of the marker to remove any remaining ink.
  3. Drill a quarter-inch hole for the pushbutton switch at the spot where your thumb naturally rests on the grip. The pushbutton is a hair larger than 1/4" diameter so you'll need to grind at the edges a bit to widen the hole until the pushbutton fits in snugly.
  4. With a fine piece of sandpaper, scuff up the surface of the LED - this helps diffuse the light to improve tracking.
  5. Remove and discard the nut and washer from the LED holder.
  6. Put the LED stems through the LED holder and slide the rubber plug onto the stems until it’s tight – having the LED on the outside of the LED holder helps more light shine out to improve tracking.
  7. Solder an 8 inch length of red wire to the positive lead, which is the longer one (you can also tell by looking inside the bulb - the positive electrode is the smaller of the two).
  8. Solder an 8-inch length of black wire to the negative lead.
  9. Thread the wires through the body of the pen and use a set of pliers to screw the LED holder into the tip of the pen. This takes SIGNIFICANT force to expand the hole slightly, but the end result is nice and snug.
  10. Fish the black wire through the hole for the pushbutton.
  11. Snip the wire about an inch from the hole, then solder that to one terminal of your pushbutton.
  12. Slip the remaining black wire back into the pen body and solder it to the other pushbutton terminal.
  13. Gently press your pushbutton into place.
  14. To leave more room for the battery, trim the excess wire leaving about an inch extending from the bottom of the pen body. Also trim the battery holder wires to about 1.5 inches each.
  15. Solder the battery holder wires to those of the pen (red-to-red!) [NOTE: this is your first chance to test your connections. The IR LED is invisible to the eye but it IS visible to the CCD of digital cameras. Point your camera at the light, press the button and you should be able to see it light up on the camera's screen.]
  16. Slide the battery and wire into the back of the pen and push the end cap back on as far as it'll go comfortably.


Good luck... let me know how it goes!

November 6, 2008

Connecting a Wiimote to your PC

Clear, simple instructions on connecting a Wiimote to your PC.

You can do some pretty cool stuff these days with just a Wiimote, some software and a Bluetooth-capable computer. If you haven't seen Johnny Lee's Wiimote experiments yet you must check them out - it's required viewing!

If you're yearning to join the Wiimote hacker's club, the first skill to learn is how to make a Bluetooth connection between your Wiimote and your computer. You can find the steps elsewhere on the web, but I have a few tips that might save you some time. (Note: I'm using Windows Vista and these steps will vary on other operating systems)



First, from the Control Panel open Bluetooth Devices.

Now on the Devices tab click the Add button at the bottom.

This will bring up the Add Bluetooth Device Wizard. Click the checkbox that says "My device is set up and ready to be found".

Now, before clicking Next, press 1 and 2 simultaneously on your Wiimote. The blue lights on the wiimote should start blinking letting you know it's ready to make a connection. Click Next.

After a moment the wizard should locate a device named Nintendo RVL-CNT-01. That's your Wiimote. Select the device and click Next.

The next screen asks about setting up a Bluetooth passkey, which we don't need. So, select the fourth option - Don't use a passkey.

By this time your Wiimote has probably stopped searching for a connection (the blinking only lasts for 20 seconds). So, before clicking Next reactivate the connection mode by pressing 1 and 2 once again.

Now click Next.

That's it - if all has gone well you'll see this success dialog.

You can click Finish and know that your computer now has access to your Wiimote.
NOTE: once you've made the connection, the Nintendo device icon remains in your Bluetooth Devices list, HOWEVER, whenever the Bluetooth connection shuts down -- if your computer goes to sleep, or if the Wiimote gets turned off, for example -- you'll need to run through these steps once again. It may seem like a pain, but it only takes a couple system crashes before this will become a habit :)

Now that your Wiimote is connected you need a way to confirm that it's working -- I like this Wiimote Data Visualizer written by Matthias Shapiro.

It shows the current state of all the controls on the Wiimote and even lets you toggle the blue lights and trigger the Wiimote's Rumble feedback.



Good luck to you and happy hacking!