Puppeteering with Zigfu

I started off my research looking at potential technologies for puppeteering and interacting with digital versions of toy hacks in realtime, with a view to developing these into interactive exhibits.

The most promising looking solution for puppeteering, and the one we investigated first is something called Zigfu. Zigfu's ZDK development kit allows you to get the joint positions from a Microsoft Kinect sensor into other software, including a game engine called Unity3D. These can then be linked to the joints of a rigged 3D model, so that you get one to one puppeteering of the model's limbs. 

After much faffing around with Kinect drivers, I was able to get the example project running. I also did some research on Unity3D, and basic 3D modelling software so we could begin creating custom content to modify the project with.

I used Google sketchup to create some robot parts, and exported them as .obj files (a standard 3D model filetype) These could then be imported to Unity, positioned and linked to the existing model. I also customised the model's texture to have a CommuniToy logo, and attached some virtual lights to the model's hands.

This is really rather fun, and very entertaining to watch, although as you can see in the video, the tracking is not particularly smooth or accurate (especially when you turn to the side or turn around) The Kinect also requires you to do a "calibration pose" (stand like a cactus) before it will pick you up, and this can sometimes take a rather long time, with no feedback as to what you are doing wrong. Once you are tracked it also often loses track of your limbs. For these reasons, the technology was not really reliable enough at the time to be suitable for an installation, and we did not look into this much further.

In July 2014, Microsoft released the new version of their Kinect sensor for windows. It is massively improved in almost every way possible, and is extremely promising for developing puppeteering based interactions. Unfortunately, as with all new technologies initial support is extremely patchy, and it takes a while for it to be integrated into existing software, or for new products to be built around it. At the time of writing (August 2014) there is no support for the new Kinect in Zigfu, NImate, Faceshift, or any of the accessible solutions for realtime puppeteering, although this is definitely something we will be keeping a close eye on.

IdentiToy