Hand Tracking for 3D editing

In this page we present the software we developed to edit 3D scenes and characters though the use of hand tracking devices.

Leap Modal Controller

LeapModalController-IllustrationWe developed a Blender addon to enable the use of a Leap Motion to edit position and orientation of objects in space using hands movement.

Download

The addon can be downloaded here: http://www.dfki.de/~fanu01/leap_modal_controller/LeapModalController-v1_0.py

For  a quick test, a Blender test scene can be found here: http://www.dfki.de/~fanu01/leap_modal_controller/LeapModalController-v1_0.blend. In the scene you will find the script (click on the “Run Script” button to initialise it) as well as a character exported from MakeHuman and a Monkey head to play around. Created and tested on Mac with Blender version 2.66.1 (or 2.66a).

Installation

  • Prerequisite. You need the “websocket-client” module (Home page https://pypi.python.org/pypi/websocket-client/). Tested with version 0.11.0.
    • Download directly the version for python 3 HERE: https://raw.githubusercontent.com/alexisheloir/slsi-nui-interaction-scripts/master/3rdParty/Downloads/websocket.py.
    • You must “install” it in the Python 3 distribution embedded in Blender.
      • On Mac :
        • Right click on the “Blender” app icon;
        • Select “Show Package Contents”;
        • Follow the path “Contents/MacOS/2.66/python/lib/python3.3/”;
      • On WIndows:
        • Explore the path similar to “C:\Program Files\Blender Foundation\Blender\2.66\python\lib\”
      • Copy the file “websocket.py” into this last “python3.3” or “lib” folder.
      • Restart Blender, if it was already open.
  • Install the LeapModalController-vX_Y.py file using the standard Blender addon installation procedure (File -> User Preferences -> Addons -> “Install from File…”). Or copy paste the script text into a script buffer and execute it.

Usage

A video tutorial is available on Youtube:

You can also download the movie from this address: http://www.dfki.de/~fanu01/leap_modal_controller/LeapModalController-VideoTutorial.mov

Once initialized in Blender, the script will override the behavior of keys G, R, T, F and H. F and H and functions tailored for MakeHuman characters.

G – Select an object or a PoseBone, press G to “grab”, and use your fingertip to move the object in the scene. Press G again to confirm or ESC to cancel.

R – Select an object or a PoseBone, press R to “rotate”, and use your hand palm to rotate the object on place. Press R again to confirm or ESC to cancel.

T – Select an object or a PoseBone, press T to “translate and rotate” at the same time, and use your hand palm to translate and rotate the object in the scene. Press T again to confirm or ESC to cancel.

F – Select a MakeHuman hand IK controller, press F, and perform “circles” in space to adjust the position of the elbow, which will rotate around the line between the shoulder and the wrist. Press F again to confirm or ESC to cancel.

H – Select a MakeHuman armature, press H to move both “hands” of the character as you would move them in first person perspective. Press H again to confirm or ESC to cancel. Works best with a camera positioned in a first person view. If auto-recording is enabled, hands controllers location and rotation is written in the timeline.

shift + H – Same as H, but in “mirror” mode. Character hands will move as you would see in a mirror. Works best with a camera framing the character frontally.

Known Issues

Using Leap SDK 0.8 we experienced problems in connecting directly to the Leap daemon through web sockets. Repeatedly connecting and disconnecting form the device cause the Leap software to hang.

To overcome this problem I developed a little python script that creates a long-term websocket connection to the Leap daemon process and forwards information on localhost through UDP packets.

The script can be downloaded here: http://www.dfki.de/~fanu01/leap_modal_controller/LeapForwarder.zip. It has been tested on Mac and python 2.7. Should run on windows flawlessly.

The Leap Modal Controller script can be configured to listen for UDP packets (instead of calling the Leap daemon) by setting to True the USE_UDP_SOCKET boolean flag at the beginning of the script:

USE_UDP_SOCKET = True

This problem should be solved from SDK 0.8.1, but we have not tested it yet.

Credits

Leap Modal Controller has been developed by Fabrizio Nunnari, in collaboration with Alexis Heloir for the Sign Language Synthesis and Interaction research group at DFKI / MMCISaarbrücken, Germany.