Hand Gesture Recognition

Authors : Nadim Awad / Sinduja Eswar / Deepthi Gadiraju / Nandini Prabhu / Brandon Reiss
Source : Download
Quick Power Point Description : Download


We use a modified version of camshift to track the hand in the frame. We then send the backprojection to the hand gesture recognition filter that will determine the convexity defects in the convex hull around the hand. From there we can determine basic gestures such as rock, paper, scissors. Depending on the current active gesture, we manipulate either the arms or the legs of a puppet in motion builder connected through the Vicon RTE server library.

Camshift for Hand Tracking

Camshift uses color data to locate the largest sub-region of an image whose color probability distribution most closely matches a template color histogram. To do that, we first obtain a region of the image to sample for color data. We then construct a histogram containing probabilities of hue values contained in the region. When a new image is taken, it is sampled around the previous window to find the new window that best fits the histogram signature.
The image on the right :
  • To visualize the source pixels for the adapted histogram, boxes are drawn around the sampled subregions.
  • Boxes in higher shades of green are more similar to the original histogram.
  • Subregions below a threshold of similarity to the original histogram are boxed in red and ignored.
  • The cyan ellipse is the track window and the purple box is the predictive track window.

Gesture Recognition

To determine the current gesture we first compute the convex hull of the hand and obtain a sequence of convexity defects for the hull. Only thos defects having depths greater than a predefined threshold value are taken into account and used to count the number of fingers. This step disregards the insignificant defects formed over the rest of the hand. Once the gesture is identified this information along with the velocity information obtained in the predictive windowing step of the camshift module is used to control the hand and feet motion of a digital puppet.
Instead of working with the entire captured image, we provide the gesture recognition module with the back projection of the tracked hand obtained from the camshift module. This effectively suppresses the interference of defects arising out of other irrelevant objects in the scene.

OpenCV - MotionBuilder from Nadim Awad on Vimeo.