The journey of gesture control technology is a fascinating exploration of human-computer interaction, offering increasingly intuitive ways to engage with digital devices. From early wearables like the DataGlove, to groundbreaking systems such as the Kinect and Myo Armband and the outcomes of today in the form of neural wristbands, the evolution of this technology has transformed the landscape of digital interaction.
A Historical Perspective
Gesture control technology began its journey with Myron W. Krueger's VIDEOPLACE in 1985, enabling body movements to interact with computer graphics. It saw its first commercial success with the DataGlove (1987), which translated hand and finger movements into inputs using fiber optics and magnetic tracking systems.
In 1989, Mattel (to which the technology was licensed) released the Power Glove - a controller accessory for the Nintendo Entertainment System. It had traditional NES (Nintendo Entertainment System) joypad controller buttons on the forearm (directional pad and buttons), and buttons . The player can perform various hand motions to control a character on-screen. It could detect roll, and used sensors to detect four positions per finger -for four fingers.
While it sold nearly one million units and was a commercial success, because the controls for the glove were incredibly obtuse it became impractical for gaming. However, it was adopted by the emerging Virtual Reality community in the 1990’s to interact with 3D worlds since it was cheaper than the DataGlove.
Power Glove, American model [Wikipedia]
Microsoft Kinect
Fast-forward to 2010, Microsoft's Kinect introduced gesture control to mainstream gaming, offering full-body motion tracking and voice recognition without the need for physical controllers. The motion included the gestures:
Wave: Raise one hand and wave it side to side - used to start interactions or select items.
Push: Extend your hand forward as if pressing a button - used to select or activate items.
Swipe: Move your hand horizontally or vertically across your body - used to navigate menus or move between screens.
Raise Hand: Lift one hand above your head and hold it - used to initiate interactions or bring up menus.
Grip/Release: Close your hand into a fist to "grip" and open it to "release" - used to drag and drop objects.
Steering Wheel: Hold your hands as if gripping a steering wheel and turn them - used to simulate steering in driving games.
Leap Motion Controller
In 2013, Leap Motion launched its first product - Leap Motion Controller, a groundbreaking device that allows users to control and interact with their computers using natural hand and finger movements. The use of two infrared cameras and three LEDs allows for an interactive 3D space, tracking the precise movements of the user’s hands and fingers with great accuracy. The gestures supported by the Leap Motion Controller are:
Point: Extend a finger to point and select items.
Pinch: Pinch fingers together to grab and manipulate objects.
Swipe: Move a hand or finger horizontally or vertically to navigate menus and screens.
Circle: Move a finger in a circular motion to perform specific commands.
Grab: Close a hand into a fist to “grab” objects and open it to release them
The Leap Motion Controller revolutionized gesture control by providing high-precision tracking in a compact, affordable device, paving the way for new applications in various fields, including virtual reality, education, and digital art. It was compatible with HTC, Oculus and additional headsets and offered after-market gesture control functionality.
Thalmic Labs Myo Arnband
Then came the Myo Armband, developed by Thalmic Labs in 2014. It further refined gesture precision and versatility, with applications extending into virtual reality, education, and beyond. Myo Armband uses EMG (electromyography) sensors to detect muscle activity and motion sensors to track arm movements, allowing users to control digital devices through gestures
The Myo Armband marked a significant advancement in wearable technology since the Nintendo Power Glove. The gestures it supports are:
The Myo Armband gesture set
Double Tap: tap your index on the thumb twice - used to select items
Wave Left: Move your palm left - used for navigation or switching between items.
Wave Right: Move your palm right - used for navigation or switching between items.
Spread: Spread your fingers wide - used to pause or resume actions.
Fist: Clench your fist - used to select items or perform actions like clicking.
Rotate, Pan: wrist movements - used to adjust volume or scroll through lists.
Modern Developments and Wearable Advancements
Gesture control has since evolved from camera-based systems to integrated wearable devices, built-in gesture control in face worn devices. Innovations such as Microsoft's HoloLens in 2016, demonstrate how wearable and embedded technologies have transformed the user experience, providing intuitive control via hand gestures and movements. They include gestures like:
Bloom: This was the core system gesture, used to open the Start menu. The user would hold their hand up, palm open, then spread their fingers outward like a blooming flower.
Air Tap: The Air Tap was used for selecting or clicking on items. Users would hold their hand up in front of the HoloLens, then tap their index finger and thumb together to simulate a click.
Tap and Hold: Similar to the Air Tap, but users would hold their fingers together after the tap to allow dragging objects or interacting with more complex interface elements.
HoloLens required users to position their hand in front of their nose to perform gestures, obstructing their real-world view because the gesture camera’s field of view was at its center:
HoloLens field of view - Source: Microsoft
Gesture control technology using internal cameras and sensors has since been adopted by devices like the Oculus Quest (2019), HTC Vive Focus 3 (2021), and culminated in the Apple Vision Pro (2023), hailed by many for its accurate, natural and intuitive gestures. These devices feature hand-tracking capabilities, enabling users to navigate interfaces, manipulate objects, and perform actions within virtual spaces using simple gestures for point, click, and drag functions.
Unlike vision-based systems, wearable-based gesture control relies on snug-fit devices with hardware, such as biopotential sensors and motion trackers. These advancements ensure accuracy in detecting movements and electrical activity, making wearables ideal for versatile applications, from immersive virtual environments to everyday computing tasks.
Looking Ahead - The Future
As gesture control technology advances, its integration into wearables promises a future of seamless, natural interaction. By addressing challenges like gesture recognition accuracy and user comfort, these systems are poised to redefine how we interact with digital environments, whether for augmented reality, gaming, or professional applications.
This blog post is the first in a series of posts which portray how gesture control has evolved from wearables to vision-based gesture detection, while the gestures evolved from crude body movements to subtle finger movements. To delve deeper into the evolution of gesture control and its cutting-edge applications, explore our latest White Paper here.
Comments