The Impact of Gesture Recognition on UX Design

Gesture recognition is the detection and understanding of human gestures and movements by computing devices. In the last few years, the rapid advance of gesture recognition technology along with the falling cost (and size) of 3D sensors have spurred greater adoption of touch-free interaction, creating a growing market.

We see gesture-based interfaces incorporated into more of our devices, fueling a paradigm shift in the way we interact with our personal computers. Through the combination of 3D sensors and sophisticated software to analyze the data, computers are gaining the intelligence to understand people’s natural movements enabling us to beginning to communicate with devices the way we do with each other.

But what does this new technology of gesture recognition mean for UX design and the designers creating these new user interface experiences?

Shifting our Paradigm: Touch-free Interaction and gesture recognition

Very often new technologies suffer from what Scott Jenson calls, “Default Thinking“. As Jenson puts it, “the problem is simple, but pernicious: designers think of new technologies in terms of yesterday’s tasks, failing to clearly see the real potential of the new technologies”.

It’s easy for gesture-based interfaces to fall prey to this trap. For example, common mistakes include:

• Extending a mouse + keyboard model, where your finger replaces the mouse (note: fingers rarely make good substitutes for a mouse)
• Assuming the same approach that works for touch-based systems will apply to hands-free design

Without changing the UI experience to match the input method users will feel the disconnection between them, with the likely result being a frustrating and ineffective experience.

New Input Methods Require New Design Principles

What are the implications of motion tracking and gesture recognition technology on design interfaces?

We’ve identified five key design features that are important in creating effective gesture-based interfaces. While this list will continue to be refined with time, it offers a starting toolkit for designers as they create 3D NUI experiences.

1. Remove the grid: create menus and visual UIs that consider human body ergonomics

• WIMP (Windows, Icons, Menus, Pointers) is the familiar structure of today. Open almost any computer and you will see files, programs, and other icons organized around a grid-based, horizontal & vertical layout.
• Once you incorporate gesture, though, it is immediately apparent that using the grid-based model fails to account for natural human motions. In particular, it’s uncomfortable for a person to make selections using hand gestures that require moving along a strict horizontal or vertical grid. When a user makes a selection in the far corner of the screen, the rest of the menu may be easily obscured by her hand. Furthermore, her arms will tire quickly from having to reach and move across a strict line.
• Building successful applications that include gesture will require designers to step outside the existing paradigm, observe and incorporate how users naturally move their arms and hands, and create interfaces that are well attuned to human motions.
• Example: At Omek we created an “arc-based system” that enables a user to rest their elbow on their desk while still controlling the menu, offering a much more comfortable interface. Read more about this design on our blog.

2. Lose the cursor – who needs it?

• Gesture interfaces inherently require use of our hands and fingers to control objects on a screen. So do we also need a cursor to offer the user indication and context of where they are? Perhaps not. One solution that we deployed in our Practical UI application was to create a semi-transparent “mirror” of the user’s hand. It gave a fully realistic, yet true context of where one’s hand was at any point, while remaining subtle enough that it wasn’t distracting for users.

3. “Keep your hands where I can see them” — Center design around the area that matters, the “Effective Interaction Zone”

• A key feature of an effective gesture-based application is a design that avoids “gorilla arm”. This can be accomplished by centering the selection items within what we have titled, the Effective Interaction Zone, an area within the screen that allows users to make choices without straining or lifting their arms.
• A second attribute of an Effective Interaction Zone is ensuring the user stays within the active field of view of the camera, thus maintaining consistent tracking.

4. Leverage the 3rd dimension

• 3D cameras add a new opportunity to enhance how users interact with content. Effortlessly and intuitively, a user can “pick up”, rotate, and examine an object by just turning their wrist as they would with a live object. Take, for example, a user shopping online for shoes – she can easily pick up one up and rotate it to view it from all sides. She might even choose to pick up two different shoes, one with each hand, and compare them against each other.
• What does this mean for designers? We see two major implications. First, content will need to be enhanced to support viewing objects from all sides. Second, designers can create more fun and engaging experiences for their users by taking advantage of this new way of engaging with content.

5. Use your sixth sense

• A defining and unique feature of gesture recognition is the consistent “tracking” via a 3D camera, providing the ability to predict user intention. If a user moves their hand to the right, the camera senses, or “sees” the hand moving to the right. This feature can be used to anticipate what a user intends to do – for example, as a user’s hand moves closer to the screen, the object they are reaching for may become slightly larger, making it easier for the user to make their selection.

Gesture recognition offers amazing opportunities to completely redefine how we interact with our computing devices. That said, gesture is a way to enhance and transform existing experiences, not replace them. While we don’t see the mouse and keyboard being eliminated in the foreseeable future, we do see close-range gesture recognition as a valuable and fun way to deliver effective, meaningful, and engaging experiences. Regardless of the input method used, it is always critical to study and evaluate how users actually engage with their devices and then to build the interface in support of those natural movements.