Paper Reading #24: Gesture Avatar: A Technique for Operating Mobile User Interfaces Using Gestures

Reference Information
Gesture Avatar: A Technique for Operating Mobile User Interfaces Using Gestures
Hao Lu, Yang Li
Presented at CHI 2011, May 7-12, 2011, Vancouver, British Columbia, Canada

Author Bios

• Hao Lu is a graduate student in the University of Washington's Computer Science and Engineering Department and DUB Group. He is interested in new interaction methods.
• Yang Li is a senior research scientist at Google and was a research associate at the University of Washington. He holds a PhD in Computer Science from the Chinese Academy of Sciences. He is primarily interested in gesture-based interaction, with many of his projects implemented on Android.

Summary
Hypothesis
Does Gesture Avatar outperform Shift? Is it slower on large objects and faster on small ones? Is it less error-prone? Will mobile situations affect it less than Shift?

Methods
The walking tasks were performed on a treadmill. All participants used touchscreen phones extensively. A within-subjects factorial user study had half the participants use Shift then Gesture Avatar and then vice versa for the other half. For each technique, the task was performed while sitting and walking. Users were asked to locate a single highlighted target out of 24 small letter boxes. Ambiguity was simluated by controlling the distance between objects and the number of letters used. Finger positions were stabilized.

Results
Gesture Avatar is faster than Shift for small targets and slower for large ones. Medium sizes did not significantly different times. Shift while sitting was faster than Shift while standing, but Gesture Avatar found no difference. Gesture Avatar was far more consistent than Shift across the varying tests and also had lower error rates. Most participants preferred Gesture Avatar.

Contents
Touchscreen mobile devices struggle with low precision due to the size of fingers and covering up the target. Increasing the size of widgets is not always feasible. The authors developed Gesture Avatar, which combines the visibility of traditional GUIs and the casual interaction of gestures. The user's gesture is drawn to the screen and can be manipulated to affect the underlying widget. The translucent bounding box of a drawn gesture is associated with an element and can be tapped to use the widget. Gestures can be arbitrary and have multiple types of interaction. An incorrect association can be corrected with directional gestures or by dismissing it. The previous work in this area does not focus on gestures or requires significant adjustment of the UI.

The system first distinguishes taps from gestures and then displays the gesture with its bounding box and target. Gestures are first classified as characters or shapes and then identified, with shapes being compared to on-screen objects. A 2D Gaussian distribution is used to determine the target. Possible uses include a mobile browser, media player, moving the caret in a text box, and navigating a map program. In all cases, the program was wrapped into the UI as an additional interaction layer.

Gesture Avatar can be accelerated in cases of low ambiguity and can track moving targets. The stroked gesture is shown to be consistent with the avatar metaphor. It struggles with mode switching from gestures to actions.

Discussion
The authors of this paper tested Gesture Avatar against a preexisting method and supposed that theirs would perform better in many respects. Their user test was comprehensive in the range of variables, so I have no doubt that Gesture Search performs better than Shift in a number of ways.

When I was reading this paper, I couldn't help but think of Li's Gesture Search (to the point where I would accidentally type "Search" instead of "Avatar"). The upcoming release of Android 4 strives for a more consistent interface, and I can't think of what would be more consistent than combining these two systems on a mobile device. I would have tested this myself, but the interface is not yet available.

This system seems like it could help considerably with navigating webpages, much less the other examples provided. I frequently need to visit the cached version of a website in Google's search results, but usually my fingers, which are very far from huge, trigger another action instead. A targeted gesture avatar could dramatically reduce that frustration, which for me would make the software definitely worth the download for that alone.