Gesture Search: A Tool for Fast Mobile Data Access
Yang Li
Presented at UIST'10, October 3-6, 2010, New York, New York, USA
Author Bio
Yang Li is a senior research scientist at Google and was a research associate at the University of Washington. He holds a PhD in Computer Science from the Chinese Academy of Sciences. He is primarily interested in gesture-based interaction, with many of his projects implemented on Android.
Summary
Hypothesis
How effective is a search tool for mobile phones that allows for faster data access by drawing only a few gestures? Does the search performance optimize over time? How different are user queries for the same item? Can touch events be differentiated from gestures on the basis of velocity?
Methods
To test if gestures were distinguishable from touch events, data was collected on touch events and gestures, even with gestures outside Gesture Search's list. Seven participants found contacts and locations on the phone, yielding over 560 touch events. Gestures were pulled from published data and users who opted in to sending their data. The authors measured the squareness of these events.
To test Gesture Search's effectiveness, a longitudinal study was performed on company Android users. The test was over everyday usage of the program, with a survey at the end, so data was filtered based on the level of usage. Data was logged to a server, including the size of the dataset and actions performed. The number of unique queries versus the number of unique items accessed was assessed.
Results
Gestures generally have larger bounding boxes than touch events, but possibly ambiguous gestures make a time delay necessary. Even events like scrolling tended to have narrower bounding boxes. Squareness allowed quicker prediction of gestures.
Users tended to search for contacts or applications and accessed their desired results by drawing more gestures. Most queries used two or less gestures and involved no re-writing. Roughly half the time, the top choice was selected. Most users did not access most of their available items, but the complexity of queries did not change much with variation in the size of the item dataset. Generally, a unique query was used to access a unique item, though sometimes multiple queries were used to access an item. Users generally found the program useful and liked not needing to type or navigate a UI, though they felt it needed better integration.
Contents
Data access on smartphones is hindered by the small screen size and deep hierarchies. Both desktops and smartphones have keyword-based search tools available, but the small key size of phones makes these applications less than ideal. Voice-activated tools are not always correct. Gestures are also used, but can suffer from difficulty in recognizing many different symbols.
Gesture Search, which the author developed for Android, tries to provide multiple interpretations of an entered gesture against its dataset and updates its search ranking based on selected items for faster access. The gestures are characters based on a time-out system to allow for easier user memorization. The entire screen is used to draw, so users draw on the search results. In case of ambiguity, all relatively likely results are returned, with the matching characters highlighted. A smaller version of the drawn search query is displayed on-screen. Users can erase all or part of their query through swipes and use multiple prefixes through the use of spaces.
Users do not have to do anything to start drawing. When the system cannot tell if a touch event is happening, the UI automatically processes it, but the gesture is kept in a buffer to determine if a series of touches were a character. When this is detected, the gesture layer does not send more information to the list and the color of the strokes changes. A threshold for low probability interpretations was derived. The program is optimized for real-time searching. The mapping of a partially complete query to an item is done through a probabilistic, high-order Hidden Markov Model. Items only appear if they exceed the threshold found from inference of all possible paths.
Discussion
The author produced a gesture-based system and wanted to test whether it would work both in theory and actuality. On both accounts, the data suggests the system was widely successful. His claims were backed initially by the small-scale test, but also by the public release of the program. He checked all of the details thoroughly to produce a solid application, so I am completely convinced that this is effective.
I found this was an innovative approach to searching that also learned with a greater accuracy than I expected. The flexibility of the system in character detection is a major boon for those of us who can't write very legibly, and the real-time searching is speedy.
The only real drawback I could think of for this program is the length of time needed to call up the app, for which some of the users in the study make quick workarounds. I could see how handling touch events while waiting for gestures could cause a problem with accidental selection, but this is largely prevented thanks to squareness measurements.
No comments:
Post a Comment