Paper Reading #24: Gesture Avatar: A Technique for Operating Mobile User Interfaces Using Gestures

Reference Information
Gesture Avatar: A Technique for Operating Mobile User Interfaces Using Gestures
Hao Lu, Yang Li
Presented at CHI 2011, May 7-12, 2011, Vancouver, British Columbia, Canada

Author Bios

• Hao Lu is a graduate student in the University of Washington's Computer Science and Engineering Department and DUB Group. He is interested in new interaction methods.
• Yang Li is a senior research scientist at Google and was a research associate at the University of Washington. He holds a PhD in Computer Science from the Chinese Academy of Sciences. He is primarily interested in gesture-based interaction, with many of his projects implemented on Android.

Summary
Hypothesis
Does Gesture Avatar outperform Shift? Is it slower on large objects and faster on small ones? Is it less error-prone? Will mobile situations affect it less than Shift?

Methods
The walking tasks were performed on a treadmill. All participants used touchscreen phones extensively. A within-subjects factorial user study had half the participants use Shift then Gesture Avatar and then vice versa for the other half. For each technique, the task was performed while sitting and walking. Users were asked to locate a single highlighted target out of 24 small letter boxes. Ambiguity was simluated by controlling the distance between objects and the number of letters used. Finger positions were stabilized.

Results
Gesture Avatar is faster than Shift for small targets and slower for large ones. Medium sizes did not significantly different times. Shift while sitting was faster than Shift while standing, but Gesture Avatar found no difference. Gesture Avatar was far more consistent than Shift across the varying tests and also had lower error rates. Most participants preferred Gesture Avatar.

Contents
Touchscreen mobile devices struggle with low precision due to the size of fingers and covering up the target. Increasing the size of widgets is not always feasible. The authors developed Gesture Avatar, which combines the visibility of traditional GUIs and the casual interaction of gestures. The user's gesture is drawn to the screen and can be manipulated to affect the underlying widget. The translucent bounding box of a drawn gesture is associated with an element and can be tapped to use the widget. Gestures can be arbitrary and have multiple types of interaction. An incorrect association can be corrected with directional gestures or by dismissing it. The previous work in this area does not focus on gestures or requires significant adjustment of the UI.

The system first distinguishes taps from gestures and then displays the gesture with its bounding box and target. Gestures are first classified as characters or shapes and then identified, with shapes being compared to on-screen objects. A 2D Gaussian distribution is used to determine the target. Possible uses include a mobile browser, media player, moving the caret in a text box, and navigating a map program. In all cases, the program was wrapped into the UI as an additional interaction layer.

Gesture Avatar can be accelerated in cases of low ambiguity and can track moving targets. The stroked gesture is shown to be consistent with the avatar metaphor. It struggles with mode switching from gestures to actions.

Discussion
The authors of this paper tested Gesture Avatar against a preexisting method and supposed that theirs would perform better in many respects. Their user test was comprehensive in the range of variables, so I have no doubt that Gesture Search performs better than Shift in a number of ways.

When I was reading this paper, I couldn't help but think of Li's Gesture Search (to the point where I would accidentally type "Search" instead of "Avatar"). The upcoming release of Android 4 strives for a more consistent interface, and I can't think of what would be more consistent than combining these two systems on a mobile device. I would have tested this myself, but the interface is not yet available.

This system seems like it could help considerably with navigating webpages, much less the other examples provided. I frequently need to visit the cached version of a website in Google's search results, but usually my fingers, which are very far from huge, trigger another action instead. A targeted gesture avatar could dramatically reduce that frustration, which for me would make the software definitely worth the download for that alone.

Paper Reading #20: The Aligned Rank Transform for Nonparametric Factorial Analyses Using Only ANOVA Procedures

Reference Information
The Aligned Rank Transform for Nonparametric Factorial Analyses Using Only ANOVA Procedures
Jacob O. Wobbrock, Leah Findlater, Darren Gergle, James J. Higgins
Presented at CHI 2011, May 7-12, 2011, Vancouver, British Columbia, Canada

Author Bios

• Jacob O. Wobbrock is an Associate Professor in the Information School and an Adjunct Associate Professor in the University of Washington's Department of Computer Science and Engineering. He focuses on novel interaction techniques.
• Leah Findlater will be affiliated with University of Maryland's College of Information Studies, but is currently in the Information School at the University of Washington.
• Darren Gergle is an Associate Professor in Northwestern University's Department of Communication Studies and Department of Electrical Engineering. His work is in HCI related to visual information.
• James J. Higgins is a Professor in Kansas State University's Department of Statistics.

Summary
Hypothesis
How well do aligned rank transform (ART) analysis on existing data sets correspond with the authors' results performed through other measures?

Methods
The authors used their software with data sets from published HCI work and compared the results with the original authors' findings. One case evaluated the use of ART to provide interaciton effects. The second showed how ART  is not bound to the distributional assumptions of ANOVA. The third is nonparametric testing of repeated measures data.

Results
The first case study revealed a possible interaction that the Friedman test used in the original could not have found. It also found interactions that even an inappropriate test revealed.

The second study originally found minimal interactions because the data was lognormal. The ART test revealed that the interactions were far more significant.

The third study found that ART reduced the skew in the data and revealed all of the significant interactions, which could not be found in the original study.

Contents
Nonparametric data appears frequently in multi-factor HCI experiments, but current methods are likely to violate ANOVA assumptions or do not allow for the examine of interaction effects. Methods exist to solve this problem but are not widely available or easy to use. The authors developed a generalizable system that relies on ART to align and rank data before performing F-tests. ARTool is the desktop version and ARTweb runs online.

ART is usable in similar situations as parametric ANOVA but it does not require a continuous or ordinal response variable and does not need to be normally distributed. Rank tests apply ranks to data sets, and alignment aligns data depending on the effect to remove all effects except one. The prodcedure follows five steps. The first is to compute residuals, which is the response minus the average of all response who have a factor level that matches the response in question. Then, the estimated effects are calculated for all main and interaction effects. The authors presented a generalized version for an n-way interaction for the response. The aligned response is then found by adding the results from the previous two steps. The averaged ranks are assigned with the smallest rank from the previous step receiving a rank of 1 and so on. Then a full-factorial ANOVA is performed on the result of the previous step. The two opportunities to asess correctness are that the results of the third step should have a column that sums to 0. An ANOVA on the results of the third step should show all of the effects stripped out except the effect for which the data was aligned.

ARTool parses long-format data tables and produces aligned and ranked response for all main and interaction effects. It produces descriptive error messages in cause of a problem and has an output that contains (2+N)+2(2^N-1) columns. The system does not work in case of extreme skew of data and is best in randomized designs.

Discussion
The authors developed a system that applies a vetted statistical method and then validated it against pre-existing papers. Their method confirmed the results but also produced interesting venues of future work that went unfound. Because of this, I found their results to be believable.

I held concerns about ART because such a useful method should be standard in the average statistical package. On the other hand, it may just be a matter of the method's having gone undiscovered in CHI.

Possible future work could include comparisons of the various statistical methods the authors discussed with ART on the same data set. This could help to validate it as a means of evaluating CHI results.

Book Reading #4: Obedience to Authority

Milgram's Obedience to Authority was a fascinating, if not disturbing, read. One of my biggest complaints about conference papers is that the page limit puts a harsh constraint on the extent of detail that authors can provide about their experiment. With an entire book to fill, Milgram more than filled the pages and probably could have gone on for much longer. Like Norman, he clearly had a lot to say; however, Milgram actually avoided repeating details wherever possible. That meant that each page had a new detail about the failings of humanity when authority gets involved.

I was greatly concerned that humanity as a whole is so easily manipulated by a man in a lab coat stating that the experiment must continue at all costs. I started the book knowing that I was going to be alarmed by the results, but that didn't make the crushing knowledge that Milgram initially expected results like I did—with people stopping sooner than they did—any less painful. The whole time I kept thinking to myself that I wouldn't be like the people who were obedient, but the more I read, the more I considered that my morality might just fail as theirs did.

I was surprised that no one at all had realized that the shocking experiment was a sham. Surely someone had enough electrical cognizance to think that the hand plate would shock the person holding down the actor too. There was even an electrician who took part in the experiment and was taken in by the ruse. I don't even know how the huge numbers of people—from all backgrounds too—couldn't figure out that they weren't actually going to hurt someone.

The women who deluded herself into thinking that she was more reluctant to continue the experiment was fascinating. I didn't realize just how much memories could be manipulated to make it seem as though we are more benign that we actually act when presented with authority. I suspect that was the case with many of the people involved with the experiment. We are largely good people and something that lets us believe that we acted better than we really did gets considered as a possibility.

I appreciated that Milgram thoroughly considered possible factors that could affect our willingness to comply with authority. The study on conflicting authorities was a relief to read, since it suggested that people make the right decision when presented with an apparent moral quandary. On the other hand, the fact that we readily follow in with other people bothered me. The nature tendency is to do what everyone else wants us to do, even if that is utterly wrong.

The discussion on the importance of obedience in the creation of society provided some explanation of why it is our nature to obey. While I do have to agree that obedience and hierarchy form a good society, these can be corrupted to serve any extremely detrimental cause. Perhaps we as a whole need to rebel against our natural tendencies whenever we feel that something is wrong, since internal morality so frequently tends to be in line with what it should be.

I had expected the book to be considerably more dull than it actually was. This is due at least in part to Milgram, though a large portion of it was because of the interesting nature of the material. Milgram's writing is almost understated; he allows the content to be the predominant focus, not flowery prose. His matter-of-a-fact tone makes it seem as though he means to discuss the most boring topic on earth, which simply makes the material that much more intriguing since he is talking about something that cannot be considered mundane in any sense.

Paper Reading #19: Reflexivity in Digital Anthropology

Reference Information
Reflexivity in Digital Anthropology
Jennifer A. Rode
Presented at CHI 2011, May 7-12, 2011, Vancouver, British Columbia, Canada

Author Bios
Jennifer A. Rode is an Assistant Professor at Drexel's School of Information and a fellow in Digital Anthropology at University College London.Her dissertation research involved an ethnography to examine gender and domestic end-user programming for computer security.

Summary
Hypothesis
What are the contributions of anthropological ethnographies to the study of HCI? How can reflexive ehthnographies contribute?

Methods
The author referred to a wide body of other works to define the various forms of ethnography, especially with respect to HCI. She also determined which factors of modern anthropological ethnographies are missing form HCI works. She then discussed the different ways of producing an ethnography and design together.

Results
The voice of ethnographers is critical to make a complete ethnography. Without it, the understanding of the reasons behind a certain design are untold. Discussing rapport, participant-observation, and use of theory all aid in producing a reflexive ethnography, which can be more useful than positivistic works in design. Iterative ethnographies feature grounded theories and increase the users' interest in a design. However, all forms of ethnography have their individual merits.

Contents
HCI is slow to adopt changing practices in anthropology, especially digital anthropology, which is the comparative ethnography of the effects of techonology on how humans experience life. While there is work in HCI that is in that area, it is not reflexive. Reflexivity is one of two anthropological approaches and embraces intervention as a means of gathering data. The social-technological gap can be studied through reflexive works. The other approach is positivism, which celebrates data above all else. Ethnographies are useful to aid in the design process.

There are three main forms of writing ethnographies: realist, confessional, and impressionistic. The realist form is the only form that is commonly accepted in HCI papers. It bears several common factors with positivism and focuses the need for experimental authority, typical forms, the native's point of view, and interpretive omnipotence. The first indicates that the researcher minimizes reactivity to observed events. Typical forms include data and precision. The native's point of view implies that the ethnography is precisely in line with the views of those being observed. Interpretive omnipotence does not allow the observed to participate in the writing and does not allow for uncertainty of the information. Confessional ethnographies reveal the author's biases to address the inherent subjectivity of ethnographical work. Impressionistic ethnographies create a narrative of the observed's daily life. Confessional and impressionist ethnographies are norammly only found as a portion of a realist work. The author emphasized that none of the presented styles are lacking in rigor.

There are several elements of modern anthropological works that tend to be absent from CHI: discussing rapport, participant-observation, and use of theory. Rapport enables access to vaild data and the discussion of it is an explanation of experimental method. It helps to explain data, but is frequently considered to be understood implicitly, which detracts from the quality of the ethnography because the unknown is not enumerated. Participant-observation is being a part of the participant's daily life. It tests hypotheses through experiences rather than external validation and is not discussed first-hand in realist works. Use of theory entails using previous theories as a basis for newer ones that are only formed after working with a participant.

HCI ethnographies are usually formative, summative, or iterative. Formative ones focus on current technology to improve or produce new ones and are the most common in HCI. Summative ones evaluate technology after its design is finished. Iterative ones produce a design in stages, with participants actively aiding in its design both directly and indirectly. It requires a particularly lengthy period of time to produce.

Discussion
The author sought to analyze the contributions of anthropological ethnographies to the study of HCI. Her paper studied several HCI papers with various qualities and cited prominent papers in ethnographical work. Her work was sufficiently explanatory to let me believe in the validity of her findings that the three forms of ethnography are valid for CHI papers.

With as much as Rode frowned upon positivistic papers, I was a little surprised to see her conclusion at first. On second reading, though, I realized that she was merely suggesting that other forms of ethnographies should have merit too, not that she was advocating reflexivity over positivism. The tone of the paper as a whole led me to that first conclusion, so perhaps it would have been worth it to more clearly explain her stance before the final page.

As far as future work, iterative ethnographies could be useful in the design process, but it might be worth looking into to see if the extra time expended in iterative design results that much more of a better product. Since the type of paper the author proposed is very rare, more research should be done to see if it is a viable alterative.

Book Reading #3: Emotional Design: Why We Love (Or Hate) Everyday Things

I feel there is no better way to describe Donald Norman's Emotional Design that to compare with Design of Everyday Things. It was, simply put, a radical departure from that previous work of his that we read. It seemed to me that the author had some sort of enlightenment, since he went from disapproving of things that probably “won an award” to celebrating that visual aesthetic is also an important factor in the likability of a device. Honestly, I wonder what could have caused such a radical change in viewpoint. I would expect him to have grown more disillusioned with the visceral as time passed, not less so. It could be argued that he had an enlightenment that brought him from relative ignorance (though he hardly believed himself ignorant in the past) to the realization that people like things for more than just their uses. After all, someone had to design the thing in the first place; who would design what they did not like in someone short of being forced to do so? The worship of image that he claimed was detrimental in Design was a perfectly valid reason to produce an object in Emotional. If I had been told when reading Design that Norman was going to transform his mentality to nearly its polar opposite, I would have recommended that the person who informed me should stop lying. It is, after all, quite rare to completely alter one's viewpoint rather than simply changing it slightly. Fifteen years is a long time, but I am not certain that it is sufficiently long of a period.

I liked how he divided the dominant draws of an object into the visceral, behavioral, and reflective factors. His previous book left me wondering why he was so adamantly against beautiful objects that are just that and nothing more. In this book, though, he revealed the hypocrisy of his works: he too likes useless objects. Some of them are more than just knick-knacks, like his impossible tea kettle; they occupy a none-too-ignorable amount of space that he could have filled with more functional objects, like those he praised in Design. Considering he was so concerned with the operability of everyday objects in Design, I have to wonder if he was at all thinking about things that he didn't technically manipulate on a daily basis, but simply observed. Perhaps he thought that curiosities were not everyday enough to be considered. Regardless, I was pleased to see that he extended his definition of everyday objects to include those that are simply observed, not manipulated.

If I had a choice to read either this book or Norman's previous book and this one, I definitely would have selected just this one. Norman apparently is particularly fond of reusing material extensively. When I saw some of the examples (and images!) in Design reappear here, I was a little astonished. (I am, however, grateful that he spared us the wish for a portable computer organizer in this book.) I would understand the reuse of content if he was refuting his previous ideas, but only did this with some of the concepts. Without any background in his prior works, I feel I would have understood his ideas thoroughly. Of course, for all I know, he might have written two more books extolling the visceral and reflective attributes of objects. What I do know is that I hardly needed a further explanation of the same, with the thoroughness with which Norman writes. Perhaps that is why I am so frustrated with Norman. I don't mind his writing style too much, besides the fact that he tends to overexplain a concept. With his previous book, the last chapter taught me all I needed to know. Similarly, this one taught me all that I needed to know about the other book. His writing was actually less dense and more accessible, but still just as informative.

Paper Reading #18: Biofeedback Game Design: Using Direct and Indirect Physiological Control to Enhance Game Interaction

Reference Information
Biofeedback Game Design: Using Direct and Indirect Physiological Control to Enhance Game Interaction
Lennart E. Nacke, Michael Kalyn, Calvin Lough, Regan L. Mandryk
Presented at CHI 2011, May 7-12, 2011, Vancouver, British Columbia, Canada

Author Bios

• Lennart E. Nacke is an Assistant Professor of HCI and Game Science at the University of Ontario Institute of Technology. As a postdoctoral researcher, he worked in the Interaction Lab of the University of Saskatchewan and studied affective computing.

• Michael Kalyn is an undergraduate student in the Interaction Lab of the University of Saskatchewan's Department of Computer Science. His focuses are in affective feedback and interfacing sensors.

• Calvin Lough is affiliated with the Interaction Lab of the University of Saskatchewan's Department of Computer Science. His research is in affective computing.

• Regan L. Mandryk is an Assistant Professor in the Interaction Lab of the University of Saskatchewan's Department of Computer Science. Her research focuses on affective computing and ubiquitous and mobile gaming.

Summary
Hypothesis
How do users respond to physiological sensors that work with game controllers? Which types of physiological sensors work for certain game tasks?

Methods
The authors developed a side-scrolling shooter that uses a traditional control as the normal form of input. Physiological sensors augment the controller through indirectly or directly controlled input. There were thus three versions of a game that participants played. Two game conditions used  direct sensors and four used indirect. The direct measures were respiration and EMG on the leg, and the indirect included GSR and EKG. Both physiological games had the eye gaze power-up, though the control lacked it. All participants played the three games, presented in random order, after playing a training level. The players completed questionnaires that asked about their experience after each game and again after completing all the levels. The players were not very experienced with side-scrolling shooters and mostly used Nintendo's Wii and DS as their sole form of novel input.

Results
Players found physiological controls to be more fun than just playing with the controller, with 90% preferring some sort of physiological control. The pleased users liked the increased level of involvement and variations. The participants agreed that the physiological control was novel, involving a little learning at first, but then feeling quite natural. Users preferred eye gaze the most, but only 1/20 of the votes were for indirect sensing. Overall, direct input was preferred to indirect in each category tested. The GSR and EKG sensors were difficult to use, EMG responses were split fairly evenly, and the respiration sensor was liked due to its ease of use. The temperature sensor was easy at first, but users found it tedious over time. Users preferred multiple forms of input. Direct controls were more real-time and suited for controlling the player icon, but indirect controls were slower to respond and thus were better to control the environment. Direct controls were also thought to increase the player's sense of accomplishment. Natural mappings were preferred. The players were comfortable with wearing sensors if they contribute to gameplay.

Contents
The authors developed a classification of direct and indirect sensor input to work with traditional game control. Current physiological game design paradigms revolve around indirectly controlled signals, like heart rate. Eye gaze and muscle flexion are directly controlled. Computer games allow for a low-risk way of testing physiological HCI. This sort of interaction is called affective gaming and relies on the player's emotional state. Replacing traditional controls with biofeedback has not worked well in the past. Adapting affective games use biofeedback to alter the technical parameters or user preferences based on controller movement or button pressure, for example. Indirect controls allow players to learn to control their brainwaves, though a previous study showed that people liked explicit biofeedback in first-person shooters. Various physiological measures include eye gaze, electromyography (EMG), galvanic skin response (GSR), electrocardiography (EKG), respiration sensors, and temperature sensors. The premise of biofeedback training is to turn indirect physiological measures into increasingly directly controlled ones. Indirect sensors are available to consumers, but direct ones are not available at the moment.

The authors' game used controller mappings common in Xbox 360 shooter games with physiological input controlled separately. The game features bosses and checkpoints, which can only be activated when a player kills all the enemies. The size of enemy targets' shadows increased based on physiological control and provided a larger hit box. The backup flamethrower's range is variable. Speed and jump height also varied, though these factors were tied together. The rate of snowfall during the final boss fight also changed. An eye gaze power-up was included, but only lasted for 20 seconds to reduce eye strain and maintain game balance. The sensors were integrated through a custom C# library.

Discussion
The authors tested user preference for biofeedback controls as part of a game and which types of controls are ideal for certain tasks. Their user test was small and focused on a single genre, but they created enough of a basic framework that a large body of future work will be able to rely upon.

I was initially concerned that the sensors that were applied to users would be cumbersome. However, the players were okay with most sensors so long as they contributed to gameplay and were not excessively taxing. That brought up an important limitation: the types of sensors used must not tire the user.

I hope that future work is performed with regard to additional genres. The next foreseeable step is in first-person shooters, which are closely related to their side-scrolling brethren. I would be very interested in seeing how well "God games" like Civilization could be played with biofeedback, since that seems like the least likely effective genre imaginable for the technology.

Paper Reading #17: Privacy Risks Emerging from the Adoption of Innocuous Wearable Sensors in the Mobile Environment

Reference Information
Privacy Risks Emerging from the Adoption of Innocuous Wearable Sensors in the Mobile Environment
Andrew Raij, Animikh Ghosh, Santosh Kumar, Mani Srivastava
Presented at CHI 2011, May 7-12, 2011, Vancouver, British Columbia, Canada

Author Bios

• Andrew Raij is a Post Doctoral Fellow in University of Memphis's Computer Science Department as a member of Dr. Kumar's lab. He is interested in persuasive interfaces.

• Animikh Ghosh is a junior research associate at SETLabs and was a research assistant to Dr. Kumar at the University of Memphis. He is interested in privacy risks from participatory sensing.

• Santosh Kumar is an Associate Professor in the University of Memphis's Computer Science Department. He leads the Wireless Sensors and Mobile Ad Hoc Networks Lab.

• Mani Srivastava is a professor in UCLA's Electrical Engineering Department. He worked at Bell Labs, which he considers to be the driving force behind his interest in mobile and wireless systems.

Summary
Hypothesis
How comfortable are individuals with the possibility that their data may be made public? How can we reduce the risk of sensitive data leakage?

Methods
The authors administered a survey to users who had their data stored, people who did not have data stored, and people with stored data who were informed of the extent of the stored data to determine their comfort levels with the possibility of data compromises. The people with stored data thus took the survey twice, after having participated in the cooperating study of AutoSense that stored their data. The participants were college students. The survey measured their level of concern about data disclosure both with and without data restrictions and abstractions. Participants were informed of the informed data through the Aha visualization system.

Results
Positively-associated activities, like exercise, were acceptable to share, as was location. The group with no data stake and the group who had not yet learned of the extent of the data collected about them had similar levels of unconcern for data storage. After learning about their data, the second group had higher concern ratings. Some participants referenced that they expect physiological states to remain private. Adding a temporal context increased concern, with increasing abstractions reducing the concern. Duration was less worrisome than a timestamp. Increasing the publicity of data sets concerned users, especially when the identity was included in the data. Participants seemed initially naive to the danger of shared data, with the exception of location. The differing concerns about certain activities suggested that privacy should be handled to different extents depending on the study.

Contents
Wearable sensors record sensitive physical and physiological data, to which machine learning algorithms can be applied. This algorithms reveal a wealth of private information about behavioral states and activities, including stress levels and addictions. These inferences can be shared without the user's permission, potentially revealing private data or identifying the individual. Data sets produced from tests of wearable sensors cannot be released for that reason. Most notably, seemingly innocuous data can be combined to produce informed inferences about a person. Sensor data is hard to anonymize because it is inherently sensitive and quasi-identifying.

The authors produced a framework that focuses on how to displace the boundary where privacy and publicity are in tension. It covers measurements, behaviors, contexts, restrictions, abstractions, and privacy threats. Behaviors and contexts derive from measurements. Contexts can be further subdivided into temporal, physical, physiological, and social contexts. Restrictions and abstractions safeguard data. The former removes data from the set, and the latter tries to reduce the extent of exposure in the set.

The authors developed the Aha visualization system to provide four visualizations of individual behavior, including daily life and stress.

Discussion
The authors wanted to find out how concerned users were about sensor data being used to determine things about them and how to prevent identifiable information from being released. Their survey was well-founded and their framework seems reasonable, so I am convinced that this paper is sound.

I was very interested to see just how much could be determined about a person through seemingly unrelated data points. It was actually extremely disturbing to think that so much information could be inferred through accelerometers and stress meters.

I would be very interested in seeing this survey being expanded to cover a variety of demographics. While I would think that college students would be the most knowledgeable about the extent of information they are revealing, I am curious to see what a child or senior citizen might think. Perhaps generation gaps would emerge--or everyone would be equally ignorant of the dangers. Either way, I would love to see those results.

Paper Reading #15: Madgets: Actuating Widgets on Interactive Tabletops

Reference Information
Madgets: Actuating Widgets on Interactive Tabletops
Malte Weiss, Florian Schwarz, Simon Jakubowski, Jan Borchers
Presented at UIST'10, October 3-6, 2010, New York, New York, USA

Author Bios

• Malte Weiss is a PhD student in the Media Computing Group at RWTH Aachen University. His work focuses on interactive surfaces and tangible user interfaces.

• Florian Schwarz is a Diploma Thesis student in the Media Computing Group at RWTH Aachen University. His work is in interactive tabletop computing.

• Simon Jakubowski is a student assistant in the Media Computing Group at RWTH Aachen University. He is working on two other projects besides this one.

• Jan Borchers is the head of the Media Computing Group at RWTH Aachen University. He holds a PhD from Darmstadt University of Technology in Computer Science and now explores HCI.

Summary
Hypothesis
How can we create tangible magnetic widgets for use on tabletop computers?

Methods
The authors created a prototype of the Madgets system and various sample controls for it, including radio buttons and slider knobs. They considered general purpose widgets, height, force feedback, water wheel Madgets (which transfer energy), and mechanical audio feedback as possibilities.

Results
General purpose widgets provide both haptic and visual feedback. The physical configuration can be saved for later use. Multiple users can also use the system since the system is actuated by electromagnets, which enables a tangible presence of another user. They also support ad-hoc usage.

Height is a possible feature, as the electromagnets can keep a Madget in place while lifting parts of it, as in a radio button. A user can feel both the shape of the button and its current state. A clutch control can lock or unlock moving parts to disable it, similar to current GUIs "graying out" and option.

Force feedback can be generated through resistance to the moving of a part of a Madget. The algorithm used allows for Madgets to vibrate and also create dynamic notches when a user reaches a certain step in a scale

The water wheels transfer energy from the table to the Madget. Inductive energy transfer, performed through plates, allows for power to be sent to a Madget without additional components. Motors work through a rotational actuation of a part and can be combined to produce a more complex system.

Mechanical audio feedback can occur through a magnetic pulse that triggers a noise of some sort.

Prototypes are quickly producible and do not take much time to program into the system with dynamic mappings.

Contents
The Madgets created by the authors are translucent, tangible widgets that resemble common controls like sliders and are relevant for general usage. They have magnets attached to them that can be actuated independently through an array of electromagnets. Actuated tangibles include moving the Madget across the surface, and force feedback. They are low-power and low-cost. The devices are unpowered and passive, which makes them easy to produce and hide the underlying technology from the user. Controls can be relabeled dynamically.

The sensing technique requires uniform backlighting, provided by an electroluminescent foil. An array of electromagnets, controlled by an Arduino through shields that provide various output channels to generate pulse width modification, actuates objects separately. To track the physical devices, a visual sensing technique, which does not interfere with the electromagnets, was used. Diffused Surface Illumination detects touch events but is also precise enough to differentiate Madgets from fingers. The controls are illuminated through the LCD, so that they are dynamically labeled. They are mounted on cylindrical markers that are used to determine where the Madgets are and encode the type of Madget. Moving parts also have a marker. Permanent magnets are attached for actuation. Each rigid body in a Madget can have different actuation forces applied to its magnets tangentially or normally. The forces are computed to move a permanent magnet a certain distance based on polarization and the position of the magnet. Linear optimization is used to resolve their computations. The Coin-or programming library minimizes the desired function, and thus the total force, power and heat production. The weights are applied dynamically to optimize performance and reliability with a frame rate of 30fps. Overheating electromagnets are considered weightier than others to reduce their usage. The widgets have gradient fiducials to increase the resolution of each sensing dot based on the radius of the object, detected by its brightness. The table is pre-calibrated for this with no objects, a white sheet, and then a dark gray sheet successively placed on it to determine the thresholds. The algorithm can identify multiple touches and dragging gestures. It first detects widget footprints and then focuses on remaining input.

Discussion
The authors created a system that features magnetic widgets as a new, cheap form of interaction. While their results certainly suggest that more work in this field is viable, the lack of user testing concerned me. I don't doubt that users could quickly adapt to using the devices as an extension of touchscreens, but the effect of automatic actuation was untested, which leaves me concerned about the effectiveness of the system.

The authors proposed that work should be done in avoiding Madget collisions, which struck me as a very good idea. While a good interface should mean that this situation would never occur, designs should always assume that some error will be made. If the system inherently tries to avoid crashing Madgets, then it is far less likely that such a problem will occur.

I was particularly intrigued by the sheer cheapness of producing one's own widgets, especially with a 3D printer available. The system was specifically built to allow the average designer to easily program any device into the system, which increases its accessibility that much more.

Paper Reading #14: TeslaTouch: Electrovibration for Touch Surfaces

Reference Information
TeslaTouch: Electrovibration for Touch Surfaces
Olivier Bau, Ivan Poupyrev, Ali Israr, Chris Harrison
Presented at UIST'10, October 3-6, 2010, New York, New York, USA

Author Bios

• Olivier Bau is a Post-Doctoral Research Scientist in Disney Research in the Interaction Design group. His PhD is in Computer Science and was received from INRIA Saclay.

• Ivan Poupyrev is a Senior Research Scientist in Disney Research in the Interaction Design group and studies interactive technologies and interface design.

• Ali Israr has a PhD in Mechanical Engineering from Purdue and now works in the Interaction Design @ Disney group in Disney Research. His interests are in disseminating haptic technologies in consumer-grade products.

• Chris Harrison is a PhD student at Carnegie Mellon's Human Computer Interaction Institute. He is also a Microsoft Research PhD Fellow and focuses his research on novel interaction techniques.

Summary
Hypothesis
How effective is electrovibration produced by contact with a touch screen as dynamic tactile feedback?

Methods
All tests grounded users.

The first test determined how users perceived the produced sensations. Four frequency and amplitude combinations were used and represented the limits of the range of each. The participants filled out questionnaires that asked them to describe the sensation, the texture of the sensation, and the dimensions of the sensation on a Likert scale.

The second test was used to determine the threshold of voltage for the detection of a sensation. The thresholds were estimated for five frequencies and narrowed down through a series of minor changes both up and down in voltage. After 12 reversals from increasing to decreasing voltage or vice versa, the test was concluded. The amount of change decreased after 3 reversals. Frequency and amplitude thresholds were determined similarly. The absolute detection threshold test presented participants with two screens, only one of which had a tactile screen, and challenged participants to pick the one with the electrode. The absolute discrimination test had another screen and allowed participants as much time as they needed.

Results
Grounding increased the intensity of the tactile sensation. For the first test, low frequencies felt rougher than higher frequencies. The effect of amplitude depended on the frequency. High frequency and increased amplitude increased the smoothness, while low frequency and decreased amplitude increased the stickiness. The participants tended to describe the sensations as combinations of vibration and friction.

The second test found detection threshold levels similar to force detection threshold levels found by another group, suggesting that the two are related. These results were placed onto a chart for use by future designers that show the relationship between frequency and voltage. The relationship between these is not linear. The frequency just-noticeable-differences (JND), which indicate when a user perceived a difference between two surfaces, resembled those produced by mechanical vibration. The amplitude JND was slightly lower than previously found results, but was constant.

Contents
A lack of haptic feedback inhibits user interaction, but most of the research in that field focuses on using electromechanical actuators through either the screen or a pen used to interface with the surface. Electrovibration uses electrostatic friction between the screen and a user's fingers. It is quick, dynamic, uses little power, is scalable, and can be added to current devices fairly easily. Electrovibration is based on Mallinckrodt's 1954 discovery and uses alternating voltage on the surface to produce an intermittent attraction force that modulates friction and creates the feeling of a rubbery surface. Unlike other tactile actuation systems, it does not send charges into the skin as the sensation is produced by a mechanical action and directly actuates the fingers. Electrovibration produces a flat response, is more subtle than traditional mechanical vibration, is entirely noiseless, and is reliable. TeslaTouch delivers its interaction through a clear electrode on the surface of the device.

In this paper, the system used a diffuse illumination multitouch tabletop device. The transparent electrode is excited with a periodic electrical signal that spans the whole surface and is spatially uniform. With sufficient amplitude, the attractive force increases the dynamic friction. Only fingers that are moving feel the sensation since it is through friction. The current is limited to 0.5mA, which is safe. The tabletop surface was built with several panels, tracked interaction through infrared cameras, and ran in real time. The developed technology was deliberately general and can be used in a variety of ways not tested in the paper. Specific sensations can be engineered for a variety of situations, including painting programs, non-visual information layers, conventional GUI widgets, direct manipulation, and rubbing interactions. Multitouch interactions are supported so long as only one finger is moving at a time.

Discussion
The authors created a system that allows for tactile sensations to be perceived through electrovibration. Their extensive user testing convinced by that their results are well-founded. More importantly, they also produced a chart that can be tested later for accuracy.

I could see this being used in a variety of touch devices, as it is less power-hungry than mechanical vibration. The fact that it is scalable makes me think that this is a viable technology on tabletop devices or on ubiquitous computing devices. This is not as useful with the shoe-based UI (though slightly less than impossible), but any surface that requires finger-based interaction might find this a simple and cost-effective way to incorporate tactile feedback.

I was concerned that the entire surface receives the same sensation of friction modulation. If I slightly move the wrong hand with multitouch input, I would not expect to feel the texture that my other hand is feeling. All the possible multitouch ideas presented here rely on the fact that one of the hands is not moving, which dramatically reduces the possibilities for the technology.

Book Reading #2: Design of Everyday Things

Donald Norman's Design of Everyday Things provided some of the most illustrative examples I have seen in a long time. The only downside is that the rest of the book was comparably fluff and certainly lighter. Part of the struggle I had with this book is that he shifts so frequently from his concepts to dwell on a multi-page example—having only finished a fragment of a page with the new concept. It didn't help that many of his references were vastly outdated, in particular the desire for a portable, handheld computer that could manage his calendar. That's what perplexed me the most with this book. I got the exact same experience (a sort of a blast from the past) reading the allegedly more current edition as I did with the 1989 edition. Honestly, if he really felt that such a dated book needed an update, he simply could have written a new one. As past students of CHI would note, it's not as though he is adverse to writing. I figured that if he left the book as it stood in 1989 and then wrote a sequel of sorts to it, similar to Levy's addendum to Hackers, we could still gain the perspective that 1980s Norman wanted to impart while building on new ideas that Norman realized. That said, considering Norman's dramatic reversal of his stance in his book on emotional design, a simple afterword would not suffice; the entire book would need to be rewritten (and not just by substituting LaserDiscs for DVDs).

I was astonished at how much Norman was willing to repeat himself. In the end, it boiled down to the fact that his main idea on behavioral design has only so many facets. That is, it is easy to see a design that fails (for proof, see just about every page in the book), but it is much harder to discover one that works and still somehow meets everyone's criteria. As a point of amusement, Norman claimed that an outline-based word processor generally increased repeated chunks of text. He may not have used such software, but it certainly appeared as much from his frequent loose rephrasings of previous chapters. Worse still, some of his definitions for terms repeated endlessly throughout the book (notably without a glossary) were only vaguely defined. For a man obsessed with usability, his book didn't seem to be as much.

His general attitude throughout the book was wholly patronizing, which I did not enjoy. Every time he claimed a non-functional design probably earned an award, I more and more saw Norman as a hipster of sorts, decrying the efforts of “try-hards.” He tends to ignore the fact that plenty of innovative, functional designs also have a certain elegance to them. That the designer wanted more than just a boring object is hardly a reason to shun it. Nonetheless, his disillusionment hangs heavily over the book, suggesting that awards as a whole are a sham. Certainly, some award-winning designs are completely unusable and some snubbed ones are simple to understand conceptually, but these categories are anything but mutually exclusive, which is something that the author insists must be the case. On the other hand, I cannot help but wonder if the world of the fairly recent past did indeed have Norman's claimed level of disjointedness. I would like to think that in the numerous years since this book was first published, the categories for awards also consider usability.

What really struck me the most was the insistence that programmers should not be user interface designers. As a diehard Linux user, being forced to use a system that does not function in the way that I understand it to makes no sense. Of course Linux distros have interface designers, but the key difference to me is that they tend to think like programmers. Terminal windows? Geared towards programmers. The GIMP? Makes no sense if you aren't a programmer. OpenOffice? Horrible if you only know Microsoft Office, wonderful otherwise (especially if you are a programmer). Maybe it takes a different way of thinking, reinforced through many, many late nights of coding, but I am willing to bet that a UI design who only has experience with the computer illiterate would struggle immensely to build a UI for a group of programmers. Sometimes it really does take a man on the inside, so to speak, to get the optimal result.