Paper Reading #2: Hands-On Math: A page-based multi-touch and pen desktop for technical work and problem solving

Reference Information
Hands-On Math: A page-based multi-touch and pen desktop for technical work and problem solving
Robert Zeleznik, Andrew Bragdon, Ferdi Adeputra, Hsu-Sheng Ko
Presented at UIST'10 October 3-6, 2010, New York, New York, USA

Author Bios

• Robert Zeleznik is the Director of Research at Brown University and is the CTO of FluiditySoftware.com with a Masters degree from Brown.
• Andrew Bragdon is a PhD student of Computer Science at Brown and studies gestural techniques.
• Ferdi Adeputra is affiliated with Brown University and has work in code bubbles.
• Hsu-Sheng Ko is affiliated with Brown and has work in gesture selection.

Summary

Hypothesis
Can Computer Algebra Tools (CAS) with a multi-touch user interface cause students and others to learn and work more efficiently?

Methods
The participants were undergraduates who needed math for their classes. Users created and manipulated pages, performed a quick calculation, performed multi-step derivation, graphed an equation, used palm-detecting options, used the Web, and manipulated the contents of a page.

Results
The system was largely successful, though students noted it might work better on a portable system. Page manipulation was successful, though users needed instruction first. Paper folding, bimanual gestures and the rectangular lasso selection were not popular, unlike space insertion. Palm recognition required multiple instructions as reminders. Most were comfortable with both hands on the screen. Panning graph contents suggested that finger posture was difficult to perform at first, like the hidden menus. The ability to manipulate math was popular, with even more functionality desired. The scope of expressions being limited to a single page alleviated some fears. Gesture recognition was an occasional problem. Users frequently approached gestures through an inefficient physical strategy. Overall, a more mature version of the system appeared viable.

Contents
This technology seeks to combine paper for sequential computation and CAS for efficient answers to yield an intuitive interface. For this paper, the complexity of the math was limited to high school-level. This work differs from prior work in that it tries to generalize the application to simply allow for interaction and manipulation of mathematical equations. The system uses Microsoft Surface and an infrared lightpen. It uses the StarPad SDK to recognize handwriting and convert it to a typeface, create graphs, cause computations with extended notations, and provide manipulable web access.

The technology combines the space of whiteboards with the organization of paper. Pages are managed through swipes. The system ignores large contact points as well as non-dominant hand touches to hold the paper in place. A bar containing a live view of the user's workspace can be opened. Users can fold a section of the page similar to code folding. The system provides hidden menus that only appear when an object is moved and do not interfere with dragging. The pen is touch-activated and combines touch and pen tools. The features include lasso selection, inserting space, and clipboard pasting. The non-dominant palm and fingers can be used to provide convenient additional commands. Expressions are manipulated through context-specific pinching, dragging, and stretching, but a hidden menu can help to differentiate actions. Any manipulations appear on the following line.

Discussion
The authors sought to design a hybrid of CAS and paper that was intuitive and largely succeeded. While some of the anticipated features caused problems, as an investigative work of the promise of such a system, the system worked. I am convinced that the results found here are correct.

As someone who frequently makes algebraic errors in math, this technology seems to vastly reduce the chance for error while rewarding exploration of unusual techniques. While this is not the first work to correct that problem, this seems like the most probable and logical solution.

The authors mentioned that this system would be ideal for educational work, which I agree with. Children in particular would benefit from the innovative control scheme, which would allow them to learn both visually and physically.