Paper Reading #4: Gestalt: Integrated Support for Implementation and Analysis in Machine Learning

Reference Information
Gestalt: Integrated Support for Implementation and Analysis in Machine Learning
Kayur Patel, Naomi Bancroft, Steven M. Drucker, James Fogarty, Andrew J. Ko, James A. Landay
Presented at UIST'10, October 3-6, 2010, New York, New York, USA

Author Bios

• Kayur Patel is a computer science PhD student at the University of Washington and is interested in machine learning.
• Naomi Bancroft was an undergraduate at the University of Washington and is currently working for Google.
• Steven M. Drucker is a principal research at Microsoft Research and is an affliate professor of the Unversity of Washington.
• James Fogarty is an Assistant Professor at the University of Washington and is a key member of the university's Human-Computer Interaction group.
• Andrew J. Ko is an Assistant Professor at the University of Washington and directs the USE research group at that university.

Summary
Hypothesis
Does the system presented in this paper aid in reducing the debugging time for a machine learning system?

Methods
The study compared bug-finding performance with a similar task in MATLAB. The participants, graduate computer science students, had some experience in machine learning. Participants had to write code to connect data in the MATLAB environment, though the remainder of the debugging task, involving sentiment analysis and gesture recognition, was identical. The measurements were the number of bugs found and the number of bugs fixed.

Results
Users found and fixed far more bugs with the Gestalt environment than in the MATLAB one. Some used the visualization scripting feature of Gestalt. Participants enjoyed the connectedness of Gestalt. Most time was spent analyzing for errors.

Contents
Gestalt is a general-purpose tool for the application of machine learning and supports the implementation of a classification pipeline, analysis of the data in the pipeline, and  transitions between implementation and analysis. Analysis currently requires extensive developer time, but it performed repeatedly through the learning process. Two possible applications are sentiment analysis and gesture recognition. Classification of data ensures the accuracy of models, but suffers the problems of verifiability and sparseness. Systems are often tested without the data of user and then tested against that user's data for verification. Developers have two high-level perspectives: implementation and analysis. As hiding steps in the pipeline prevents generality, for flexibility, Gestalt operates similarly to an IDE. Information is stored in a relational data table to eliminate the need for data conversion. Gestalt allows developers to write code to produce generalized visualizations.

The authors reference domain-specific tools as inspiration for many of the generalized features of Gestalt. Disconnected generalized systems influenced the interconnectedness of the program. MATLAB is a commonly used tool in machine learning due to its connected nature, but does not have data representations reflecting the diversity of machine learning.

Discussion
The authors attempted to create a system that was more useful for machine learning debugging than a popular solution. I am somewhat convinced that they succeeded, since the small sample size could suggest that these users were all flukes who preferred Gestalt. However, given that I think I would prefer Gestalt over MATLAB, their results hold some merit.

I find any innovation that makes the debugging process at least little easier to be worth looking into. This seemed to cut down on the difficulty of debugging machine learning problems more than the standard process.

As future work, I would be interested in seeing some of the generalized concepts seen here extended into general-purposed IDEs. Data visualization in particular could make finding any bug significantly easier, as it is a problem not just found in machine learning.