Paper Reading #6: TurKit: Human Computation Algorithms on Mechanical Turk

Reference Information
TurKit: Human Computation Algorithms on Mechanical Turk
Greg Little, Lydia B. Chilton, Max Goldman, Robert C. Miller
Presented at UIST'10, October 3-6, 2010, New York, New York, USA

Author Bios

• Greg Little is a professor in MIT's Computer Science and Artificial Intelligence Lab. He hopes to build a computer that can house consciousness.
• Lydia B. Chilton is a graduate student at the University of Washington and interned for Microsoft Research Asia.
• Max Goldman is a graduate student at MIT's Computer Science and Artificial Intelligence Lab and User Interface Design Group.
• Robert C. Miller is an Associate Professor at MIT and leads their User Interface Design Group. He has a PhD from Carnegie Mellon.

Summary
Hypothesis
Does the TurKit toolkit provide an effective way of integrating iterative programming with human computation?

Methods
The authors tested various example applications in lab and evaluated performance from 20 scripts running nearly 30,000 tasks in a year. The applications included iterative writing, blurry text recognition, decision theory experimentation, psychophysics experimentation

Results
Iterative writing found most contributions added to the text. Blurry text recognition gradually improved with iterations, though initial bad guesses were detrimental. TurKit facilitated decision theory and psychophysics experimentation, though the tests were run with a version that made parallelization more difficult. The authors found that waiting for humans required time that was an order of magnitude greater than running most of the scripts, suggesting that recording executions would reduce time. The need for determinism was not clear in many cases though. TurKit does not scale well.

Contents
TurKit is a toolkit to facilitate human computation algorithms, in which certain functions are delegated to a human. It uses Amazon's Mechanical Turk (MTurk) platform, which relies on paid humans to collect and organize data. TurKit uses crash-and-rerun to allow programmers to write imperative programs that rely on MTurk. Crash-and-rerun allows for modification and re-execution of a script without rerunning the more costly side functions. TurKit's implementation of this stores the trace leading to the current state. Certain actions can be designated to only run once, saving time and resolving non-determinism. Crashing a script is similar to blocking.

An extension of JavaScript, TurKit Script, provides a wrapper for MTurk. It allows for parallel computing through forks. An online interface facilitates script design and execution. Three primary MTurk functions used are text entry, voting, and sorting.

Discussion
The authors wanted to know if TurKit provided a useful and efficient way to integrate human computation into iterative programming. Since the overhead incurred by the system was so dramatically less than the human time required, they achieved their goal. This intuitively makes sense, so I support the authors' claim.

This work is significant because it allows for human computation to be systematic and minimized, reducing time and financial costs. Notably, it stores costly state information in a database in a manner similar to a stack-trace of a debugger. This storage allows a programmer to modify the script for re-execution while still using the same data, which is incredibly useful.

Some of the language features seemed unintuitive, and the wrapping of the once keyword selectively made the language a bit of a guess-and-check type. I for one would need to constantly refer to the API to see what function used an implicit once.