We were able to achieve the high-quality results we were looking for with significant cost and time savings. The project finished on time and under budget.
Computer Scientist, Asset Control & Behavior at ARL
The U.S. Army Research Laboratory (ARL) develops technologies for use by the larger Army and Department of Defense (DoD) community.
ARL was developing machine-based algorithms to recognize and label human actions in a large database of videos.
In order to validate the effectiveness of their algorithms, ARL needed to compare data generated by their system with human assessments of the same videos. Using their internal staff to review videos would require five full-time engineers for twenty weeks, which was not only cost prohibitive to ARL, but an inefficient use of engineering resources.
ARL chose Amazon Mechanical Turk to get quick, cost-effective access to a large on-demand workforce. ARL designed a scalable process for managing quality using Known Answers (supported via the Review Policy functionality available in the Mechanical Turk API) to measure Worker performance.
In each HIT, ARL required Workers to view and label human actions in 20 ten-second videos. Two of the 20 videos in each HIT had Known Answers. When a Worker's answers matched both Known Answers correctly, ARL would trust the labels generated for the other 18 videos in the HIT. Alternatively, if a Worker's answers did not match both of the Known Answers, ARL would exclude the Worker's labels from their results.
Leveraging Mechanical Turk, ARL was able to generate accurate labels within their time and cost constraints. As a result of this successful outcome, ARL extended the program and expanded the volume of labels generated by Mechanical Turk Workers.
Learn more about Army Research Lab by visiting their website, www.arl.army.mil