Approval Voting and Incentives in Crowdsourcing

author: Nihar B. Shah, Department of Electrical Engineering and Computer Sciences, UC Berkeley
published: Sept. 27, 2015,   recorded: July 2015,   views: 3482

See Also:

Download slides icon Download slides: icml2015_shah_crowdsourcing_01.pdf (8.6 MB)

Help icon Streaming Video Help

Related Open Educational Resources

Related content

Report a problem or upload files

If you have found a problem with this lecture or would like to send us extra material, articles, exercises, etc., please use our ticket system to describe your request and upload the data.
Enter your e-mail into the 'Cc' field, and we will keep you updated with your request's status.
Lecture popularity: You need to login to cast your vote.


The growing need for labeled training data has made crowdsourcing an important part of machine learning. The quality of crowdsourced labels is, however, adversely affected by three factors: (1) the workers are not experts; (2) the incentives of the workers are not aligned with those of the requesters; and (3) the interface does not allow workers to convey their knowledge accurately, by forcing them to make a single choice among a set of options. In this paper, we address these issues by introducing approval voting to utilize the expertise of workers who have partial knowledge of the true answer, and coupling it with a (“strictly proper”) incentive-compatible compensation mechanism. We show rigorous theoretical guarantees of optimality of our mechanism together with a simple axiomatic characterization. We also conduct preliminary empirical studies on Amazon Mechanical Turk which validate our approach.

Link this page

Would you like to put a link to this lecture on your homepage?
Go ahead! Copy the HTML snippet !

Reviews and comments:

Comment1 Adrianna, November 9, 2015 at 6:07 a.m.:

Very very cool!

Comment2 santosh, November 9, 2015 at 7:16 a.m.:

i want to read more about this. where can i get more information?

Write your own review or comment:

make sure you have javascript enabled or clear this field: