0.25
0.5
0.75
1.25
1.5
1.75
2
Statistical Quality Estimation for General Crowdsourcing Tasks
Published on Sep 27, 20133849 Views
One of the biggest challenges for requesters and platform providers of crowdsourcing is quality control, which is to expect high-quality results from crowd workers who are neither necessarily very cap
Related categories
Chapter list
Statistical Quality Estimation for General Crowdsourcing Tasks00:00
Motivating example - 100:09
Motivating example - 200:30
Motivating example - 300:50
Motivating example - 401:07
Our target: Quality control for crowdsourcing tasks with unstructured response formats01:22
Background: Tasks with unstructured response formats constitute the majority in crowdsourcing02:23
Our approach for quality estimation03:21
Our approach for quality estimation: Ask crowdsourcing workers to review the outputs03:33
Problem setting: Estimate the true quality of the outputs from their given grade labels - 104:18
Problem setting: Estimate the true quality of the outputs from their given grade labels - 204:55
Two-stage model: Considering the abilities of both the author and reviewer05:08
Model of creation stage: The generative process of the true output quality07:39
Model of review stage: The generative process of the true output quality - 108:13
Model of review stage: The generative process of the true output quality - 208:46
Summary of the two-stage model - 109:28
Summary of the two-stage model - 209:48
Summary of the two-stage model - 310:09
Datasets: Logo designing, image description, and language translation tasks10:31
Evaluation methodology: Correlations and nDCG@1 with ground truth12:08
Baselines: Averaging and ordinal label aggregation13:04
Result 1: Our method outperforms in estimating the qualities in two datasets14:09
Result 2: Our method is good at finding the best output in all datasets15:17
Summary: Two-stage model accurately estimates output qualities in general crowdsourcing tasks15:44