en
en-es
0.25
0.5
0.75
1.25
1.5
1.75
2
Statistical Quality Estimation for General Crowdsourcing Tasks
Published on 2013-09-273860 Views
One of the biggest challenges for requesters and platform providers of crowdsourcing is quality control, which is to expect high-quality results from crowd workers who are neither necessarily very cap
Related categories
Presentation
Statistical Quality Estimation for General Crowdsourcing Tasks00:00
Motivating example - 100:09
Motivating example - 200:30
Motivating example - 300:50
Motivating example - 401:07
Our target: Quality control for crowdsourcing tasks with unstructured response formats01:22
Background: Tasks with unstructured response formats constitute the majority in crowdsourcing02:23
Our approach for quality estimation03:21
Our approach for quality estimation: Ask crowdsourcing workers to review the outputs03:33
Problem setting: Estimate the true quality of the outputs from their given grade labels - 104:18
Problem setting: Estimate the true quality of the outputs from their given grade labels - 204:55
Two-stage model: Considering the abilities of both the author and reviewer05:08
Model of creation stage: The generative process of the true output quality07:39
Model of review stage: The generative process of the true output quality - 108:13
Model of review stage: The generative process of the true output quality - 208:46
Summary of the two-stage model - 109:28
Summary of the two-stage model - 209:48
Summary of the two-stage model - 310:09
Datasets: Logo designing, image description, and language translation tasks10:31
Evaluation methodology: Correlations and nDCG@1 with ground truth12:08
Baselines: Averaging and ordinal label aggregation13:04
Result 1: Our method outperforms in estimating the qualities in two datasets14:09
Result 2: Our method is good at finding the best output in all datasets15:17
Summary: Two-stage model accurately estimates output qualities in general crowdsourcing tasks15:44