0.25
0.5
0.75
1.25
1.5
1.75
2
Let’s Agree to Disagree: Fixing Agreement Measures for Crowdsourcing
Published on Apr 08, 2019112 Views
Related categories
Chapter list
Let’s Agree to Disagree: Fixing Agreement Measures for Crowdsourcing00:00
but the real title should have been...00:00
The Elephant in the Room00:00
This Talk is Based on the Following Paper00:00
Setting00:06
Agreement Formalization00:26
Agreement Formalization: Items00:28
Agreement Formalization: Assessors00:30
Agreement Formalization: matrix of rating00:32
Agreement Formalization: This matrix is often very sparse in crowdsourcing00:43
There are Several Agreement Measures00:52
Current Agreement Measures Are Inadequate01:01
Problems: there is more variability of judgments in the centre of the scale w.r.t. scale boundaries.01:25
Problems: the concentration point can be different for different items01:53
Problems: additional information is often not considered (e.g., gold questions)02:09
Problems: different ideas of “agreement by chance” definition02:28
Real Problems with State-of-the-Art Measures: Krippendorff’s Alpha03:15
Our Measure03:49
Our Measure: random judgments04:33
Our Measure: agreement04:39
Our Measure: agreement around scale boundaries04:43
Our Measure: disagreement04:54
Our Measure: we should have a minimal number of parameters05:01
Our Measure: Part 705:06
Our Measure, Bayesian inference: Part 105:26
Our Measure, Bayesian inference: Part 205:35
Our Measure, Bayesian inference: Part 305:39
Our Measure, Bayesian inference: Part 405:44
Our Measure, Bayesian inference: Part 505:49
Our Measure, Bayesian inference: Part 605:52
Our Measure, Bayesian inference: Part 706:01
Interpretation06:17
Examples of Shapes: Part 106:48
Examples of Shapes: Part 206:56
Examples of Shapes: Part 307:07
Examples of Shapes: Part 407:12
Examples of Shapes: Part 507:25
Robustness07:34
Confidence Interval, Robustness07:48
Done / Ongoing / Future Developments08:38
Properties Summary09:36
Resources09:51