en-es
en
0.25
0.5
0.75
1.25
1.5
1.75
2
Domain-Independent Quality Measures for Crowd Truth Disagreement
Published on 2013-11-282220 Views
Using crowdsourcing platforms such as CrowdFlower and Amazon Mechanical Turk for gathering human annotation data has become now a mainstream process. Such crowd involvement can reduce the time need
Related categories
Presentation
Domain-Independent Quality Measures for Crowd Truth00:00
Crowd Truth01:45
Background03:41
Crowd-Watson Adaptation to Newspapers Event Extraction06:06
Example - 106:58
Example - 206:59
Example - 307:57
Role-Filler Taxonomies08:55
How do we represent & measure disagreement in a way that it can be harnessed?10:16
Events semantics are hard10:33
Events have multiple dimensions - 110:43
Events have multiple dimensions - 211:40
Each dimension has different granularity 12:51
Why do people disagree? - 113:45
Why do people disagree? - 214:01
Disagreement Analytics14:56
Experimental Setting16:05
Annotation Example16:23
Event Type Disagrement16:54
Event Location Disagrement18:06
Event Time Disagrement18:12
Event Participant Disagrement18:15
Comparative Annotation Distribution - 118:19
Comparative Annotation Distribution - 219:23
Sentence Clarity19:34
Spam Detection19:54
What more ...20:09
Conclusions20:54