en
0.25
0.5
0.75
1.25
1.5
1.75
2
Generalized Agreement Statistics over Fixed Group of Experts
Published on 2011-11-302589 Views
Generalizations of chance corrected statistics to measure interexpert agreement on class label assignments to the data instances have traditionally relied on the marginalization argument over a variab
Related categories
Presentation
Generalized Agreement Statistics over Fixed Set of Experts00:00
Background and Settings00:23
Many such scenarios00:54
An Example01:16
Another Example01:43
An example 02:04
Two Problems - 102:13
Two Problems - 202:25
General Agreement Statistic - 103:13
General Agreement Statistic - 203:30
Modus operandi04:40
Problem04:54
Back to the agreement statistic06:10
Traditional Approaches: Inter-expert Agreement - 106:37
Traditional Approaches: Inter-expert Agreement - 208:10
Traditional Approaches: Inter-expert Agreement - 308:38
Not applicable in fixed rater scenario09:13
Inter-rater agreement over fixed set of raters09:41
Approach - 109:48
Approach - 210:05
Approach - 310:27
Inter-rater Agreement: Fixed rater scenario10:33
Simulations on synthetic data - 110:41
Simulations on synthetic data - 211:20
Simulations on synthetic data - 311:31
Simulations on synthetic data - 411:43
Simulations on UCI data11:53
Inter-rater Agreement Conclusions: An upper bound on the variability12:00
Agreement of a classifier against a group: Two Traditional Approaches12:15
Marginalization Approach and Issues in Fixed experts setting - 112:56
Marginalization Approach and Issues in Fixed experts setting - 212:57
Marginalization Approach and Issues in Fixed experts setting - 312:58
Consensus Approach and Issues12:58
Consensus approach fails in assessing classifier performance13:26
Agreement against Fixed Experts’ group - 113:43
Agreement against Fixed Experts’ group - 213:54
Agreement against Fixed Experts’ group - 314:00
Agreement against Fixed Experts’ group - 414:12
Agreement against Fixed Experts’ group - 514:34
Agreement against silver standard - 114:58
Agreement against silver standard - 215:28
Agreement against silver standard - 315:48
Conclusion15:58
Importance of time travel17:13
Thank You17:22