Generalized Agreement Statistics over Fixed Group of Experts

produced by: Data & Web Mining Lab
author: Mohak Shah, Bosch Research and Technology Center North America
published: Nov. 30, 2011,   recorded: September 2011,   views: 2567


Related Open Educational Resources

Related content

Report a problem or upload files

If you have found a problem with this lecture or would like to send us extra material, articles, exercises, etc., please use our ticket system to describe your request and upload the data.
Enter your e-mail into the 'Cc' field, and we will keep you updated with your request's status.
Lecture popularity: You need to login to cast your vote.


Generalizations of chance corrected statistics to measure interexpert agreement on class label assignments to the data instances have traditionally relied on the marginalization argument over a variable group of experts. Further, this argument has also resulted in agreement measures to evaluate the class predictions by an isolated classifier against the (multiple) labels assigned by the group of experts. We show that these measures are not necessarily suitable for application in the more typical fixed experts' group scenario. We also propose novel, moremeaningful, less variable generalizations for quantifying both the inter-expert agreement over the fixed group and assessing a classifier's output against it in a multiexpert multi-class scenario by taking into account expert-specific biases and correlations.

See Also:

Download slides icon Download slides: ecmlpkdd2011_shah_statistics_01.pdf (873.1┬áKB)

Help icon Streaming Video Help

Link this page

Would you like to put a link to this lecture on your homepage?
Go ahead! Copy the HTML snippet !

Write your own review or comment:

make sure you have javascript enabled or clear this field: