Crowdsourcing for Relevance Evaluation

author:Daniel E. Rose, A9.com Inc.
published: Nov. 19, 2008, recorded: October 2008, views: 109

You might be experiencing some problems with Your Video player.

Slides

Slides
0:00	Conference on Information and Knowledge Management
0:09	Crowdsourcing for Relevance Evaluation
0:21	What Is A9
1:07	What A9 Isn't
1:44	Product Search
3:47	Evaluating Relevance
4:29	Human Judgments, Standard Collection
4:49	Variant 1: Exhaustive Editorial Review
5:36	Variant 2: Pooling
6:35	Problems with Standard Collections
8:13	Variant 3: New Experiment, New Test
9:16	Alternative: Automated Metrics
10:29	Reconciling Two Approaches
11:08	Crowdsourcing
12:14	Amazon Mechanical Turk
13:25	MTurk: How it works
14:46	Some Sample HITs
15:08	Hypothetical Relevance Evaluation Task: World Facts
15:39	Qualifying Workers
16:39	Creating HITs (1)
17:12	Creating HITs (2)
17:43	Cost
18:11	Quality
19:57	Assessing TERC
22:46	Conclusions
23:51	Questions

Related content

Visitors who watched this lecture also watched...

39:34

Statistical Learning as the Ultimate Agile Development Tool
2513 views - Peter Norvig, 2008

05:21:05

Query Log Mining
266 views - Ricardo Baeza-Yates, Fabrizio Silvestri, 2009

01:09:02

Unsolved Problems in Search (and how we might approac them)
264 views - Bruce Croft, 2008

20:17

How does Clickthrough Data Reflect Retrieval Quality?
79 views - Thorsten Joachims, 2008

33:59

Social Event
88 views - James Shanahan, 2008

05:25:02

Log-linear Models and Conditional Random Fields
883 views - Charles Elkan, 2008

26:23

Toward Next Generation Search: Business, Product, Science, Infrastructure, and Talent
31 views - William Chang, 2008

21:41

Why E-Discovery is a CIKM-Hard Problem
27 views - David Andreoff Evans, 2008

46:32

Humane Data Mining
64 views - Rakesh Agrawal, 2008

23:11

Welcome and Opening Remarks
74 views - James Shanahan, 2008

Report a problem or upload files

If you have found a problem with this lecture or would like to send us extra material, articles, exercises, etc., please use our ticket system to describe your request and upload the data.
Enter your e-mail into the 'Cc' field, and we will keep you updated with your request's status.

Lecture popularity: You need to login to cast your vote.