The Selective Labels Problem: Evaluating Algorithmic Predictions in the Presence of Unobservables

Evaluating whether machines improve on human performance is one of the central questions of machine learning. However, there are many domains where the data is {\em selectively labeled} in the sense t