There are numerous papers which present methods to address web-search related challenges such as relevance and ranking, query processing, and classication. Unfortunately, many of these methods are ineective in a large-scale commer- cial setting, despite statistically signicant experimental results. To help bridge this gap between academic and commercial settings, this lecture examines the components of large-scale commercial search engines, then proposes ve classes of problems encountered by researchers in this area - biases; bad or dierent assumptions about statistics, users, queries or web contents; insucient or miss- ing data; inconsistencies related to evaluations and objectives; and policies or external factors, including resource limitations. Using real stories and personal experiences, the lecture illustrates examples of these problems, along with a few proposed approaches to deal with or reduce their consequences or eects. In addition to the classes of problems, there are several fundamental prop- erties of the web that are often not considered suciently when performing experiments or dening problems, resulting in unrealistic experiments or ob- jectives. Even within a search engine, overlooking key properties such as the non-stationarity of the users and the web, can result in ineective evaluations, and may even lead to failed subsystems. Fortunately, very simple approaches can often be highly eective. This lec- ture helps put context on how commercial search engines work, what problems they face, what eective solutions require, and how evaluations and problem denitions could be changed to more eectively predict success in a commercial setting - while still retaining interest of researchers.