Traditional off-line learning methods are often not appropriate for applications in user modelling and user interfaces since to be useful the system must learn about the user or context during the process of interaction 'on the fly'. This immediately raises the fundamental problem of trading off exploration and exploitation in that as information is learnt the system may be tempted to act in line with this insight rather than further exploring alternatives. Machine learning has developed a number of models that attempt to capture and analyse this trade-off, from the simplest bandit problem to the full Markov decision processes underlying reinforcement learning. The workshop would include tutorials covering the bandit analysis as well as its relevance to user modelling. Reinforcement learning would also be included with particular emphasis on applications in user interfaces. It is also hoped to launch a challenge in this area. This workshop comes under the Thematic Programme 4: Online User Modelling and Reinforcement Learning and is a core meeting of the PASCAL Network.