Learning in the traditional sense focuses on finding a point estimate for the parameters of its model. Bayesian approaches extend this to posterior distributions but are computationally intractable for Markov random fields and inference can easily get trapped in local modes. A third possibility is to define a dynamical system, herding, that mixes very efficiently over an attractor set and where each point on this set defines an "energy function" over some state space. The collection of all these energy minima represent a sample collection that shares certain moment statistics with the input data. I will briefly introduce this system and present very preliminary ideas on the following issues:

• Can we learn (hyper)parameters for herding so that it can be run with the data decoupled from it?
• Which herding systems should be considered equivalent and what are the implications for the use of fast weights in ML learning?
• Among all non-equivalent herding systems that reproduce the same moment constraints, can we learn the one that performs optimal in terms of generalization?
• Can we characterize the attractor set of herding and is its dynamics chaotic?
• How useful can herding be for producing deep representations?