Traditional image models enforce global smoothness, and more recently Markovian Field priors. Unfortunately global models are inadequate to represent the spatially varying nature of most images, which are much better modeled as piecewise smooth. This paper advocates the concept of local influence neighbourhoods (LINs). The influence neighbourhood of a pixel is defined as the set of neighbouring pixels which have a causal influence on it. LINs can therefore be used as a part of the prior model for Bayesian denoising, deblurring and restoration. Using LINs in prior models can be superior to pixel-based statistical models since they provide higher order information about the local image statistics. LINs are also useful as a tool for higher level tasks like image segmentation. We propose a fast graph cut based algorithm for obtaining optimal influence neighbourhoods, and show how to use them for local filtering operations. Then we present a new expectation-maximization algorithm to perform locally optimal Bayesian denoising. Our results compare favourably with existing denoising methods.