It is well known that certain classes of Gaussian process arise naturally as solutions to stochastic differential equations, for example the Ornstein-Uhlenbeck process arises as the stationary solution of a simple linear stochastic differential equation. In this work we introduce some initial results on the approximation of the solution of general stochastic differential equations by Gaussian processes. We employ a variational framework, where we seek a Gaussian process approximation to the posterior distribution of the state of a system whose dynamics are governed by a stochastic differential equation. The application for this work is approximate inference within stochastic dynamic models, in particular models used in weather forecasting.