Complex systems (living organisms, the brain, society, the economy, etc.) seem to depend on a huge number of details which makes them nearly irreducible, so that they cannot be described in terms of a small number of variables. This poses fundamental difficulties for the modeling of such systems and the parametrization or calibration of any model that we may propose to describe them. Furthermore, this irreducibility also implies the existence of strong random correlations between a large number of the components of the system that are not necessarily close neighbours in a geometric sense, or not necessarily linked by strong, direct interactions. This makes the system sensitive to changes in the external control parameters, to boundary conditions, etc., and poses a serious challenge to computer simulations. These ideas are illustrated on some toy models: a spin glass, a random cellular automaton, and a game theoretical model.