Compressed sensing and matrix completion are techniques by which simplicity in data can be exploited for more efficient data acquisition. For instance, if a matrix is known to be (approximately) low rank then it can be recovered from few of its entries. The design and analysis of computationally efficient algorithms for these problems has been extensively studies over the last 8 years. In this talk we present a new algorithm that balances low per iteration complexity with fast asymptotic convergence. This algorithm has been shown to have faster recovery time than any other known algorithm in the area, both for small scale problems and massively parallel GPU implementations. The new algorithm adapts the classical nonlinear conjugate gradient algorithm and shows the efficacy of a linear algebra perspective to compressed sensing and matrix completion.