Simultaneous Localization and Mapping (SLAM) and Structure from Motion (SFM) are important and closely related problems in robotics and vision. I will show how both SLAM and SFM instances can be posed in terms of a graphical model, a factor graph, and that inference in these graphs can be understood as variable elimination. The overarching theme of the talk will be to emphasize the advantages and intuition that come with seeing these problems in terms of graphical models. For example, common computational tricks, such as the Schur complement trick in SFM, are simple choices about the order in which to eliminate the graph. In addition, while the graphical model perspective is completely general, linearizing the non-linear factors and assuming Gaussian noise yields the familiar direct linear solvers such as Cholesky and QR factorization. Based on these insights, we have developed both batch and incremental algorithms defined on graphs in the SLAM/SFM domain. In addition to direct methods, we recently worked on efficient iterative methods that use subgraphs of these factor graphs as pre-conditioners in a conjugate gradient scheme. Finally, we are now looking into how optimal control can be seamlessly integrated with the estimation algorithms for use in autonomous vehicles.