There exist almost as many superpixel segmentation algorithms as applications they can be used for. So far, the choice of the right superpixel algorithm for the task at hand is based on their ability to resemble human-made ground truth segmentations (besides runtime and availability). We investigate the equally important question of how stable the segmentations are under image changes as they appear in video data. Further we propose a new quality measure that evaluates how well the segmentation algorithms cover relevant image boundaries. Instead of relying on human-made annotations, that may be biased by semantic knowledge, we present a completely data-driven measure that inherently emphasizes the importance of image boundaries. Our evaluation is based on two recently published datasets coming with ground truth optical flow fields. We discuss how these ground optical truth fields can be used to evaluate segmentation algorithms and compare several existing superpixel algorithms.