In this paper, we present a novel object of interest (OOI) extraction scheme that can work robustly for digital measurable image (DMI) sequences collected by mobile mapping systems (MMS). The proposed method integrates tracking and segmentation in a unified framework. We incorporate a new object-shaped kernel with the scale invariant mean shift algorithm to track the OOI through the DMI sequence and thus keep the temporal consistency. The well-known GrabCut approach for static 2D image segmentation is generalized to the DMI sequence for OOI segmentation. Experimental results on real DMI sequence collected by VISAT<sup>TM</sup> MMS demonstrate that the proposed approach is robust to the challenges such as low frame rate, large inter-frame displacement of the OOI and background clutter.