This work presents a method for the detection, tracking and spatial matching of connected components in a 3D video stream. The video image information is combined with 3D sites in order to align pieces of surfaces that are seen in subsequent frames. This is a key step in 3D video analysis for enabling several applications such as compression, geometric integration and scene reconstruction, to name a few. Our approach is to detect salient features in both image and world spaces for further alignment of texture and geometry. We use a projector-camera system to obtain high quality texture and geometry at 30 fps. Successful experimental results corroborating our method are shown.