Segmentation of videos into layers of foreground objects and background has many important applications, such as video compression, human computer interaction, and motion analysis. Most existing methods work on image pixels or color segmentations which are computation expensive. Some methods require extensive manual input, static cameras, and/or rigid scenes. In this paper we propose a fully automatic segmentation method based on sequential clustering of sparse image features. The sparseness makes the method computation efficient. We use both edge and corner features to capture the outline of the foreground objects. Sequential linear regression is applied to the movement sequences of image features in order to compute the motion parameters for foreground objects and background layers, and consider the temporal smoothness simultaneously. Foreground layer is then extracted by a pyramidal Markov Random Field (MRF) model taking into account the spatial smoothness constraint. Experimental results on videos taken by webcams are shown and discussed.