In this paper we introduce an unsupervised scheme for the segmentation of motion layers in video sequences. The number of layers is automatically determined by the method. Our approach first extracts the motion models thanks to a RANSAC-based random sampling algorithm improved by the use of geodesic distance information. Then those models are assigned to pixels in the color image by minimizing an energy functional thanks to graph-cut. Our energy takes into account motion residuals, color distributions, geodesic distance as well as temporal consistency of the layers. Moreover, we define a smoothness term that enforces a patch-wise spatial coherency on areas where optical flow is reliable and a pixel-wise coherency on occluded areas. The method leads to promising results on the tested sequences.