We propose a new method for unsupervised face recognition from time-varying sequences of face images obtained in real-world environments. Two types of forces, attraction and repulsion, operate across the spatio-temporal facial manifolds, to autonomously organize the data without relying on any category-specific information provided in advance. Experiments with real-world data gathered over a period of several months and including both frontal and side-view faces were used to evaluate the method and encouraging results were obtained. The proposed method can be used in video surveillance systems or for content-based information retrieval.