Retrieving information about the occurrences of persons in a video is an important task in many video indexing and retrieval applications. The problem is to answer the question "In which shots and scenes does person X appear?". In this paper, we present an automatic video annotation system with respect to a person?s appearance based on state-of-the-art algorithms for face detection, tracking and recognition. In contrast to many related approaches, knowledge about the persons in a given video is not assumed in advance. Adaboost is employed after an initial clustering of faces to select the best features describing a person?s face. These features are then used to train new classifiers based only on the faces extracted from the video under consideration. Several possibilities to train Adaboost and Support Vector Machine (ensemble) classifiers directly on a video are compared. Finally, experimental results demonstrate the effectiveness of correcting in-plane face rotation and of the employed self-supervised learning method.