It is a challenging task to accurately model the performance of a face recognition system, and to predict its individual recognition results under various environments. This paper presents generic methods to model and predict the face recognition performance based on analysis of similarity measurement. We first introduce a concept of "perfect recognition", which only depends on the intrinsic structure of a recognition system. A metric extracted from perfect recognition similarity scores (PRSS) allows modeling the face recognition performance without empirical testing. This paper also presents an EM algorithm to predict the recognition rate of a query set. Furthermore, features are extracted from similarity scores to predict recognition results of individual queries. The presented methods can select algorithm parameters offline, predict recognition performance online, and adjust face alignment online for better recognition. Experimental results show that the performance of recognition systems can be greatly improved using presented methods.