In many vision problems, the appearances of the observed images, e.g. the human facial images, are often influenced by multiple underlying factors. In this paper, a kernel-based factorization framework is proposed to analyze a multifactor dataset. Specifically, we perform N-mode Singular Value Decomposition (N-mode SVD) in a higher dimensional feature space instead of the input space by using kernel approaches. Given an input sample, its specific underlying factors which may be all absent in the training set can be extracted and translated from one sample to another by using kernel-based ?translation?. Therefore our framework is suitable for tasks of new image synthesis and underlying factor recognition. We demonstrate the capabilities of our framework on ensembles of facial images subjected to different person identities, view-points and illuminations with high-quality synthetic faces and high face recognition accuracy.