The Non-negative Matrix Factorization technique (NMF) has been recently proposed for dimensionality reduction. NMF is capable to produce a region- or part-based representation of objects and images. This paper experimentally compares NMF to Principal Component Analysis (PCA) in the context of image patch classification. A first finding is that the two techniques are complementary and that their respective performance is correlated to the within class scatter. This paper also analyses different techniques to combine these complementary methods. In the first combination scheme the best technique for each class is chosen and the results are merged. The second combination scheme builds a hierarchy of classifiers where again for each classification task the best technique is chosen. Additionally, incorporation of the classification results of neighboring image patches further improves the overall results.