We consider the problem of learning with instances defined over a space of sets of vectors. We derive a new positive definite kernel f(A, B) defined over pairs of matrices A; B based on the concept of principal angles between two linear subspaces. We show that the principal angles can be recovered using only inner-products between pairs of column vectors of the input matrices thereby allowing the original column vectors of A, B to be mapped onto arbitrarily high-dimensional feature spaces. We apply this technique to inference over image sequences applications of face recognition and irregular motion trajectory detection.