In this paper, we propose a novel approach to matching human actions using semantic correspondence between human bodies with an eye towards invariant analysis of activity. The correspondences are used to provide geometric constraints between multiple anatomical landmarks (e.g. hands, shoulders and feet) to match actions performed from different viewpoints and in different environments. The fact that the human body has certain anthropometric proportion allows innovative use of the machinery of epipolar geometry to provide constraints to accurately analyze actions performed by different people leading to some interesting results. Temporally invariant matching is performed, using non-linear time warping, to ensure that similar actions performed at different rates are accurately matched as well. Thus, the proposed algorithm guarantees that both temporal and view invariance is maintained in matching. We demonstrate the versatility of our algorithm in a number of challenging sequences and applications.