This paper presents a new method to recognize and restructure the human motions in space-time using the tracking information of human joints. A specific posture can be recognized in actual frame by finding the correspondence between actual frame and key frame. Within key frame prototype information has been extracted to represent defined posture (one step of action). By recognizing some person's postures in order, it is possible to map body locations from the key frames to actual frames in long video sequence and confirm the posture-sequence (action). The proposed method is tolerant to substantial deformation between image and prototype and recognizes qualitatively similar joints moving traces that compose human action. The proposed method is achieved by using a LKT feature tracker approach to track joints in key frames and actual frames. Then, the factorized sampling method is applied to replace one tracker with tracker-set to enhance the joints tracking accuracy. The experiments results demonstrate that accuracy and efficiency of recognizing non-rigid human actions.