Viewpoint variance is one of the inevitable problems in vision based sign language recognition. However, most researchers avoid this problem by assuming a special view, especially the front view. In the paper, we propose a verification method for viewpoint invariant sign language recognition. In general, there are two major variances between two video sequences of the same sign: performance variance and viewpoint variance. For small performance variance, DTW can help us eliminate it. When there is only viewpoint variance between two sequences, we can consider the two sequences as obtained synchronously by a stereo vision system. Thus, for the current input, we can judge whether the known template is the matched one by verifying whether the two sequences can be considered as obtained by a stereo vision system. Our experiments demonstrate the efficiency of the proposed method. Furthermore, such verification method can be easily extended to other recognition tasks.