A method is presented to estimate 3D head poses effectively by hybrid sensing of depth and gray information. Depth information is used to generate clean head segmentation even in a cluttered scene. Based on the segmentation result, sparse optical flow at head region is extracted and used for 3D head motion estimation in video rate. This method is shown to be effective through experiments on video sequences. Our method provides an alternative way for 3D head pose estimation from an image sequence in the current computer vision literature. Moreover, the depth information is incorporated into the estimation step as regularization for noisy motion estimation problem.