In this paper, we present a Smooth Pursuit Eye Movement (SPEM) coding scheme to control the encoder generated Variable Bit Rate (VBR) data over a Constant Bit Rate (CBR) channel for real-time streaming video applications. This proposed rate control integrates both frame and block layer rate control to regulate the bit rate using the combination of feed forward control, feed-backward control, and model-based approaches together with characteristics of human visual perception. As a result, our proposed SPEM rate control method encodes the video sequence with less frame skipping and with higher encoded video quality compared to TMN8, which is currently implemented in the UBC H263+codec.