Motion compensation is formulated as quantization of the true motion field and optimized within an overall rate-distortion framework. The desired accuracy in motion estimation is achieved by designing the codebook of the motion quantizer for the target rate. We propose an iterative algorithm for the codebook design which optimizes the compression performance of the coder by directly minimizing the overall rate-distortion cost. A multistage structure is imposed on the codebook to obtain a practical motion quantization scheme, which is optimized through a joint design of the stages. Simulation results on video sequences are presented which demonstrate that RD optimized motion vector quantization outperforms conventional motion compensation and can achieve significant gains of 1.3 dB in the PSNR of the reconstructed picture. Further, the multiresolution nature of multistage motion quantization makes it applicable for scalable video coding applications.