The major complexity source in H.264 is the rate distortion (R-D) optimization. This work proposes a method to reduce the complexity of R-D analysis, paying a little video degradation, by estimating the encoding modes that are more likely used by the encoder. Since the encoding modes depend mainly on the typology of the video and on the quantization parameter used to encode each block, the work proposes to group the modes into subsets that depend on the quantizer level and the characteristics of the video. In particular, the motion compensability of each frame modulates the intra modes, while texture difficulty addresses the inter modes. The R-D algorithm evaluates only the modes included into the resulting subset. The usage of subset modes reduces the complexity of the encoding process (40-70%) with respect to the original R-D optimized encoder, with a slight PSNR reduction (0-0,30 dB).