In the scalable video coding extension of H.264 standard, the concept of inter-layer prediction is adopted to reduce the redundancy between the base layers and the enhancement layers. Thus, additional coding modes should be calculated during the macroblock mode decision procedure at the enhancement layers. It increases the computational complexity and may obstruct its practical applications. A fast mode decision scheme is developed in this paper to speed up the encoding of the enhancement layers. It makes use of reducing the number of the candidate modes and early termination the mode decision procedure adaptively. The experimental results show the proposed algorithm can achieve a reduction of encoding time by 30% to 70% with negligible bit-rate increases.