In video transmission systems, compressed video is delivered over channels that are not necessarily error free. Channel errors can lead to a mismatch in the encoder/decoder prediction loop which will propagate the errors to the succeeding frames. As a result, the quality of the received video at the decoder side may drop significantly. In this paper, we aim to reduce the introduced mismatch by modifying the reference frame. Our technique makes use of previously Intra coded blocks and a new leaky prediction structure in order to improve the robustness. This technique was applied on temporal and spatial scalability of scalable extension of H.264/AVC. Simulation results show the effectiveness of our scheme especially for medium and high motion sequences.