A large demand on memory bandwidth puts limitation on the performance improvement of an H.264/AVC decoder. In order to reduce the amount of frame memory access, this paper proposes a new memory architecture that uses a cache memory to store the data for motion compensation. Optimized for the data access pattern of motion compensation, two new cache organizations are proposed to reduce cache conflict misses. A split-index direct mapped cache maps a page of a frame memory into two consecutive cache lines by turns. Another cache organization, called a circular cache, does not fetch the entire line of a cache but reads only the necessary part of a cache line. A block pre-fetching scheme is proposed to prefetch a block of data which is likely to be used by the motion compensation of the next block. With the circular cache and block pre-fetching, a frame memory bandwidth requirement is reduced by an average of 72% which is 5% better than the previous work. In addition, the number of the non-sequential data accesses is reduced by an average of 68%.