In the streaming video processing domain, the only way to meet strict performance and quality requirements and yet to provide the area- and power-wise optimal platform is to apply buffering of the pixel data. Hence, the importance of careful design of the memory subsystem of streaming video SoC is significant. This paper presents the design of a memory sub-system of the high-performance motion estimation processor. The motion estimator is a VLIW-based application specific instruction-set processor (ASIP). Through the reorganization of the pixels within the buffer and taking into account physical features of the instantiated memory banks, the buffer performance is improved by a factor of four compared to earlier designs. We further improve the buffer performance by a factor of two through algorithmic modification. The algorithmic modification does not impair the quality which we demonstrate through evaluation on a set of video sequences.