We present a new window-based stereo matching algorithm which focuses on robust outlier rejection during aggregation. The main difficulty for window-based methods lies in determining the best window shape and size for each pixel. Working from the assumption that depth discontinuities occur at colour boundaries, we segment the reference image and consider all window pixels outside the image segment that contains the pixel under consideration as outliers and greatly reduce their weight in the aggregation process. We developed a variation on the recursive moving average implementation to keep processing times independent from window size. Together with a robust matching cost and the combination of the left and right disparity maps, this gives us a robust local algorithm that approximates the quality of global techniques without sacrificing the speed and simplicity of window-based aggregation.