Aggressive hardware-based and software-based prefetch algorithms for hiding memory access latencies were proposed to bridge the gap of the expanding speed disparity between processors and memory subsystems. As smaller L1 caches prevail in deep submicron processor designs in order to maintain short cache access cycles, cache pollution caused by ineffective prefetches is becoming a major challenge. When too aggressive prefetching are applied, ineffective prefetches not only can offset the benefits of beign prefetches due to pollution but also throttle bus bandwidth, leading to overall performance degradation. In this paper, a hardware based cache pollution filtering mechanism is proposed to differentiate good and bad prefetches dynamically using a history table. Two schemes - Per-Address (PA) based and Program Counter (PC) based - for triggering prefetches are proposed and evaluated. Our cache pollution filters work in tandem with both hardware and software prefetchers. As shown in the analysis of our simulated results, the cache pollution filters can significantly reduce the number of ineffective prefetches by over 90%, alleviating the excessive memory bandwidth induced by them. The IPC is improved by up to 9% as a result of reduced cache pollution and less competition for the limited number of cache ports.