The Community for Technology Leaders
Green Image
Issue No. 01 - January (2011 vol. 60)
ISSN: 0018-9340
pp: 120-134
Zhenyu Zhong , McAfee Inc., Alpharetta GA
Kang Li , The University of Georgia, Athens
ABSTRACT
Statistical-based Bayesian filters have become a popular and important defense against spam. However, despite their effectiveness, their greater processing overhead can prevent them from scaling well for enterprise level mail servers. For example, the dictionary lookups that are characteristic of this approach are limited by the memory access rate, therefore relatively insensitive to increases in CPU speed. We conduct a comprehensive study to address this scaling issue by proposing a series of acceleration techniques that speed up Bayesian filters based on approximate classifications. The core approximation technique uses hash-based lookup and lossy encoding. Lookup approximation is based on the popular Bloom filter data structure with an extension to support value retrieval. Lossy encoding is used to further compress the data structure. While these approximation methods introduce additional errors to a strict Bayesian approach, we show how the errors can be both minimized and biased toward a false negative classification. We demonstrate a 6{\times} speedup over two well-known spam filters (bogofilter and qsf) while achieving an identical false positive rate and similar false negative rate to the original filters.
INDEX TERMS
Computer systems organization, performance attributes, information systems applications, miscellaneous, spam, bloom filter, approximation.
CITATION
Zhenyu Zhong, Kang Li, "Speed Up Statistical Spam Filter by Approximation", IEEE Transactions on Computers, vol. 60, no. , pp. 120-134, January 2011, doi:10.1109/TC.2010.92
107 ms
(Ver 3.1 (10032016))