The Community for Technology Leaders
2013 IEEE 13th International Conference on Data Mining (2006)
Hong Kong
Dec. 18, 2006 to Dec. 22, 2006
ISSN: 1550-4786
ISBN: 0-7695-2701-9
pp: 11-20
Shin Ando , Yokohama National University, Japan
Einoshin Suzuki , Kyushu University, Japan
Detection of rare and exceptional occurrences in large-scale databases have become an important practice in the field of knowledge discovery and information retrieval. Many databases include large amount of noise or irrelevant data, whose distribution often overlaps with the subsets of exceptional data containing useful knowledge. This paper addresses the problem of finding a small subset of "minority" data whose distribution overlaps with, but are exceptional to or inconsistent with that of the majority of the database. In such a case, conventional distance-based or density-based approaches in Outlier Detection are ineffective due to their dependence on the structure of the majority or the prerequisite of critical parameters. <p>We formalize the task as an estimation of a model of the minority subset which provides a simple description of the subset and yet maintains divergence from that of the majority. This estimation is formalized as a minimization problem using an information theoretic framework of Rate Distortion theory. We further introduce conditions of the majority to derive an objective function which factorizes the property of the minority and dependence to the structure of the majority. The proposed method shows improvements from conventional approaches in artificial data and a promising result in document retrieval problem.</p>
Shin Ando, Einoshin Suzuki, "An Information Theoretic Approach to Detection of Minority Subsets in Database", 2013 IEEE 13th International Conference on Data Mining, vol. 00, no. , pp. 11-20, 2006, doi:10.1109/ICDM.2006.19
86 ms
(Ver )