Issue No. 01 - January (1982 vol. 4)
John E. Shore , SENIOR MEMBER, IEEE, Information Technology Division, Naval Research Laboratory, Washington, DC 20375.
Robert M. Gray , FELLOW, IEEE, Department of Electrical Engineering, Stanford University, Stanford, CA 94305.
This paper considers the problem of classifying an input vector of measurements by a nearest neighbor rule applied to a fixed set of vectors. The fixed vectors are sometimes called characteristic feature vectors, codewords, cluster centers, models, reproductions, etc. The nearest neighbor rule considered uses a non-Euclidean information-theoretic distortion measure that is not a metric, but that nevertheless leads to a classification method that is optimal in a well-defined sense and is also computationally attractive. Furthermore, the distortion measure results in a simple method of computing cluster centroids. Our approach is based on the minimization of cross-entropy (also called discrimination information, directed divergence, K-L number), and can be viewed as a refinement of a general classification method due to Kullback. The refinement exploits special properties of cross-entropy that hold when the probability densities involved happen to be minimum cross-entropy densities. The approach is a generalization of a recently developed speech coding technique called speech coding by vector quantization.
J. E. Shore and R. M. Gray, "Minimum Cross-Entropy Pattern Classification and Cluster Analysis," in IEEE Transactions on Pattern Analysis & Machine Intelligence, vol. 4, no. , pp. 11-17, 1982.