This Article 
 Bibliographic References 
 Add to: 
Candid Covariance-Free Incremental Principal Component Analysis
August 2003 (vol. 25 no. 8)
pp. 1034-1040

Abstract—Appearance-based image analysis techniques require fast computation of principal components of high-dimensional image vectors. We introduce a fast incremental principal component analysis (IPCA) algorithm, called candid covariance-free IPCA (CCIPCA), used to compute the principal components of a sequence of samples incrementally without estimating the covariance matrix (so covariance-free). The new method is motivated by the concept of statistical efficiency (the estimate has the smallest variance given the observed data). To do this, it keeps the scale of observations and computes the mean of observations incrementally, which is an efficient estimate for some well-known distributions (e.g., Gaussian), although the highest possible efficiency is not guaranteed in our case because of unknown sample distribution. The method is for real-time applications and, thus, it does not allow iterations. It converges very fast for high-dimensional image vectors. Some links between IPCA and the development of the cerebral cortex are also discussed.

[1] I. Sirovich and M. Kirby, Low-Dimensional Procedure for the Characterization of Human Faces J. Optical Soc. Am. A, vol. 4, no. 3, pp. 519-524, Mar. 1987.
[2] M. Turk and A. Pentland, Eigenfaces for Recognition J. Cognitive Neuroscience, vol. 3, no. 1, pp. 71-86, 1991.
[3] H. Murase and S.K. Nayar, “Visual Learning and Recognition of 3-D Objects from Appearance,” Int'l J. Computer Vision, vol. 14, pp. 5-24, 1995.
[4] Y. Cui and J. Weng, Appearance-Base Hand Sign Recognition from Intensity Image Sequences Computer Vision and Image Understanding, vol. 78, pp. 157-176, 2000.
[5] S. Chen and J. Weng, State-Based SHOSLIF for Indoor Visual Navigation IEEE Trans. Neural Networks, vol. 11, no. 6, pp. 1300-1314, 2000.
[6] G.H. Golub and C.F. vanLoan, Matrix Computations. Baltimore, Md.: The Johns Hopkins Univ. Press, 1989.
[7] Proc. NSF/DARPA Workshop Development and Learning, J. Weng and I. Stockman, eds., Apr. 2000.
[8] J. Hertz, A. Krogh, and R.G. Palmer, Introduction to the Theory of Neural Computation. Addison-Wesley, 1991.
[9] E. Oja, Subspace Methods of Pattern Recognition. Letchworth, U.K.: Research Studies Press, 1983.
[10] E. Oja and J. Karhunen, On Stochastic Approximation of the Eigenvectors and Eigenvalues of the Expectation of a Random Matrix J. Math. Analysis and Application, vol. 106, pp. 69-84, 1985.
[11] T.D. Sanger, Optimal Unsupervised Learning in a Single-Layer Linear Feedforward Neural Network IEEE Trans. Neural Networks, vol. 2, pp. 459-473, 1989.
[12] Y. Zhang and J. Weng, Convergence Analysis of Complementary Candid Incremental Principal Component Analysis Technical Report MSU-CSE-01-23, Dept. of Computer Science and Eng., Michigan State Univ., East Lansing, Aug. 2001.
[13] M. Fisz, Probability Theory and Mathematical Statistics, third ed. John Wiley&Sons, 1963.
[14] J. Weng,T. S. Huang,, and N. Ahuja,Motion and Structure from Image Sequences, Springer Series on Information Sciences. Berlin: Springer-Verlag, 1993.
[15] N.L. Owsley, Adaptive Data Orthogonalization Proc. IEEE Int'l Conf. Acoustics, Speech, and Signal Processing, pp. 109-112, Apr. 1978.
[16] P.A. Thompson, An Adaptive Spectral Analysis Technique for Unbiased Frequency Estimation in the Presence of White Noise Proc. 13th Asilomar Conf. Circuits, Systems, and Computers, pp. 529-533, 1979.
[17] E. Kreyszig, Advanced Engineering Mathematics, John Wiley&Sons, New York, 1983.
[18] P.J. Phillips, H. Moon, P. Rauss, and S.A. Rizvi, "The FERET Evaluation Methodology for Face-Recognition Algorithms," Computer Vision and Pattern Recognition, pp. 137-143, 1997.
[19] W.H. Press, B.P. Flannery, S.A. Teukolsky, and W.T. Vetterling, Numerical Recipes in C, second ed. Cambridge Univ. Press, 1986.
[20] J. Weng, W.S. Hwang, Y. Zhang, C. Yang, and R. Smith, Developmental Humanoids: Humanoids that Develop Skills Automatically Proc. First IEEE-RAS Int'l Conf. Humanoid Robots, Sept. 2000.
[21] J. Rubner and K. Schulten, Development of Feature Detectors by Self-Organization Biological Cybernetics, vol. 62, pp. 193-199, 1990.
[22] Principles of Neural Science, third ed. E.R. Kandel, J.H. Schwartz, and T.M. Jessell, eds., Norwalk, Conn.: Appleton and Lange, 1991.

Index Terms:
Principal component analysis, incremental principal component analysis, stochastic gradient ascent (SGA), generalized hebbian algorithm (GHA), orthogonal complement.
Juyang Weng, Yilu Zhang, Wey-Shiuan Hwang, "Candid Covariance-Free Incremental Principal Component Analysis," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 25, no. 8, pp. 1034-1040, Aug. 2003, doi:10.1109/TPAMI.2003.1217609
Usage of this product signifies your acceptance of the Terms of Use.