Issue No.08 - August (2009 vol.31)
pp: 1525-1531
Marc Sobel , Fox School of Business and Management, Philadelphia
Longin Jan Latecki , Temple University, Philadelphia
This paper addresses the problem of piecewise linear approximation of point sets without any constraints on the order of data points or the number of model components (line segments). We point out two problems with the maximum likelihood estimate (MLE) that present serious drawbacks in practical applications. One is that the parametric models obtained using a classical MLE framework are not guaranteed to be close to data points. It is typically impossible, in this classical framework, to detect whether a parametric model fits the data well or not. The second problem is related to accurately choosing the optimal number of model components. We first fit a nonparametric density to the data points and use it to define a neighborhood of the data. Observations inside this neighborhood are deemed informative; those outside the neighborhood are deemed uninformative for our purpose. This provides us with a means to recognize when models fail to properly fit the data. We then obtain maximum likelihood estimates by optimizing the Kullback-Leibler Divergence (KLD) between the nonparametric data density restricted to this neighborhood and a mixture of parametric models. We prove that, under the assumption of a reasonably large sample size, the inferred model components are close to their ground-truth model component counterparts. This holds independently of the initial number of assumed model components or their associated parameters. Moreover, in the proposed approach, we are able to estimate the number of significant model components without any additional computation.
Maximal likelihood estimate (MLE), expectation maximization (EM), Kullback-Leibler divergence (KLD), sparse EM, piecewise linear approximation.
Marc Sobel, Longin Jan Latecki, "Piecewise Linear Models with Guaranteed Closeness to the Data", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.31, no. 8, pp. 1525-1531, August 2009, doi:10.1109/TPAMI.2009.13
[1] M.J. Beal and Z. Ghahramani, “The Variational Bayesian EM Algorithm for Incomplete Data,” Bayesian Statistics, vol. 7, J.M. Bernardo, M.J. Bayarri, J.O.Berger, A.P. Dawid, D. Heckerman, A.F.M. Smith, and M. West, eds., Oxford Univ. Press, 2003.
[2] L. Breiman, W. Meisel, and E. Purcell, “Variable Kernel Estimates of Multivariate Densities,” Technometrics, vol. 19, no. 2, pp. 135-144, 1977.
[3] A. Dempster, N. Laird, and D. Rubin, “Maximum Likelihood from Incomplete Data via the EM Algorithm,” J. Royal Statistical Soc., Series B, vol. 39, no. 1, pp. 1-38, 1977.
[4] P. Green, “Reversible Jump Markov Chain Monte Carlo Computation and Bayesian Model Determination,” Biometrica, vol. 82, pp. 711-732, 1995.
[5] M.J.R. Healy and M. Wesmacott, “Missing Values in Experiments Analyzed on Automatic Computers,” Applied Statistics, vol. 5, pp. 203-206, 1956.
[6] R. Lakaemper and L.J. Latecki, “Decomposition of 3D Laser Range Data Using Planar Patches,” Proc. IEEE Int'l Conf. Robotics and Automation, 2006.
[7] L.J. Latecki, M. Sobel, and R. Lakaemper, “New EM Derived from Kullback-Leibler Divergence,” Proc. ACM SIGKDD Int'l Conf. Knowledge Discovery and Data Mining, 2006.
[8] D.O. Loftsgaarden and C.P. Quesenberry, “A Nonparametric Estimate of a Multivariate Density Function,” Annals Math. Statistics, vol. 36, pp. 1049-1051, 1965.
[9] R. Neal and G. Hinton, “A View of the EM Algorithm that Justifies Incremental, Sparse, and Other Variants,” Learning in Graphical Models, M.I.Jordan, ed., Kluwer, 1998.
[10] P.L. Rosin, “Techniques for Assessing Polygonal Approximations of Curves,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 6, pp. 659-666, June 1997.
[11] T.P. Ryan, Modern Regression Methods. Wiley, 1997.
[12] G.R. Terrell and D.W. Scott, “Variable Kernel Density Estimation,” The Annals of Statistics, vol. 20, no. 3, pp. 1236-1265, 1992.
[13] N. Ueda and Z. Ghahramani, “Bayesian Model Search for Mixture Models Based on Optimizing Variational Bounds,” Neural Networks, vol. 15, pp.1223-1241, 2002.
[14] N. Ueda, R. Nakano, Z. Ghahramani, and G.E. Hinton, “SMEM Algorithm for Mixture Models,” Neural Computation, vol. 12, no. 9, pp. 2109-2128, 2000.