Subscribe
Issue No.01 - Jan. (2013 vol.25)
pp: 119-130
Thierry Denoeux , Université de Technologie de Compiègne, CNRS, Compiègne
ABSTRACT
We consider the problem of parameter estimation in statistical models in the case where data are uncertain and represented as belief functions. The proposed method is based on the maximization of a generalized likelihood criterion, which can be interpreted as a degree of agreement between the statistical model and the uncertain observations. We propose a variant of the EM algorithm that iteratively maximizes this criterion. As an illustration, the method is applied to uncertain data clustering using finite mixture models, in the cases of categorical and continuous attributes.
INDEX TERMS
Data models, Bayesian methods, Clustering algorithms, Uncertainty, Hidden Markov models, Probability distribution, Probability density function, mixture models, Uncertain data mining, Dempster-Shafer theory, Evidence theory, clustering, EM algorithm
CITATION
Thierry Denoeux, "Maximum Likelihood Estimation from Uncertain Data in the Belief Function Framework", IEEE Transactions on Knowledge & Data Engineering, vol.25, no. 1, pp. 119-130, Jan. 2013, doi:10.1109/TKDE.2011.201
REFERENCES
 [1] C.C. Aggarwal and P.S. Yu, "A Survey of Uncertain Data Algorithms and Applications," IEEE Trans. Knowledge and Data Eng., vol. 21, no. 5, pp. 609-623, May 2009. [2] C.C. Aggarwal, Managing and Mining Uncertain Data, series Advances in Data Base Systems, vol. 35. Springer, 2009. [3] R. Cheng, M. Chau, M. Garofalakis, and J.X. Yu, "Guest Editors' Introduction: Special Section on Mining Large Uncertain and Probabilistic Databases," IEEE Trans. Knowledge and Data Eng., vol. 22, no. 9, pp. 1201-1202, Sept. 2010. [4] M.A. Cheema, X. Lin, W. Wang, W. Zhang, and J. Pei, "Probabilistic Reverse Nearest Neighbor Queries on Uncertain Data," IEEE Trans. Knowledge and Data Eng., vol. 22, no. 4, pp. 550-564, Apr. 2010. [5] S. Tsang, B. Kao, K. Yip, W. Ho, and S. Lee, "Decision Trees for Uncertain Data," IEEE Trans. Knowledge and Data Eng., vol. 23, no. 1, pp. 64-78, Jan. 2011. [6] H.-P. Kriegel and M. Pfeifle, "Density-Based Clustering of Uncertain Data," Proc. 11th ACM SIGKDD Int'l Conf. Knowledge Discovery in Data Mining, pp. 672-677, 2005, [7] W.K. Ngai, B. Kao, C.K. Chui, R. Cheng, M. Chau, and K.Y. Yip, "Efficient Clustering of Uncertain Data," Proc. Sixth Int'l Conf. Data Mining (ICDM '06), pp. 436-445, 2006. [8] B. Kao, S.D. Lee, F. Lee, D. Cheung, and W.-S. Ho, "Clustering Uncertain Data Using Voronoi Diagrams and R-Tree Index," IEEE Trans. Knowledge and Data Eng., vol. 22, no. 9, pp. 1219-1233, Sept. 2010. [9] S. Günnemann, H. Kremer, and T. Seidl, "Subspace Clustering for Uncertain Data," Proc. SIAM Int'l Conf. Data Mining (SDM '10), pp. 385-396, 2010. [10] C.C. Aggarwal, "On Density Based Transforms for Uncertain Data Mining," Proc. IEEE 23rd Int'l Conf. Data Eng. (ICDE '07), pp. 866-875, 2007. [11] C.C. Aggarwal and P.S. Yu, "Outlier Detection with Uncertain Data," Proc. SIAM Int'l Conf. Data Mining (SDM '08), pp. 483-493, 2008. [12] J. Bi and T. Zhang, "Support Vector Classification with Input Data Uncertainty," Proc. Advances in Neural Information Processing Systems 17, L.K. Saul, Y. Weiss, and L. Bottou, eds., pp. 161-168, 2005. [13] L. Billard and E. Diday, Symbolic Data Analysis. Wiley, 2006. [14] L.A. Zadeh, "Fuzzy Sets as a Basis for a Theory of Possibility," Fuzzy Sets and Systems, vol. 1, pp. 3-28, 1978. [15] J. Gebhardt, M.A. Gil, and R. Kruse, "Fuzzy Set-Theoretic Methods in Statistics," Fuzzy Sets in Decision Analysis, Operations Research and Statistics, R. Slowinski, ed., pp. 311-347, Kluwer Academic Publishers, 1998. [16] P. Cazes, A. Chouakria, E. Diday, and Y. Schektman, "Extension de l'analyse en Composantes Principales à Des Données de type Intervalle," Revue de Statistique Appliquée, vol. 14, no. 3, pp. 5-24, 1997. [17] T. Denœux and M.-H. Masson, "Principal Component Analysis of Fuzzy Data Using Autoassociative Neural Networks," IEEE Trans. Fuzzy Systems, vol. 12, no. 3, pp. 336-349, June 2004. [18] P. Giordani and H.A.L. Kiers, "A Comparison of Three Methods for Principal Component Analysis of Fuzzy Interval Data," Computational Statistics and Data Analysis, vol. 51, no. 1, pp. 379-397, 2006. [19] K.C. Gowda and E. Diday, "Symbolic Clustering Using a New Similarity Measure," IEEE Trans. Systems, Man and Cybernetics, vol. 22, no. 2, pp. 368-378, Mar./Apr. 1992. [20] P. D'Urso and P. Giordani, "A Weighted Fuzzy C-Means Clustering Model for Fuzzy Data," Computational Statistics and Data Analysis, vol. 50, no. 6, pp. 1496-1523, 2006. [21] F.D.T.D. Carvalho and Y. Lechevallier, "Partitional Clustering Algorithms for Symbolic Interval Data Based on Single Adaptive Distances," Pattern Recognition, vol. 42, no. 7, pp. 1223-1236, 2009. [22] H. Tanaka, "Fuzzy Data Analysis by Possibilistic Linear Models," Fuzzy Sets and Systems, vol. 24, pp. 363-375, 1987. [23] R. Coppi, "Management of Uncertainty in Statistical Reasoning: The Case of Regression," Int'l J. Approximate Reasoning, vol. 47, no. 3, pp. 284-305, 2008. [24] M.B. Ferraro, R. Coppi, G.G. Rodríguez, and A. Colubi, "A Linear Regression Model for Imprecise Response," Int'l J. Approximate Reasoning, vol. 51, no. 7, pp. 759-770, 2010. [25] T. Denœux and M.-H. Masson, "Multidimensional Scaling of Interval-Valued Dissimilarity Data," Pattern Recognition Letters, vol. 21, pp. 83-92, 2000. [26] M.-H. Masson and T. Denœux, "Multidimensional Scaling of Fuzzy Dissimilarity Data," Fuzzy Sets and Systems, vol. 128, no. 3, pp. 339-352, 2002. [27] P.-A. Hébert, M.-H. Masson, and T. Denœux, "Fuzzy Multidimensional Scaling," Computational Statistics and Data Analysis, vol. 51, no. 1, pp. 335-359, 2006. [28] A.P. Dempster, "Upper and Lower Probabilities Induced by a Multivalued Mapping," Annals of Math. Statistics, vol. 38, pp. 325-339, 1967. [29] A.P. Dempster, "Upper and Lower Probabilities Generated by a Random Closed Interval," Annals of Math. Statistics, vol. 39, no. 3, pp. 957-966, 1968. [30] G. Shafer, A Mathematical Theory of Evidence. Princeton Univ. Press, 1976. [31] P. Smets, "The Combination of Evidence in the Transferable Belief Model," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 12, no. 5, pp. 447-458, May 1990. [32] P. Smets and R. Kennes, "The Transferable Belief Model," Artificial Intelligence, vol. 66, pp. 191-243, 1994. [33] T. Denœux, "A $k$ -Nearest Neighbor Classification Rule Based on Dempster-Shafer Theory," IEEE Trans. Systems, Man and Cybernetics, vol. 25, no. 5, pp. 804-813, May 1995. [34] T. Denœux and L.M. Zouhal, "Handling Possibilistic Labels in Pattern Classification Using Evidential Reasoning," Fuzzy Sets and Systems, vol. 122, no. 3, pp. 47-62, 2001. [35] T. Denœux and P. Smets, "Classification Using Belief Functions: the Relationship between the Case-Based and Model-Based Approaches," IEEE Trans. Systems, Man and Cybernetics B, vol. 36, no. 6, pp. 1395-1406, Dec. 2006. [36] S. Petit-Renaud and T. Denœux, "Nonparametric Regression Analysis of Uncertain and Imprecise Data Using Belief Functions," Int'l J. Approximate Reasoning, vol. 35, no. 1, pp. 1-28, 2004. [37] T. Denœux and M. Skarstein-Bjanger, "Induction of Decision Trees for Partially Classified Data," Proc. IEEE Int'l Conf. Systems, Man and Cybernetics (SMC '00), pp. 2923-2928, Oct. 2000. [38] Z. Elouedi, K. Mellouli, and P. Smets, "Belief Decision Trees: Theoretical Foundations," Int'l J. Approximate Reasoning, vol. 28, pp. 91-124, 2001. [39] S. Trabelsi, Z. Elouedi, and K. Mellouli, "Pruning Belief Decision tree Methods in Averaging and Conjunctive Approaches," Int'l J. Approximate Reasoning, vol. 46, no. 3, pp. 568-595, 2007. [40] S. Ben Hariz, Z. Elouedi, and K. Mellouli, "Clustering Approach Using Belief Function Theory," Proc. 12th Int'l Conf. Artificial Intelligence: Methodology, Systems, and Applications, J. Euzenat and J. Domingue, eds., pp. 162-171, 2006. [41] P. Vannoorenberghe and P. Smets, "Partially Supervised Learning by a Credal EM Approach," Proc. Eighth European Conf. Symbolic and Quantitative Approaches to Reasoning with Uncertainty (ECSQARU '05), L. Godo, ed., pp. 956-967, 2005. [42] I. Jraidi and Z. Elouedi, "Belief Classification Approach Based on Generalized Credal EM," Proc. Ninth European Conf. Symbolic and Quantitative Approaches to Reasoning with Uncertainty (ECSQARU '07), K. Mellouli, ed., pp. 524-535, Oct./Nov. 2007. [43] E. Côme, L. Oukhellou, T. Denœux, and P. Aknin, "Learning from Partially Supervised Data Using Mixture Models and Belief Functions," Pattern Recognition, vol. 42, no. 3, pp. 334-348, 2009. [44] A.P. Dempster, N.M. Laird, and D.B. Rubin, "Maximum Likelihood From Incomplete Data via the EM Algorithm," J. the Royal Statistical Soc., vol. B 39, pp. 1-38, 1977. [45] E. Ramasso, "Contribution of Belief Functions to Hidden Markov Models with an Application to Fault Diagnosis," Proc. IEEE Int'l Worshop Machine Learning for Signal Processing (MLSP '09), 2009. [46] T. Denœux, "Maximum Likelihood from Evidential Data: an Extension of the EM Algorithm," Proc. Int'l Conf. Soft Methods in Probability and Statistics (SMPS '10), C. Borgelt et al., ed., pp. 181-188, 2010. [47] D. Dubois and H. Prade, Possibility Theory: An Approach to Computerized Processing Uncertainty. Plenum Press, 1988. [48] P. Smets, "Belief Functions on Real Numbers," Int'l J. Approximate Reasoning, vol. 40, no. 3, pp. 181-223, 2005. [49] T. Denœux, "Extending Stochastic Ordering to Belief Functions on the Real Line," Information Sciences, vol. 179, pp. 1362-1376, 2009. [50] J.O. Smith, Spectral Audio Signal Processing, https://ccrma. stanford.edu/~jossasp/, Accessed Online Book, W3K Pub., Feb. 2011. [51] D. Dubois and H. Prade, "Focusing vs. Belief Revision: A Fundamental Distinction when Dealing with Generic Knowledge," Proc. Fifth Int'l Joint Conf. Qualitative and Quantitative Practical Reasoning, pp. 96-107, 1997. [52] H. Nguyen, An Introduction to Random Sets. Chapman and Hall/CRC Press, 2006. [53] M.E.G.V. Cattaneo and A. Wiencierz, "Regression with Imprecise Data: A Robust Approach," Proc. Seventh Int'l Symp. Imprecise Probabilities and Their Applications (ISIPTA '11), 2011. [54] M.A. Gil, M. López-Díaz, and D.A. Ralescu, "Overview on the Development of Fuzzy Random Variables," Fuzzy Sets and Systems, vol. 157, no. 19, pp. 2546-2557, 2006. [55] G. González-Rodríguez, A. Colubi, P. D'Urso, and M. Montenegro, "Multi-Sample Test-Based Clustering for Fuzzy Random Variables," Int'l J. Approximate Reasoning, vol. 50, no. 5, pp. 721-731, 2009. [56] G. Qi, W. Liu, and D.A. Bell, "Measuring Conflict and Agreement Between Two Prioritized Belief Bases," Proc. 19th Int'l Joint Conf. Artificial intelligence, pp. 552-557, 2005. [57] L.A. Zadeh, "Probability Measures of Fuzzy Events," J. Math. Analysis and Applications, vol. 10, pp. 421-427, 1968. [58] T. Denœux, "Maximum Likelihood Estimation from Fuzzy Data Using the Fuzzy EM Algorithm," Fuzzy Sets and Systems, vol. 183, no. 1, pp. 72-91, 2011. [59] B. Quost and T. Denœux, "Clustering Fuzzy Data Using the Fuzzy EM Algorithm," Proc. Fourth Int'l Conf. Scalable Uncertainty Management (SUM '10), A. Deshpande and A. Hunter, eds., pp. 333-346, Sept. 2010. [60] G.J. McLachlan and T. Krishnan, The EM Algorithm and Extensions. Wiley, 1997. [61] B.R. Cobb and P.P. Shenoy, "On the Plausibility Transformation Method for Translating Belief Function Models to Probability Models," Int'l J. Approximate Reasoning, vol. 41, no. 3, pp. 314-330, 2006. [62] G. Shafer, "Constructive Probability," Synthese, vol. 48, no. 1, pp. 1-60, 1981. [63] L.A. Goodman, "Exploratory Latent Structure Analysis Using Both Identifiable and Unidentifiable Models," Biometrika, vol. 61, no. 2, pp. 215-231, 1974. [64] G. Celeux and G. Govaert, "Clustering Criteria for Discrete Data and Latent Class Models," J. Classification, vol. 8, pp. 157-176, 1991. [65] L. Hubert and P. Arabie, "Comparing Partitions," J. Classification, vol. 2, no. 1, pp. 193-218, 1985. [66] J.D. Banfield and A.E. Raftery, "Model-Based Gaussian and Non-Gaussian Clustering," Biometrics, vol. 49, pp. 803-821, 1993. [67] G. Celeux and G. Govaert, "Gaussian Parsimonious Clustering Models," Pattern Recognition, vol. 28, no. 5, pp. 781-793, 1995.