CSDL Home IEEE Transactions on Pattern Analysis & Machine Intelligence 2013 vol.35 Issue No.06 - June

Subscribe

Issue No.06 - June (2013 vol.35)

pp: 1328-1342

Shuai Huang , Arizona State University, Tempe

Jing Li , Arizona State University, Tempe

Jieping Ye , Arizona State University, Tempe

Adam Fleisher , Banner Alzheimer's Institute, Phoenix

Kewei Chen , Banner Alzheimer's Institute, Phoenix

Teresa Wu , Arizona State University, Tempe

Eric Reiman , Banner Alzheimer's Institute, Phoenix

ABSTRACT

Structure learning of Bayesian Networks (BNs) is an important topic in machine learning. Driven by modern applications in genetics and brain sciences, accurate and efficient learning of large-scale BN structures from high-dimensional data becomes a challenging problem. To tackle this challenge, we propose a Sparse Bayesian Network (SBN) structure learning algorithm that employs a novel formulation involving one L1-norm penalty term to impose sparsity and another penalty term to ensure that the learned BN is a Directed Acyclic Graph (DAG)—a required property of BNs. Through both theoretical analysis and extensive experiments on 11 moderate and large benchmark networks with various sample sizes, we show that SBN leads to improved learning accuracy, scalability, and efficiency as compared with 10 existing popular BN learning algorithms. We apply SBN to a real-world application of brain connectivity modeling for Alzheimer's disease (AD) and reveal findings that could lead to advancements in AD research.

INDEX TERMS

Algorithm design and analysis, Bayesian methods, Input variables, Machine learning, Accuracy, Brain models,data mining, Bayesian network, machine learning

CITATION

Shuai Huang, Jing Li, Jieping Ye, Adam Fleisher, Kewei Chen, Teresa Wu, Eric Reiman, the Alzheimer's Disease Neuroimaging Initiative, "A Sparse Structure Learning Algorithm for Gaussian Bayesian Network Identification from High-Dimensional Data",

*IEEE Transactions on Pattern Analysis & Machine Intelligence*, vol.35, no. 6, pp. 1328-1342, June 2013, doi:10.1109/TPAMI.2012.129REFERENCES

- [1] N. Friedman, M. Linial, I. Nachman, and D. Péer, "Using Bayesian Networks to Analyze Expression Data,"
J. Computational Biology, vol. 7, pp. 601-620, 2000.- [2] A.S. Rodin and E. Boerwinkle, "Mining Genetic Epidemiology Data with Bayesian Networks I: Bayesian Networks and Example Application (Plasma apoE Levels),"
Bioinformatics, vol. 21, no. 15, pp. 3273-3278, 2005.- [3] B.G. Marcot, R.S. Holthausen, M.G. Raphael, M. Rowland, and M. Wisdom, "Using Bayesian Belief Networks to Evaluate Fish and Wildlife Population Viability under Land Management Alternatives from an Environmental Impact Statement,"
Forest Ecology and Management, vol. 153, nos. 1-3, pp. 29-42, 2001.- [4] M.E. Borsuk, C.A. Stow, and K.H. Reckhow, "A Bayesian Network of Eutrophication Models for Synthesis, Prediction, and Uncertainty Analysis,"
Ecological Modelling, vol. 173, pp. 219-239, 2004.- [5] H. Dai, K.B. Korb, C.S. Wallace, and X. Wu, "A Study of Casual Discovery with Weak Links and Small Samples,"
Proc. 15th Int'l Joint Conf. Artificial Intelligence, pp. 1304-1309, 1997.- [6] S. Mani and G.F. Cooper, "A Study in Casual Discovery from Population-Based Infant Birth and Death Records,"
Proc. AMIA Ann. Fall Symp., pp. 315-319, 1999.- [7] J.C. Rajapakse and J. Zhou, "Learning Effective Brain Connectivity with Dynamic Bayesian Networks,"
NeuroImage, vol. 37, pp. 749-760, 2007.- [8] J.N. Li, Z.J. Wang, S.J. Palmer, and M.J. McKeown, "Dynamic Bayesian Network Modeling of fMRI: A Comparison of Group-Analysis Methods,"
NeuroImage, vol. 37, pp. 749-760, 2008.- [9] J. Li and J. Shi, "Knowledge Discovery from Observational Data for Process Control through Causal Bayesian Networks,"
IIE Trans., vol. 39, no. 6, pp. 681-690, 2007.- [10] L. De Campos, "Independency Relationships and Learning Algorithms for Singly Connected Networks,"
J. Experimental and Theoretical Artificial Intelligence, vol. 10, pp. 511-549, 1998.- [11] L. De Campos and J. Huete, "A New Approach for Learning Belief Networks Using Independence Criteria,"
Int'l J. Approximate Reasoning, vol. 24, pp. 11-37, 2000.- [12] J. Pearl and T. Verma, "Equivalence and Synthesis of Causal Models,"
Proc. Sixth Conf. Uncertainty in Artificial Intelligence, 1990.- [13] P. Spirtes, C. Glymour, and R. Scheines,
Causation, Prediction and Search. Springer, 1993.- [14] C. Meek, "Causal Inference and Causal Explanation with Background Knowledge,"
Proc. 11th Conf. Uncertainty in Artificial Intelligence, 1995.- [15] G. Cooper and E. Herskovits, "A Bayesian Method for the Induction of Probabilistic Networks from Data,"
Machine Learning, vol. 9, pp. 309-347, 1992.- [16] D. Heckerman, D. Geiger, and D. Chickering, "Learning Bayesian Networks: The Combination of Knowledge and Statistical Data,"
Machine Learning, vol. 20, pp. 197-243, 1995.- [17] W. Buntine, "A Guide to the Literature on Learning Probabilistic Networks from Data,"
IEEE Trans. Knowledge and Data Eng., vol. 8, no. 2, pp. 195-210, Apr. 1996.- [18] N. Friedman and D. Koller, "Being Bayesian about Network Structure: A Bayesian Approach to Structure Discovery in Bayesian Networks,"
Machine Learning, vol. 50, pp. 95-125, 2003.- [19] D. Heckerman, "A Tutorial on Learning Bayesian Networks," Technical Report MSR-TR-95-06, Microsoft Research, 1996.
- [20] W. Lam and F. Bacchus, "Learning Bayesian Belief Networks, an Approach Based on the MDL Principle,"
Computational Intelligence, vol. 10, pp. 269-293, 1994.- [21] J.A. Suzuki, "Construction of Bayesian Networks from Databases Based on an MDL Principle,"
Proc. Ninth Conf. Uncertainty in Artificial Intelligence, pp. 266-273, 1993.- [22] R. Bouckaert, "Belief Networks Construction Using the Minimum Description Length Principle,"
Symbolic and Quantitative Approaches to Reasoning and Uncertainty, pp. 41-48, Springer, 1993.- [23] N. Friedman and M. Goldszmidt, "Learning Bayesian Networks with Local Structure,"
Proc. 12th Conf. Uncertainty in Artificial Intelligence, 1996.- [24] C. Chow and C. Liu, "Approximating Discrete Probability Distributions with Dependence Trees,"
IEEE Trans. Information Theory, vol. 14, no. 3, pp. 462-467, May 1968.- [25] D. Chickering, "Optimal Structure Identification with Greedy Search,"
J. Machine Learning Research, vol. 3, pp. 507-554, 2002.- [26] S. Acid and J. De Campos, "Searching for Bayesian Network Structures in the Space of Restricted Acyclic Partially Directed Graphs,"
J. Artificial Intelligence Research, vol. 18, pp. 445-490, 2003.- [27] R. Castelo and T. Kocka, "On Inclusion-Driven Learning of Bayesian Networks,"
J. Machine Learning Research, vol. 4, pp. 527-574, 2003.- [28] R. Larranaga, C. Kuijpers, R. Murga, and Y. Yurramendi, "Learning Bayesian Network Structures by Searching for the Best Ordering with Genetic Algorithms,"
IEEE Trans. Systems, Man, and Cybernetics, vol. 26, no. 4, pp. 487-493, July 1996.- [29] P. Larranaga, M. Poza, Y. Yurramendi, R. Murga, and C. Kuijpers, "Structure Learning of Bayesian Networks by Genetic Algorithms: A Performance Analysis of Control Parameters,"
IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 18, no. 9, pp. 912-926, Sept. 1996.- [30] D. Chickering, D. Geiger, and D. Heckerman, "Learning Bayesian Networks: Search Methods and Experimental Results,"
Proc. Preliminary Papers Fifth Int'l Workshop Artificial Intelligence and Statistics, 1995.- [31] X.W. Chen, G. Anantha, and X.T. Lin, "Improving Bayesian Network Structure Learning with Mutual Information-Based Node Ordering in the K2 Algorithm,"
IEEE Trans. Knowledge and Data Eng., vol. 20, no. 5, pp. 628-640, May 2008.- [32] P.O. Hoyer, D. Janzing, J.M. Mooij, J. Peters, and B. Scholkopf, "Nonlinear Causal Discovery with Additive Noise Models,"
Proc. Conf. Neural Information Processing Systems, 2009.- [33] J. Peng, P. Wang, N. Zhou, and J. Zhu, "Partial Correlation Estimation by Joint Sparse Regression Models,"
J. Am. Statistical Assoc., vol. 104, pp. 735-746, 2009.- [34] O. Sporns, D.R. Chialvo, M. Kaiser, and C.C. Hilgetag, "Organization, Development and Function of Complex Brain Networks,"
Trends in Cognitive Sciences, vol. 8, pp. 418-425, 2004.- [35] N. Friedman, I. Nachman, and D. Péer, "Learning Bayesian Network Structure from Massive Datasets: The 'Sparse Candidate' Algorithm,"
Proc. 15th Conf. Uncertainty in Artificial Intelligence, 1999.- [36] M. Schmidt, A. Niculescu-Mizil, and K. Murphy, "Learning Graphical Model Structures using L1-Regularization Paths,"
Proc. 22nd Nat'l Conf. Artificial Intelligence, 2007.- [37] R. Tibshirani, "Regression Shrinkage and Selection via the Lasso,"
J. Royal Statistical Soc. Series B, vol. 58, no. 1, pp. 267-288, 1996.- [38] I. Tsamardinos, L.E. Brown, and C.F. Aliferis, "The Max-Min Hill-Climbing Bayesian Network Structure Learning Algorithm,"
Machine Learning, vol. 65, no. 1, pp. 31-78, 2006.- [39] D. Margaritis and S. Thrun, "Bayesian Network Induction via Local Neighborhoods,"
Proc. Conf. Advances in Neural Information Processing Systems, 1999.- [40] J.P. Pellet and A. Elisseeff, "Using Markov Blankets for Causal Structure Learning,"
J. Machine Learning Research, vol. 9, pp. 1295-1342, 2008.- [41] E. Estrada and H. Naomichi, "Communicability in Complex Networks,"
Physics Rev. E, vol. 77, p. 036111, 2008.- [42] R. Luus and R. Wyrwicz, "Use of Penalty Functions in Direct Search Optimization."
Hungarian J. Industrial Chemistry , vol. 24, pp. 273-278, 1996.- [43] D.P. Bertsekas,
Nonlinear Programming, second ed. Athena Scientific, 1999.- [44] W. Fu, "Penalized Regressions: The Bridge vs the Lasso,"
J. Computational and Graphical Statistics, vol. 7, no. 3, pp. 397-416, 1998.- [45] T.H. Cormen, C.E. Leiserson, R.L. Rivest, and C. Stein,
Introduction to Algorithms, third ed. MIT Press, 2001.- [46] J. Friedman, T. Hastie, H. Hofling, and R. Tibshirani, "Pathwise Coordinate Optimization,"
The Annals of Applied Statistics, vol. 1, no.2, pp.302-332, 2007.- [47] C.F. Aliferis, I. Tsamardinos, and A. Statnikov, "HITON, a Novel Markov Blanket Algorithm for Optimal Variable Selection,"
Proc. AMIA Ann. Symp., 2003.- [48] I. Tsamardinos and C. Aliferis, "Towards Principled Feature Selection: Relevancy, Filters and Wrappers,"
Proc. Ninth Int'l Workshop Artificial Intelligence and Statistics, 2003.- [49] Bayesian Network Repository: http://www.cs.huji.ac.il/labs/compbioRepository , 2011.
- [50] I. Tsamardinos, A. Statnikov, L.E. Brown, and C.F. Aliferis, "Generating Realistic Large Bayesian Networks by Tiling,"
Proc. 19th Int'l FLAIRS Conf., 2006.- [51] D. Mackey,
Information Theory, Inference, and Learning Algorithms. Cambridge Univ. Press, 2003.- [52] N. Tzourio-Mazoyer, "Automated Anatomical Labelling of Activations in SPM Using a Macroscopic Anatomical Parcellation of the MNI MRI Single Subject Brain,"
NeuroImage, vol. 15, pp. 273-289, 2002.- [53] K. Supekar, V. Menon, D. Rubin, M. Musen, and M.D. Greicius, "Network Analysis of Intrinsic Functional Brain Connectivity in Alzheimer's Disease,"
PLoS Computational Biology, vol. 4, no. 6, pp. 1-11, 2008.- [54] N.P. Azari, S.I. Rapoport, C.L. Grady, M.B. Schapiro, J.A. Salerno, and A. Gonzales-Aviles, "Patterns of Interregional Correlations of Cerebral Glucose Metabolic Rates in Patients with Dementia of the Alzheimer Type,"
Neurodegeneration, vol. 1, pp. 101-111, 1992.- [55] K. Wang, M. Liang, L. Wang, L. Tian, X. Zhang, and T. Jiang, "Altered Functional Connectivity in Early Alzheimer's Disease: A Resting-State fMRI Study,"
Human Brain Mapping, vol. 28, pp. 967-978, 2007.- [56] R.L. Gould, B. Arroyo, R.G. Brown, A.M. Owen, and R.J. Howard, "Brain Mechanisms of Successful Compensation during Learning in Alzheimer Disease,"
Neurology, vol. 67, pp. 1011-1017, 2006.- [57] Y. Stern, "Cognitive Reserve and Alzheimer Disease,"
Alzheimer Disease Associated Disorder, vol. 20, pp. 69-74, 2006.- [58] K.B. Korb and A.E. Nicholson,
Bayesian Artificial Intelligence. Chapman & Hall/CRC, 2003.- [59] K.J. Friston, "Functional and Effective Connectivity in Neuroimaging: A Synthesis,"
Human Brain Mapping, vol. 2, pp. 56-78, 1994.- [60] M.D. Greicius, G. Srivastava, A.L. Reiss, and V. Menon, "Default-Mode Network Activity Distinguishes AD from Healthy Aging: Evidence from Functional MRI,"
Proc. Nat'l Academy Sciences USA, vol. 101, pp. 4637-4642, 2004.- [61] G.E. Alexander, K. Chen, P. Pietrini, S.I. Rapoport, and E.M. Reiman, "Longitudinal PET Evaluation of Cerebral Metabolic Decline in Dementia: A Potential Outcome Measure in Alzheimer's Disease Treatment Studies,"
Am. J. Psychiatry, vol. 159, pp. 738-745, 2002.- [62] H. Braak and E. Braak, "Evolution of the Neuropathology of Alzheimer's Disease,"
Acta Neurologica Scandinavica Supplementum, vol. 165, pp. 3-12, 1996.- [63] H. Braak, E. Braak, and J. Bohl, "Staging of Alzheimer-Related Cortical Destruction,"
European Neurology, vol. 33, pp. 403-408, 1993.- [64] M.D. Ikonomovic, W.E. Klunk, E.E. Abrahamson, C.A. Mathis, J.C. Price, N.D. Tsopelas, B.J. Lopresti, S. Ziolko, W. Bi, W.R. Paljug, M.L. Debnath, C.E. Hope, B.A. Isanski, R.L. Hamilton, and S.T. DeKosky, "Post-Mortem Correlates of In Vivo PiB-PET Amyloid Imaging in a Typical Case of Alzheimer's Disease,"
Brain, vol. 131, pp. 1630-1645, 2008.- [65] W.E. Klunk, H. Engler, A. Nordberg, Y. Wang, G. Blomqvist, D.P. Holt, M. Bergstrom, I. Savitcheva, G.F. Huang, S. Estrada, B. Ausen, M.L. Debnath, J. Barletta, J.C. Price, J. Sandell, B.J. Lopresti, A. Wall, P. Koivisto, G. Antoni, C.A. Mathis, and B. Langstrom, "Imaging Brain Amyloid in Alzheimer's Disease with Pittsburgh Compound-B,"
Annals of Neurology, vol. 55, pp. 306-319, 2004.- [66] P.A. Reuter-Lorenz and J.A. Mikels, "A Split-Brain Model of Alzheimer's Disease? Behavioral Evidence for Comparable Intra and Interhemispheric Decline,"
Neuropscyhologia, vol. 43, pp. 1307-1317, 2005.- [67] A.M. Lipton, R. Benavides, L.S. Hynan, F.J. Bonte, T.S. Harris, C.L. WhiteIII, and E.H. Bigio, "Lateralization on Neuroimaging Does Not Differentiate Frontotemporal Lobar Degeneration from Alzheimer's Disease,"
Dementia and Geriatric Cognitive Disorders, vol. 17, no. 4, pp. 324-327, 2004.- [68] T. Hedden, K.R. Van Dijk et al., "Disruption of Functional Connectivity in Clinically Normal Older Adults Harboring Amyloid Burden,"
J. Neuroscience, vol. 29, pp. 12686-12694, 2009.- [69] J.R. Andrews-Hanna et al., "Disruption of Large-Scale Brain Systems in Advanced Aging,"
Neuron, vol. 56, pp. 924-935, 2007.- [70] X. Wu, R. Li, A.S. Fleisher, E.M. Reiman, K. Chen, and L. Yao, "Altered Default Mode Network Connectivity in AD—A Resting Functional MRI and Bayesian Network Study,"
Human Brain Mapping, vol. 32, pp. 1868-1881, 2011.- [71] B. Efron and R.J. Ribshirani,
An Introduction to the Bootstrap. CRC Press, 1994.- [72] P. Good,
Permutation, Parametric and Bootstrap Tests of Hypotheses, third ed. Springer, 2005.- [73] N. Meinshausen and P. Buehlmann, "Stability Selection,"
J. Royal Statistical Soc., Series B, vol. 72, pp. 417-473, 2010. |