
This Article  
 
Share  
Bibliographic References  
Add to:  
Digg Furl Spurl Blink Simpy Del.icio.us Y!MyWeb  
Search  
 
ASCII Text  x  
Benjamin W. Wah, "Generalization and Generalizability Measures," IEEE Transactions on Knowledge and Data Engineering, vol. 11, no. 1, pp. 175186, January/February, 1999.  
BibTex  x  
@article{ 10.1109/69.755626, author = {Benjamin W. Wah}, title = {Generalization and Generalizability Measures}, journal ={IEEE Transactions on Knowledge and Data Engineering}, volume = {11}, number = {1}, issn = {10414347}, year = {1999}, pages = {175186}, doi = {http://doi.ieeecomputersociety.org/10.1109/69.755626}, publisher = {IEEE Computer Society}, address = {Los Alamitos, CA, USA}, }  
RefWorks Procite/RefMan/Endnote  x  
TY  JOUR JO  IEEE Transactions on Knowledge and Data Engineering TI  Generalization and Generalizability Measures IS  1 SN  10414347 SP175 EP186 EPD  175186 A1  Benjamin W. Wah, PY  1999 KW  Anomalies in generalization KW  credit assignment problem generalization KW  machine learning KW  subdomains KW  probability of win KW  VLSI cell placement and routing. VL  11 JA  IEEE Transactions on Knowledge and Data Engineering ER   
Abstract—In this paper, we define the generalization problem, summarize various approaches in generalization, identify the credit assignment problem, and present the problem and some solutions in measuring generalizability. We discuss anomalies in the ordering of hypotheses in a subdomain when performance is normalized and averaged, and show conditions under which anomalies can be eliminated. To generalize performance across subdomains, we present a measure called probability of win that measures the probability whether one hypothesis is better than another. Finally, we discuss some limitations in using probabilities of win and illustrate their application in finding new parameter values for TimberWolf, a package for VLSI cell placement and routing.
[1] A. Barr and E.A. Feigenbaum, The Handbook of Artificial Intelligence, vols. 13, William Kaufmann, Los Altos, Calif., 1981.
[2] A.R. Barron, "Approximation and Estimation Bounds for Artificial Neural Networks," Proc. Fourth Ann. Workshop Computational Learning Theory, pp. 243249, Morgan Kaufmann, Palo Alto, Calif., 1991.
[3] E.B. Baum and D. Haussler, "What Size Net Gives Valid Generalization?" D.Z. Anderson, ed., Proc. Neural Information Processing Systems, pp. 8190, American Inst. of Physics, New York, 1988.
[4] A. Blumer, A. Ehrenfeucht, D. Haussler, and M. Warmuth, "Classifying Learnable Geometric Concepts with the VapnikChervonenkis Dimension," Proc. 18th Symp. Theory Computing, pp. 273282, ACM, 1986.
[5] L.B. Booker, D.E. Goldberg, and J.H. Holland, "Classifier Systems and Genetic Algorithms," Artificial Intelligence, vol. 40, no. 1, pp. 235282, 1989.
[6] Encyclopaedia Britannica CD '95, Encyclopaedia Britannica Inc., 1995.
[7] W.W. Cohen, "Generalizing Number and Learning from Multiple Examples in Explanation Based Learning," Machine Learning, pp. 256269, 1988.
[8] G. DeJong and R. Mooney, "ExplanationBased Learning: An Alternative View," Machine Learning, vol. 1, pp. 145176, 1986.
[9] J.L. Devore, Probability and Statistics for Eng. and the Sciences, Brooks/Cole, Monterey, Calif., 1982.
[10] A. Ehrenfeucht, D. Haussler, M. Kearns, and L. Valiant, "A General Lower Bound on The Number of Examples Needed for Learning," D. Haussler and L. Pitt, eds., Proc. Workshop Computational Learning Theory, pp. 139154, Morgan Kaufmann, Palo Alto, Calif., 1988.
[11] J.J. Grefenstette, "Credit Assignment in Rule Discovery Systems Based on Genetic Algorithms," Machine Learning, vol. 3, nos. 23, pp. 225246, Oct. 1988.
[12] D. Haussler, "Generalizing the PAC Model: Sample Size Bounds from Metric DimensionBased Uniform Convergence Results," Proc. 30th Ann. Symp. Foundations Computer Science, pp. 4045. IEEE CS Press, 1989.
[13] J.H. Holland, Adaptation in Natural and Artificial Systems. Univ. of Michigan Press, Ann Arbor, 1975.
[14] J.H. Holland, "Properties of the Bucket Brigade Algorithm," J.J. Grefenstette, ed., Proc. Int'l Conf. Genetic Algorithms and Their Applications, pp. 17, Robotics Inst. of Carnegie Mellon Univ., Pittsburgh, Pa., 1985.
[15] A. Ieumwananonthachai and B.W. Wah, "Statistical Generalization of PerformanceRelated Heuristics for KnowledgeLean Applications," Int'l J. Artificial Intelligence Tools, vol. 5, nos. 12, pp. 6179, June 1996.
[16] A. Ieumwananonthachai and B.W. Wah, "Statistical Generalization of PerformanceRelated Heuristics for KnowledgeLean Applications," D. Dasgupta and Z. Michalewicz, eds., Evolutionary Algorithms in Eng. Applications, pp. 293313, SpringerVerlag, New York, 1997.
[17] M. Kearns, M. Li, L. Pitt, and L. Valiant, "Recent Results on Boolean Concept Learning," Machine Learning, pp. 337352, 1987.
[18] M. Kearns, M. Li, L. Pitt, and L.G. Valiant, "Recent Results on Boolean Concept Learning," Proc. Fourth Int'l Workshop Machine Learning, Morgan Kaufmann, Palo Alto, Calif., 1987.
[19] S. Kirkpatrick Jr., C.D. Gelatt, and M.P. Vecchi, "Optimization By Simulated Annealing," Science, vol. 220, no. 4,598, pp. 671680, May 1983.
[20] J.R. Koza, Genetic Programming, MIT Press, Cambridge, Mass., 1992.
[21] "VLSI Layout and Synthesis Benchmarks," Proc. Int'l Workshop Layout Synthesis, 1992.
[22] J.L. McClelland and D.E. Rumelhart, Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol. 1on foundations, Bradford Books (MIT Press), Cambridge, Mass., 1985.
[23] Artificial Neural Networks: Concepts and Theory, P. Mehra and B.W. Wah, eds., IEEE CSPress, Los Alamitos, Calif., 1992.
[24] P. Mehra and B.W. Wah, "A Systematic Method for Automated Learning of LoadBalancing Strategies in Distributed Systems," Int'l J. Systems Sciences, 1997.
[25] T.M. Mitchell, "Generalization As Search," Artificial Intelligence, vol. 18, pp. 203226, 1982.
[26] T. Mitchel, R. Keller, and S. KedarCabelli, “ExplanationBased Generalization: A Unifying View,” Machine Learning, pp. 47–80, 1986.
[27] D. Nguyen and B. Widrow, “The Truck Backer‐Upper: An Example of Self‐Learning in Neural Networks,” Proc. Int’l Joint Conf. Neural Networks, Vol. 2, June 1989.
[28] N.J. Nilsson, The Math. Foundations of Learning Machines, Morgan Kaufmann, Palo Alto, Calif., 1990.
[29] N. Sauer, "On the Density of Families of Sets," J. Combinatorial Theory, vol. 13, pp. 145147, 1972.
[30] C. Sechen, VLSI Placement and Global Routing Using Simulated Annealing, Kluwer, Boston, 1988.
[31] H.A. Simon and G. Lea, "Problem Solving and Rule Induction: A Unified View," Knowledge and Cognition, L.W. Gregg, ed., pp. 105127, Erlbaum, Potomac, Md., 1974.
[32] S.F. Smith, "Flexible Learning of Problem Solving Heuristics Through Adaptive Search," Proc. Int'l Joint Conf. Artificial Intelligence, pp. 422425, Morgan Kaufmann, 1983.
[33] R.S. Sutton, "Temporal Credit Assignment in Reinforcement Learning," PhD thesis, Univ. of Massachusetts, Amherst, Feb. 1984.
[34] C.C. Teng and B.W. Wah, "Automated Learning of the Minimal Configuration of A Feed Forward Neural Network," IEEE Trans. Neural Networks, vol. 7, no. 5, pp. 1,0721,085, Sept. 1996.
[35] L.G. Valiant, “A Theory of the Learnable,” Comm. ACM, vol. 27, no. 11, pp. 11341142, Nov. 1984.
[36] V.N. Vapnik and A.Y. Chervonenkis, "On The Uniform Convergence of Relative Frequencies of Events to Their Probabilities," Theoretical Probability and its Applications, vol. XVI, no. 2, pp. 264280, 1971.
[37] B.W. Wah,“Populationbased learning: A new method for learning from examples under resource constraints,” IEEE Trans. Knowledge and Data Engineering, vol. 4, no. 5, pp. 454574, Oct. 1992.
[38] B.W. Wah and A. Ieumwananonthachai, "Teacher: A GeneticsBased System for Learning and for Generalizing Heuristics," X. Yao, ed., Evolutionary Computation, World Scientific, 1998.
[39] B.W. Wah, A. Ieumwananonthachai, L.C. Chu, and A.N. Aizawa, "GeneticsBased Learning of New Heuristics: Rational Scheduling of Experiments and Generalization," IEEE Trans. Knowledge and Data Eng., vol. 7, no. 5, pp. 763785, 1995.
[40] B.W. Wah, A. Ieumwananonthachai, and Y.C. Li, "Generalization of Heuristics Learned in Genetics Based Learning," S.K. Pal and P. Wang, eds., Genetic Algorithms and Pattern Recognition, pp. 87126, CRC Press, 1996.
[41] B.W. Wah, A. Ieumwananonthachai, and T. Yu, "Genetics Based Learning and Statistical Generalization," S. Tzafestas, ed., KnowledgeBased Systems: Advanced Concepts, Tools, and Applications, pp. 319347, World Scientific, 1997.
[42] D.A. Waterman, "Generalization Learning Techniques for Automating the Learning of Heuristics," Artificial Intelligence, vol. 1, nos. 12, pp. 121170, 1970.