Subscribe

Issue No.01 - First Quarter (2013 vol.6)

pp: 46-59

M. L. Nguyen , DISCO Lab., Nanyang Technol. Univ., Singapore, Singapore

Siu Cheung Hui , DISCO Lab., Nanyang Technol. Univ., Singapore, Singapore

A. C. M. Fong , Sch. of Comput. & Math Sci., Auckland Univ. of Technol., Auckland, New Zealand

DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/TLT.2012.22

ABSTRACT

Web-based testing has become a ubiquitous self-assessment method for online learning. One useful feature that is missing from today's web-based testing systems is the reliable capability to fulfill different assessment requirements of students based on a large-scale question data set. A promising approach for supporting large-scale web-based testing is static test generation (STG), which generates a test paper automatically according to user specification based on multiple assessment criteria. And the generated test paper can then be attempted over the web by users for assessment purpose. Generating high-quality test papers under multiobjective constraints is a challenging task. It is a 0-1 integer linear programming (ILP) that is not only NP-hard but also need to be solved efficiently. Current popular optimization software and heuristic-based intelligent techniques are ineffective for STG, as they generally do not have guarantee for high-quality solutions of solving the large-scale 0-1 ILP of STG. To that end, we propose an efficient ILP approach for STG, called branch-and-cut for static test generation (BAC-STG). Our experimental study on various data sets and a user evaluation on generated test paper quality have shown that the BAC-STG approach is more effective and efficient than the current STG techniques.

INDEX TERMS

ubiquitous computing, computer aided instruction, integer programming, linear programming, quality control, testing,test paper quality, large-scale multiobjective static test generation, Web-based testing, ubiquitous self-assessment method, online learning, high-quality test papers, integer linear programming, ILP, NP-hard problem, branch-and-cut, BAC-STG,Testing, Linear programming, Web and internet services, Runtime, Sparse matrices, Integer linear programming, Computer aided instruction, Online services,integer programming, Testing, Linear programming, Web and internet services, Runtime, Sparse matrices, Integer linear programming, Computer aided instruction, Online services, Web-based testing, Testing, Linear programming, Web and internet services, Runtime, Sparse matrices, Integer linear programming, Computer aided instruction, Online services, static test generation, Testing, Linear programming, Web and internet services, Runtime, Sparse matrices, Integer linear programming, Computer aided instruction, Online services, multiobjective optimization, Testing, Linear programming, Web and internet services, Runtime, Sparse matrices, Integer linear programming, Computer aided instruction, Online services

CITATION

M. L. Nguyen, Siu Cheung Hui, A. C. M. Fong, "Large-Scale Multiobjective Static Test Generation for Web-Based Testing with Integer Programming",

*IEEE Transactions on Learning Technologies*, vol.6, no. 1, pp. 46-59, First Quarter 2013, doi:10.1109/TLT.2012.22REFERENCES

- [1] C. Ramos, C. Frasson, and S. Ramachandran, "Introduction to the Special Issue on Real World Applications of Intelligent Tutoring Systems,"
IEEE Trans. Learning Technologies, vol. 2, no. 2, pp. 62-63, Apr. 2009.- [2] P. Brusilovsky and P. Miller, "Web-Based Testing for Distance Education,"
Proc. World Conf. WWW and Internet (WebNet), vol. 99, pp. 24-30, 1999.- [3] R. Conejo, E. Guzmán, E. Millán, M. Trella, J.L. Pérez-De-La-Cruz, and A. Ríos, "SIETTE: A Web-Based Tool for Adaptive Testing,"
Int'l J. Artificial Intelligence in Education, vol. 14, no. 1, pp. 29-61, 2004.- [4] F.G. Martin, "Will Massive Open Online Courses Change How We Teach?"
Comm. ACM, vol. 55, no. 8, pp. 26-28, http://doi.acm.org/10.11452240236.2240246 , Aug. 2012.- [5] K. Hopkins,
Educational and Psychological Measurement and Evaluation. ERIC, 1998.- [6] D. Thissen, B. Reeve, J. Bjorner, and C. Chang, "Methodological Issues for Building Item Banks and Computerized Adaptive Scales,"
Quality of Life Research, vol. 16, pp. 109-119, 2007.- [7] T. Theunissen, "Binary Programming and Test Design,"
Psychometrika, vol. 50, no. 4, pp. 411-420, 1985.- [8] G. Hwang, B.M. Lin, and T. Lin, "An Effective Approach for Test-Sheet Composition with Large-Scale Item Banks,"
Computers & Education, vol. 46, no. 2, pp. 122-139, 2006.- [9] G.J. Hwang, P.Y. Yin, and S.H. Yeh, "A Tabu Search Approach to Generating Test Sheets for Multiple Assessment Criteria,"
IEEE Trans. Education, vol. 49, no. 1, pp. 88-97, Sept. 2006.- [10] A. Schrijver,
Theory of Linear and Integer Programming. John Wiley & Sons, 1986.- [11] D. Bertsimas and R. Weismantel,
Optimization over Integers. Dynamic Ideas, 2005.- [12] X.M. Hu, J. Zhang, H.S.H. Chung, O. Liu, and J. Xiao, "An Intelligent Testing System Embedded with an Ant-Colony-Optimization-Based Test Composition Method,"
IEEE Trans. Systems, Man, and Cybernetics, vol. 39, no. 6, pp. 659-669, Nov. 2009.- [13] G.J. Hwang, B. Lin, H.H. Tseng, and T.L. Lin, "On the Development of a Computer-Assisted Testing System with Genetic Test Sheet-Generating Approach,"
IEEE Trans. Systems, Man, and Cybernetics, vol. 35, no. 4, pp. 590-594, Nov. 2005.- [14] W.F. Rui, W.W. Hong, P.Q. Ke, Z.F. Chao, and J.J. Liang, "A Novel Online Test-Sheet Composition Approach for Web-Based Testing,"
Proc. Symp IT in Medicine Education, pp. 700-705, 2009.- [15] J. Adema and W. Vander Linden, "Algorithms for Computerized Test Construction Using Classical Item Parameters,"
J. Educational and Behavioral Statistics, vol. 14, no. 3, pp. 279-290, 1989.- [16] J. Adema, E. Boekkooi-Timminga, and W. van der Linden, "Achievement Test Construction Using 0-1 Linear Programming,"
European J. Operational Research, vol. 55, no. 1, pp. 103-111, 1991.- [17] IBM, "CPLEX Optimizer (11.0)," http://www-01.ibm.com/ software/integration/ optimizationcplex-optimizer/, 2012.
- [18] GUROBI, "Gurobi Optimizer (Version 5.0)," http:/www.gurobi. com/, 2011.
- [19] J.E. Mitchell, "Integer Programming: Branch-and-Cut Algorithms,"
Encyclopedia of Optimization, vol. 2, pp. 519-525, Kluwer Press, 2001.- [20] C.L. Lee, C.H. Huang, and C.J. Li, "Test-Sheet Composition Using Immune Algorithm for E-Learning Application,"
New Trends in Applied Artificial Intelligence, vol. 4570, pp. 823-833, 2007.- [21] T.F. Ho, P.Y. Yin, G.J. Hwang, S.J. Shyu, and Y.N. Yean, "Multi-Objective Parallel Test-Sheet Composition Using Enhanced Particle Swarm Optimization,"
J. Educational Technology Soc., vol. 12, no. 4, pp. 193-206, 2008.- [22] K.H. Tsai, T.I. Wang, T.C. Hsieh, T.K. Chiu, and M.C. Lee, "Dynamic Computerized Testlet-Based Test Generation System by Discrete PSO with Partial Course Ontology,"
Expert Systems with Applications, vol. 37, no. 1, pp. 774-786, 2009.- [23] M.L. Nguyen, S.C. Hui, and A.C.M. Fong, "An Efficient Multi-Objective Optimization Approach for Online Test Paper Generation,"
Proc. IEEE Symp. Computational Intelligence in Multicriteria Decision-Making (MDCM), pp. 182-189, 2011.- [24] F.B. Baker and S.H. Kim,
Item Response Theory. Marcel Dekker, 1992.- [25] P. Songmuang and M. Ueno, "Bees Algorithm for Construction of Multiple Test Forms in E-Testing,"
IEEE Trans. Learning Technologies, vol. 4, no. 3, pp. 209-221, July-Sept. 2011.- [26] E. Boekkoi-Timminga,
Simultaneous Test Construction by Zero-One Programming. Dept. of Education, Univ. of Twente, 1986.- [27] E. Boekkooi-Timminga, "The Construction of Parallel Tests from IRT-Based Item Banks,"
J. Educational and Behavioral Statistics, vol. 15, no. 2, pp. 129-145, 1990.- [28] R. Fletcher, "A Review of Linear Programming and Its Application to the Assessment Tools for Teaching and Learning (as TTle) Projects," technical report, Massey Univ., 2000.
- [29] T. Theunissen, "Some Applications of Optimization Algorithms in Test Design and Adaptive Testing,"
Applied Psychological Measurement, vol. 10, no. 4, pp. 381-389, 1986.- [30] R. Luecht, "Computer-Assisted Test Assembly Using Optimization Heuristics,"
Applied Psychological Measurement, vol. 22, no. 3, pp. 224-236, 1998.- [31] G.J. Hwang, "A Test-Sheet-Generating Algorithm for Multiple Assessment Requirements,"
IEEE Trans. Education, vol. 46, no. 3, pp. 329-337, 2003.- [32] Z. Gu, G.L. Nemhauser, and M.W.P. Savelsbergh, "Lifted Cover Inequalities for 0-1 Integer Programs: Complexity,"
INFORMS J. Computing, vol. 11, pp. 117-123, 1999.- [33] E. Johnson, G. Nemhauser, and M. Savelsbergh, "Progress in Linear Programming-Based Algorithms for Integer Programming: An Exposition,"
INFORMS J. Computing, vol. 12, no. 1, pp. 2-23, 2000.- [34] R. Baker and K. Yacef, "The State of Educational Data Mining in 2009: A Review and Future Visions,"
J. Educational Data Mining, vol. 1, no. 1, pp. 3-17, 2009.- [35] S. Cetintas, L. Si, Y. Xin, D. Zhang, and J. Park, "Automatic Text Categorization of Mathematical Word Problems,"
Proc. 22nd Int'l FLAIRS Conf., pp. 27-32, 2009.- [36] K. Koedinger, R. Baker, K. Cunningham, A. Skogsholm, B. Leber, and J. Stamper,
A Data Repository for the EDM Community: The PSLC DataShop. CRC Press, 2010.- [37] M.L. Nguyen, S.C. Hui, and A.C.M. Fong, "Content-Based Collaborative Filtering for Question Difficulty Calibration,"
Proc. Pacific Rim Int'l Conf. Trends in Artificial Intelligence, pp. 359-371, 2012.- [38] M. Kojima, N. Megiddo, and S. Mizuno, "A Primal-Dual Infeasible-Interior-Point Algorithm for Linear Programming,"
Math. Programming, vol. 61, no. 1, pp. 263-280, 1993.- [39] S. Martello and P. Toth,
Knapsack Problems: Algorithms and Computer Implementations. Wiley, 1990.- [40] S. Kullback,
Information Theory and Statistics. Dover, 1997.- [41] K. Wauters, P. Desmet, and W. van den Noortgate, "Acquiring Item Difficulty Estimates: A Collaborative Effort of Data and Judgment,"
Proc. Int'l Conf. Education Data Mining, 2011.- [42] M.L. Nguyen, S.C. Hui, and A.C.M. Fong, "Web-Based Mathematics Testing with Automatic Assessment,"
Proc. Pacific Rim Int'l Conf. Trends in Artificial Intelligence, pp. 347-358, 2012. |