The Community for Technology Leaders
RSS Icon
Issue No.08 - Aug. (2012 vol.24)
pp: 1345-1360
Sang K. Cha , Seoul National University, Seoul
Donghun Lee , SAP R&D Center Korea, Seoul
Detecting performance anomalies and finding their root causes are tedious tasks requiring much manual work. Functionality enhancements in DBMS development as in most software development often introduce performance problems in addition to bugs. To detect the problems as soon as they are introduced, which often happens during the early phases of a development cycle, we adopt performance regression testing early in the process. In this paper, we describe a framework that we developed to manage performance anomalies after establishing a set of conditions for a problem to be considered an anomaly. The framework uses Statistical Process Control (SPC) charts to detect performance anomalies and differential profiling to identify their root causes. By automating the tasks within the framework we were able to remove most of the manual overhead in detecting anomalies and reduce the analysis time for identifying the root causes by about 90 percent in most cases. The tools developed and deployed based on the framework allow us continuous, automated daily monitoring of performance in addition to the usual functionality monitoring in our DBMS development.
CUSUM chart, differential profiling, statistical process control (SPC), performance anomaly, DBMS.
Sang K. Cha, Donghun Lee, "A Performance Anomaly Detection and Analysis Framework for DBMS Development", IEEE Transactions on Knowledge & Data Engineering, vol.24, no. 8, pp. 1345-1360, Aug. 2012, doi:10.1109/TKDE.2011.88
[1] A. de Vries and B.J. Conlin, "Article: Design and Performance of Statistical Process Control Charts Applied to Estrous Detection Efficiency," J. Dairy Science vol. 86, pp. 1970-1984, 2003.
[2] A. Avritzer, J. Kondek, D. Liu, and E.J. Weyuker, "Software Performance Testing Based on Workload Characterization," Proc. Third Int'l Workshop Software and Performance (WOSP '02), pp. 17-24, 2002, doi:10.1145/584369.584373.
[3] A. Thiem and K.-U. Sattler, "An Integrated Approach to Performance Monitoring for Autonomous Tuning," Proc. IEEE Int'l Conf. Data Eng. (ICDE '09), pp. 1671-1678, 2009, doi: 10.1109/ICDE.2009.142.
[4] CMMI, http://www.sei.cmu.educmmi/, 2012.
[5] D. Thakkar, A.E. Hassan, G. Hamann, and P. Flora, "A Framework for Measurement Based Performance Modeling," Proc. Seventh Int'l Workshop Software and Performance (WOSP '08), pp. 55-66, 2008, doi: 10.1145/1383559.1383567.
[6] D.R. Slutz, "Massive Stochastic Testing of SQL," Proc. 24th Int'l Conf. Very Large Data Bases (VLDB '98), pp. 618-622, 1998.
[7] D.C. Montgomery, Introduction to Statistical Quality Control, fifth ed. John Wiley & Sons, Inc., 2005.
[8] D.M. Hawkins, "Cumulative Sum Control Charting: An Underutilized SPC Tool," Quality Eng., vol. 5, no. 3, pp. 463-477, 1993, doi: 10.1080/08982119308918986.
[9] D.M. Hawkins and D.H Olwell, Cumulative Sum Charts and Charting for Quality Improvement. Springer Verlag, 1998.
[10] E.J. Weyuker and F.I. Vokolos, "Experience with Performance Testing of Software Systems: Issues," IEEE Trans. Software Eng., vol. 26, no. 12, pp. 1147-1156, Dec. 2000, doi: 10.1109/32.888628.
[11] F. Haftmann, D. Kossmann, and E. Lo, "A Framework for Efficient Regression Tests on Database Applications," The Int'l J. Very Large Data Bases, vol. 16, no. 1, pp. 145-164, 2007, doi: 10.1007/s00778-006-0028-8.
[12] F. Haftmann, D. Kossmann, and E. Lo, "Parallel Execution of Test Runs for Database Application Systems," Proc. 31st Int'l Conf. Very Large Data Bases, pp. 589-600, 2005.
[13] G. Denaro, A. Polini, and W. Emmerich, "Early Performance Testing of Distributed Software Applications," Proc. Fourth Int'l Workshop Software and Performance (WOSP '04), pp. 94-103, 2004, doi: 10.1145/974044.974059.
[14] J.M. Lucas and R.B. Crosier, "Combined Shewhart-CUSUM Quality Control Schemes," J. Quality Technology, vol. 14, no. 2, pp. 51-59, 1982.
[15] J. Gray, P. Sundaresan, S. Englert, K. Baclawski, and P.J. Weinberger, "Quickly Generating Billion-Record Synthetic Databases," Proc. ACM SIGMOD Int'l Conf. Management of Data, 1994, doi: 10.1145/191839.191886.
[16] J.W. Cangussu, R.A. DeCarlo, and A.P. Mathur, "Monitoring the Software Test Process Using Statistical Process Control: A Logarithmic Approach," ACM SIGSOFT Software Eng. Notes, vol. 28, no. 5, pp. 158-167, 2003, doi: 10.1145/940071.940093.
[17] J. Lee, K. Kim, and S.K. Cha, "Differential Logging: A Commutative and Associative Logging Scheme for Highly Parallel Main Memory Database," Proc. 17th Int'l Conf. Data Eng. (ICDE '01), vol. 173, 2001, doi: 10.1109/ICDE.2001.914826.
[18] H. Plattner, In-Memopry Data Management Platform in SAP, SAPPHIRE NOW, HOs7DI , 2010.
[19] H. Plattner and A. Common, "Database Approach for OLTP and OLAP Using an In-Memory Column Database," Proc. 35th SIGMOD Int'l Conf. Management of Data (SIGMOD '09), June/July 2009, ACM 978-1-60558-551-2/09/06.
[20] M. Schulz and B.R. de Supinski, "Practical Differential Profiling," Proc. EuroPar, pp. 97-106, 2007.
[21] M. Hauswirth, P.F. Sweeney, A. Diwan, and M. Hind, "Vertical Profiling: Understanding the Behavior of Object-Oriented Applications," Proc. 19th Ann. ACM SIGPLAN Conf. Object-Oriented Programming, Systems, Languages, and Applications, (O'OPSLA '04), pp. 251-269, 2004, doi: 10.1145/1028976.1028998.
[22] MaxDB,, 2012.
[23] M. Stonebraker, S. Madden, D.J. Abadi, S. Harizopoulos, N. Hachem, and P. Helland, "The End of an Architectural Era: (It's Time for a Complete Rewrite)," Proc. 33rd Int'l Conf. Very Large Data Bases (VLDB '07), pp. 1150-1160, 2007.
[24] M. Woodside, G. Franks, and D.C. Petriu, "The Future of Software Performance Engineering," Proc. Future of Software Eng., pp. 171-187, 2007, doi: 10.1109/FOSE.2007.32 2007.
[25] M. Komuro, "Experiences of Applying SPC Techniques to Software Development Processes," Proc. 28th Int'l Conf. Software Eng. (ICSE '06), pp. 577-584, 2006.
[26] N. Bruno, S. Chaudhuri, and D. Thomas, "Generating Queries with Cardinality Constraints for DBMS Testing," IEEE Trans. Knowledge and Data Eng., vol. 18, no. 12, pp. 1721-1725, Dec. 2006, doi: 10.1109/TKDE.2006.190.
[27] P.E. McKenney, "Differential Profiling," Proc. Third Int'l Workshop Modeling (MASCOTS '95), pp. 237-241, 1995, doi: 10.1109/MASCOT.1995.378681.
[28] S.K. Cha and C. Song, "P∗TIME: Highly Scalable OLTP DBMS for Managing Update-Intensive Stream Workload," Proc. 30th Int'l Conf. Very Large Data Bases (VLDB '04), pp. 1033-1044, 2004.
[29] S.K. Cha, S. Hwang, K. Kim, and K. Kwon, "Cache-Conscious Concurrency Control of Main-Memory Indexes on Shared-Memory Multiprocessor Systems," Proc. 27th Int'l Conf. Very Large Data Bases (VLDB '01), pp. 181-190, 2001.
[30] S. Barber, "Beyond Performance Testing," com/developerworks/rational/ library4169.html, 2012.
[31] S. Barber, of_ performance_testing_on_an_agile_team-part-1.asp , 2010.
[32] S.D. benchmark, , 2012.
[33] S. Balsamo and A. Di Marco, "Model-Based Performance Prediction in Software Development: A Survey," IEEE Trans. Software Eng., vol. 30, no. 5, pp. 295-310, May 2004.
[34] S.P. Reiss, "Controlled Dynamic Performance Analysis," Proc. Seventh Int'l Workshop Software and Performance (WOSP '08), pp. 43-54, 2008, doi: 10.1145/1383559.1383566.
[35] SunStudio, , 2012.
[36] TPC-H, http://www.tpc.orgtpch, 2012.
[37] V. Chandola, A. Banerjee, and V. Kumar, "Anomaly Detection: A Survey," ACM Computing Surveys, vol. 41, no. 3, pp. 1-58, July 2009.
[38] V.S. Puranik, "CUSUM Quality Control Chart for Monitoring Energy Use Performance," Proc. IEEE Int'l Conf. Industrial Eng. and Eng. Management pp. 1231-1235, 2007, doi: 10.1109/IEEM.2007.4419388.
[39] VTune, asmo-na/eng/vtune239144.htm, 2010.
[40] PTU, intel- performance-tuning-utility/, 2012.
54 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool