This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
A Performance Anomaly Detection and Analysis Framework for DBMS Development
Aug. 2012 (vol. 24 no. 8)
pp. 1345-1360
ASCII Text
x 
 
Donghun Lee, Sang K. Cha, Arthur H. Lee, "A Performance Anomaly Detection and Analysis Framework for DBMS Development," IEEE Transactions on Knowledge and Data Engineering, vol. 24, no. 8, pp. 1345-1360, Aug., 2012.
 
 
BibTex
x
 
@article{ 10.1109/TKDE.2011.88,
author = {Donghun Lee and Sang K. Cha and Arthur H. Lee},
title = {A Performance Anomaly Detection and Analysis Framework for DBMS Development},
journal ={IEEE Transactions on Knowledge and Data Engineering},
volume = {24},
number = {8},
issn = {1041-4347},
year = {2012},
pages = {1345-1360},
doi = {http://doi.ieeecomputersociety.org/10.1109/TKDE.2011.88},
publisher = {IEEE Computer Society},
address = {Los Alamitos, CA, USA},
}
 
 
RefWorks Procite/RefMan/Endnote
x 
 
TY - JOUR
JO - IEEE Transactions on Knowledge and Data Engineering
TI - A Performance Anomaly Detection and Analysis Framework for DBMS Development
IS - 8
SN - 1041-4347
SP1345
EP1360
EPD - 1345-1360
A1 - Donghun Lee,
A1 - Sang K. Cha,
A1 - Arthur H. Lee,
PY - 2012
KW - CUSUM chart
KW - differential profiling
KW - statistical process control (SPC)
KW - performance anomaly
KW - DBMS.
VL - 24
JA - IEEE Transactions on Knowledge and Data Engineering
ER -
 
Donghun Lee, SAP R&D Center Korea, Seoul
Sang K. Cha, Seoul National University, Seoul
Arthur H. Lee, Claremont McKenna College, Claremont
Detecting performance anomalies and finding their root causes are tedious tasks requiring much manual work. Functionality enhancements in DBMS development as in most software development often introduce performance problems in addition to bugs. To detect the problems as soon as they are introduced, which often happens during the early phases of a development cycle, we adopt performance regression testing early in the process. In this paper, we describe a framework that we developed to manage performance anomalies after establishing a set of conditions for a problem to be considered an anomaly. The framework uses Statistical Process Control (SPC) charts to detect performance anomalies and differential profiling to identify their root causes. By automating the tasks within the framework we were able to remove most of the manual overhead in detecting anomalies and reduce the analysis time for identifying the root causes by about 90 percent in most cases. The tools developed and deployed based on the framework allow us continuous, automated daily monitoring of performance in addition to the usual functionality monitoring in our DBMS development.

[1] A. de Vries and B.J. Conlin, "Article: Design and Performance of Statistical Process Control Charts Applied to Estrous Detection Efficiency," J. Dairy Science vol. 86, pp. 1970-1984, 2003.
[2] A. Avritzer, J. Kondek, D. Liu, and E.J. Weyuker, "Software Performance Testing Based on Workload Characterization," Proc. Third Int'l Workshop Software and Performance (WOSP '02), pp. 17-24, 2002, doi:10.1145/584369.584373.
[3] A. Thiem and K.-U. Sattler, "An Integrated Approach to Performance Monitoring for Autonomous Tuning," Proc. IEEE Int'l Conf. Data Eng. (ICDE '09), pp. 1671-1678, 2009, doi: 10.1109/ICDE.2009.142.
[4] CMMI, http://www.sei.cmu.educmmi/, 2012.
[5] D. Thakkar, A.E. Hassan, G. Hamann, and P. Flora, "A Framework for Measurement Based Performance Modeling," Proc. Seventh Int'l Workshop Software and Performance (WOSP '08), pp. 55-66, 2008, doi: 10.1145/1383559.1383567.
[6] D.R. Slutz, "Massive Stochastic Testing of SQL," Proc. 24th Int'l Conf. Very Large Data Bases (VLDB '98), pp. 618-622, 1998.
[7] D.C. Montgomery, Introduction to Statistical Quality Control, fifth ed. John Wiley & Sons, Inc., 2005.
[8] D.M. Hawkins, "Cumulative Sum Control Charting: An Underutilized SPC Tool," Quality Eng., vol. 5, no. 3, pp. 463-477, 1993, doi: 10.1080/08982119308918986.
[9] D.M. Hawkins and D.H Olwell, Cumulative Sum Charts and Charting for Quality Improvement. Springer Verlag, 1998.
[10] E.J. Weyuker and F.I. Vokolos, "Experience with Performance Testing of Software Systems: Issues," IEEE Trans. Software Eng., vol. 26, no. 12, pp. 1147-1156, Dec. 2000, doi: 10.1109/32.888628.
[11] F. Haftmann, D. Kossmann, and E. Lo, "A Framework for Efficient Regression Tests on Database Applications," The Int'l J. Very Large Data Bases, vol. 16, no. 1, pp. 145-164, 2007, doi: 10.1007/s00778-006-0028-8.
[12] F. Haftmann, D. Kossmann, and E. Lo, "Parallel Execution of Test Runs for Database Application Systems," Proc. 31st Int'l Conf. Very Large Data Bases, pp. 589-600, 2005.
[13] G. Denaro, A. Polini, and W. Emmerich, "Early Performance Testing of Distributed Software Applications," Proc. Fourth Int'l Workshop Software and Performance (WOSP '04), pp. 94-103, 2004, doi: 10.1145/974044.974059.
[14] J.M. Lucas and R.B. Crosier, "Combined Shewhart-CUSUM Quality Control Schemes," J. Quality Technology, vol. 14, no. 2, pp. 51-59, 1982.
[15] J. Gray, P. Sundaresan, S. Englert, K. Baclawski, and P.J. Weinberger, "Quickly Generating Billion-Record Synthetic Databases," Proc. ACM SIGMOD Int'l Conf. Management of Data, 1994, doi: 10.1145/191839.191886.
[16] J.W. Cangussu, R.A. DeCarlo, and A.P. Mathur, "Monitoring the Software Test Process Using Statistical Process Control: A Logarithmic Approach," ACM SIGSOFT Software Eng. Notes, vol. 28, no. 5, pp. 158-167, 2003, doi: 10.1145/940071.940093.
[17] J. Lee, K. Kim, and S.K. Cha, "Differential Logging: A Commutative and Associative Logging Scheme for Highly Parallel Main Memory Database," Proc. 17th Int'l Conf. Data Eng. (ICDE '01), vol. 173, 2001, doi: 10.1109/ICDE.2001.914826.
[18] H. Plattner, In-Memopry Data Management Platform in SAP, SAPPHIRE NOW, http://www.youtube.comwatch?v=iUUH_ HOs7DI , 2010.
[19] H. Plattner and A. Common, "Database Approach for OLTP and OLAP Using an In-Memory Column Database," Proc. 35th SIGMOD Int'l Conf. Management of Data (SIGMOD '09), June/July 2009, ACM 978-1-60558-551-2/09/06.
[20] M. Schulz and B.R. de Supinski, "Practical Differential Profiling," Proc. EuroPar, pp. 97-106, 2007.
[21] M. Hauswirth, P.F. Sweeney, A. Diwan, and M. Hind, "Vertical Profiling: Understanding the Behavior of Object-Oriented Applications," Proc. 19th Ann. ACM SIGPLAN Conf. Object-Oriented Programming, Systems, Languages, and Applications, (O'OPSLA '04), pp. 251-269, 2004, doi: 10.1145/1028976.1028998.
[22] MaxDB, https://www.sdn.sap.com/irj/sdnmaxdb, 2012.
[23] M. Stonebraker, S. Madden, D.J. Abadi, S. Harizopoulos, N. Hachem, and P. Helland, "The End of an Architectural Era: (It's Time for a Complete Rewrite)," Proc. 33rd Int'l Conf. Very Large Data Bases (VLDB '07), pp. 1150-1160, 2007.
[24] M. Woodside, G. Franks, and D.C. Petriu, "The Future of Software Performance Engineering," Proc. Future of Software Eng., pp. 171-187, 2007, doi: 10.1109/FOSE.2007.32 2007.
[25] M. Komuro, "Experiences of Applying SPC Techniques to Software Development Processes," Proc. 28th Int'l Conf. Software Eng. (ICSE '06), pp. 577-584, 2006.
[26] N. Bruno, S. Chaudhuri, and D. Thomas, "Generating Queries with Cardinality Constraints for DBMS Testing," IEEE Trans. Knowledge and Data Eng., vol. 18, no. 12, pp. 1721-1725, Dec. 2006, doi: 10.1109/TKDE.2006.190.
[27] P.E. McKenney, "Differential Profiling," Proc. Third Int'l Workshop Modeling (MASCOTS '95), pp. 237-241, 1995, doi: 10.1109/MASCOT.1995.378681.
[28] S.K. Cha and C. Song, "P∗TIME: Highly Scalable OLTP DBMS for Managing Update-Intensive Stream Workload," Proc. 30th Int'l Conf. Very Large Data Bases (VLDB '04), pp. 1033-1044, 2004.
[29] S.K. Cha, S. Hwang, K. Kim, and K. Kwon, "Cache-Conscious Concurrency Control of Main-Memory Indexes on Shared-Memory Multiprocessor Systems," Proc. 27th Int'l Conf. Very Large Data Bases (VLDB '01), pp. 181-190, 2001.
[30] S. Barber, "Beyond Performance Testing," http://www-128.ibm. com/developerworks/rational/ library4169.html, 2012.
[31] S. Barber, http://www.logigear.com/newsletterexplanation_ of_ performance_testing_on_an_agile_team-part-1.asp , 2010.
[32] S.D. benchmark, http://www.sap.com/solutions/benchmarksd.epx , 2012.
[33] S. Balsamo and A. Di Marco, "Model-Based Performance Prediction in Software Development: A Survey," IEEE Trans. Software Eng., vol. 30, no. 5, pp. 295-310, May 2004.
[34] S.P. Reiss, "Controlled Dynamic Performance Analysis," Proc. Seventh Int'l Workshop Software and Performance (WOSP '08), pp. 43-54, 2008, doi: 10.1145/1383559.1383566.
[35] SunStudio, http://developers.sun.com/sunstudioindex.jsp , 2012.
[36] TPC-H, http://www.tpc.orgtpch, 2012.
[37] V. Chandola, A. Banerjee, and V. Kumar, "Anomaly Detection: A Survey," ACM Computing Surveys, vol. 41, no. 3, pp. 1-58, July 2009.
[38] V.S. Puranik, "CUSUM Quality Control Chart for Monitoring Energy Use Performance," Proc. IEEE Int'l Conf. Industrial Eng. and Eng. Management pp. 1231-1235, 2007, doi: 10.1109/IEEM.2007.4419388.
[39] VTune, http://www.intel.com/cd/software/products/ asmo-na/eng/vtune239144.htm, 2010.
[40] PTU, http://software.intel.com/en-us/articles intel- performance-tuning-utility/, 2012.

Index Terms:
CUSUM chart, differential profiling, statistical process control (SPC), performance anomaly, DBMS.
Citation:
Donghun Lee, Sang K. Cha, Arthur H. Lee, "A Performance Anomaly Detection and Analysis Framework for DBMS Development," IEEE Transactions on Knowledge and Data Engineering, vol. 24, no. 8, pp. 1345-1360, Aug. 2012, doi:10.1109/TKDE.2011.88
Usage of this product signifies your acceptance of the Terms of Use.