2015 International Conference on Parallel Architecture and Compilation (PACT) (2015)
San Francisco, CA, USA
Oct. 18, 2015 to Oct. 21, 2015
DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/PACT.2015.19
Accurate prediction of applications' performanceand functional behavior is a critical component for a widerange of tools, including anomaly detection, task scheduling and approximate computing. Statistical modeling is a very powerful approach for making such predictions and it uses observations of application behavior on a small number of training cases to predict how the application will behave in practice. However, the fact that applications' behavior often depends closely on their configuration parameters and properties of their inputs means that any suite of application training runs will cover only a small fraction of its overall behavior space. Since a model's accuracy often degrades as application configuration and inputs deviate further from its training set, this makes it difficult to act based on the model's predictions. This paper presents a systematic approach to quantify theprediction errors of the statistical models of the applicationbehavior, focusing on extrapolation, where the application configuration and input parameters differ significantly from the model's training set. Given any statistical model of application behavior and a data set of training application runs from which this model is built, our technique predicts the accuracy of the model for predicting application behavior on a new run on hitherto unseen inputs. We validate the utility of this method by evaluating it on the use case of anomaly detection for seven mainstream applications and benchmarks. The evaluation demonstrates that our technique can reduce false alarms while providing high detection accuracy compared to a statistical, input-unaware modeling technique.
Predictive models, Measurement, Training, Data models, Analytical models, Instruments, Microwave integrated circuits,input-aware technique, Prediction error characterization, extrapolation error, performance estimation, anomaly detection, statistical modeling, software performance bugs
Subrata Mitra, Greg Bronevetsky, Suhas Javagal, Saurabh Bagchi, "Dealing with the Unknown: Resilience to Prediction Errors", 2015 International Conference on Parallel Architecture and Compilation (PACT), vol. 00, no. , pp. 331-342, 2015, doi:10.1109/PACT.2015.19