This Article 
 Bibliographic References 
 Add to: 
Scenario-Based Assessment of Nonfunctional Requirements
May 2005 (vol. 31 no. 5)
pp. 392-409
This paper describes a method and a tool for validating nonfunctional requirements in complex socio-technical systems. The System Requirements Analyzer (SRA) tool validates system reliability and operational performance requirements using scenario-based testing. Scenarios are transformed into sequences of task steps and the reliability of human agents performing tasks with computerized technology is assessed using Bayesian Belief Network (BN) models. The tool tests system performance within an envelope of environmental variations and reports the number of tests that pass a benchmark threshold. The tool diagnoses problematic areas in scenarios representing pathways through system models, assists in the identification of their causes, and supports comparison of alternative requirements specifications and system designs. It is suitable for testing socio-technical systems where operational scenarios are sequential and deterministic, in domains where designs are incrementally modified so set up costs of the BNs can be defrayed over multiple tests.

[1] J.S. Anderson and B. Durley, “Using Scenarios in Deficiency-Driven Requirements Engineering,” Proc. Conf. Requirements Eng. (RE '93), 1993.
[2] J.S. Annett and K.D. Duncam, “Task Analysis and Training Design,” Occupational Psychology, vol. 41, pp. 211-221, 1967.
[3] R.W. Bailey, Human Performance Engineering: A Guide for System Designers. Prentice Hall, 1982.
[4] J.M. Carroll, Scenario-Based Design: Envisioning Work and Technology in System Development. Wiley, 1995.
[5] J.M. Carroll, M.B. Rosson, G. Chin, and J. Koenemann, “Requirements Development in Scenario-Based Design,” IEEE Trans. Software Eng., vol. 24, pp. 1156-1170, 1998.
[6] K. Casey and C. Exton, “A Java 3D Implementation of a Geon Based Visualization Tool for UML,” Proc. Int'l Conf. Principles and Practice of Programming in Java, 2003.
[7] S. Cunning Jr., “Test Scenario Generation from Structural Requirements Specification,” Proc. Symp. Eng. of Computer-Based Systems (ECBS '99), 1999.
[8] A. Davis and P. Hsia, “Giving Voice to Requirements Engineering,” IEEE Software, vol. 11, pp. 12-16, 1994.
[9] J.C.S. doPradoLeite and L.M. Cysneiros, “Nonfunctional Requirements: From Elicitation to Conceptual Models,” IEEE Trans. Software Eng., vol. 30, pp. 328-350, 2004.
[10] P. Dubois, E. Dubois, and J. Zeippen, “On the Use of a Formal Representation,” Proc. Third IEEE Int'l Symp. Requirements Eng., 1997.
[11] G. Engels, “Model-Based Verification and Validation of Properties,” Electronic Notes in Theoretical Computer Science, vol. 82, 2003.
[12] N. Fenton, “Applying Bayesian Belief Networks to Critical Systems Assessment,” Critical Systems, vol. 8, pp. 10-13, 1999.
[13] N. Fenton, “A Critique of Software Defect Prediction Models,” IEEE Trans. Software Eng., vol. 25, pp. 675-689, 1999.
[14] N. Fenton and P. Cates, “JSIMP: BN Model and Tool for the SIMP Project,” Queen Mary (Univ. of London), London, July 2003.
[15] N. Fenton, P. Krause, and M. Neil, “Software Measurement: Uncertainty and Causal Modeling,” IEEE Software, vol. 10, pp. 116-122, 2002.
[16] N. Fenton and B. Littlewood, Software Reliability and Metrics. Elsevier, 1991.
[17] N. Fenton and N. Maiden, “Making Decisions: Using BNs and MCDA,” Computer Science Dept., Queen Mary and Westfield College, London, 2000.
[18] N. Fenton and M. Neil, “Software Metrics: Successes, Failures and New Directions,” J. Systems Software, 2000.
[19] N. Fenton and S.L. Pfleeger, Software Metrics: A Rigorous Approach. London: Int'l Thomson Computer Press, 1997.
[20] A. Fuxman, M. Pistore, J. Mylopoulos, and P. Traverso, “Model Checking Early Requirements Specifications in Tropos,” Proc. Int'l Symp. Requirements Eng. 01, 2001.
[21] J. Galliers, S. Sutcliffe, and S. Minocha, “An Impact Analysis Method for Safety-Critical User Interface Design,” IEEE Trans. Software Eng., vol. 25, pp. 341-369, 1999.
[22] A. Gemino, “Empirical Comparison of Animation and Narration in Requirements Validation,” Requirements Eng., vol. 9, pp. 153-168, 2003.
[23] A. Grau and M. Kowsari, “A Validation System for Object-Oriented Specifications of Information Systems,” Proc. First East European Symp. Advances in Databases and Information Systems (ADBIS '97), 1997.
[24] A. Gregoriades, J.E. Shin, and A.G. Sutcliffe, “Human-Centred Requirements Engineering,” Proc. Conf. Requirements Eng. (RE '04), pp. 154-164, 2004.
[25] A. Gregoriades, A.G. Sutcliffe, and H. Karanikas, “Evaluation of the SRA Tool Using Data Mining Techniques,” Proc. Conf. Advanced Information Systems Eng., 2003.
[26] A. Gregoriades, A.G. Sutcliffe, and J.E. Shin, “Assessing the Reliability of Socio-Technical Systems,” Proc. 12th Ann. Int'l Council on Systems Eng. (INCOSE) Symp., 2002.
[27] K.M. Hansen, A.P. Ravn, and V. Stavridou, “From Safety Analysis to Software Requirements,” IEEE Trans. Software Eng., vol. 24, pp. 573-584, 1998.
[28] P. Haumer, K. Pohl, and K. Weidenhaupt, “Requirements Elicitation and Validation with Real World Scenes,” IEEE Trans. Software Eng., vol. 24, pp. 1036-1054, 1998.
[29] C. Heitmeyer, J. Kirby, and B. Labaw, “Applying the SCR Requirements Method to a Weapons Control Panel: An Experience Report,” Proc. Conf. FMSP 98, 1998.
[30] E. Hollnagel, Cognitive Reliability & Error Analysis Method. Elsevier Science, 1998.
[31] E. Hollnagel, Human Reliability Analysis Context and Control. New York: Academic Press, 1993.
[32] E. Hollnagel, “The Phenotype of Erroneous Actions: Implications for HCI Design,” Human-Computer Interaction and Complex Systems, G. Weir and J. Alty, eds., 1990.
[33] P. Hsia, A. Davis, and D. Kung, “Status Report: Requirements Engineering,” IEEE Software, vol. 10, pp. 75-79, 1993.
[34] R. Jeffords and C. Heitmeyer, “A Strategy for Efficient Verifying Requirements Specification Using Composition and Invariants,” Proc. European Software Eng. Conf. and ACM SIGSOFT Symp. Foundations of Software Eng., 2003.
[35] B.I. Kirwan, A Guide to Practical Human Reliability Assessment. London: Taylor and Francis, 1994.
[36] V. Lalioti, “Animation for Validation of Business System Specifications,” Proc. Hawaii Int'l Conf. System Sciences 30, The Dynamics of Business Systems Eng., pp. 7-10, Jan. 1997.
[37] V. Lalioti and P. Loucopoulos, “Visualisation of Conceptual Specifications,” Information Systems, vol. 19, pp. 291-309, 1994.
[38] N.G. Leveson, “Intent Specifications: An Approach to Building Human-Centered Specifications,” IEEE Trans. Software Eng., vol. 26, pp. 15-35, 2000.
[39] N.G. Leveson, Safeware: System Safety and Computers. Reading, Mass.: Addison Wesley, 1995.
[40] J. Mylopoulos, L. Chung, and B. Nixon, “Representing and Using Nonfunctional Requirements: A Process Oriented Approach,” IEEE Trans. Software Eng., vol. 18, pp. 483-497, 1992.
[41] J. Mylopoulos, L. Chung, and E. Yu, “From Object-Oriented to Goal-Oriented Requirements Analysis,” Comm. ACM, vol. 42, pp. 1-7, 1999.
[42] M. Neil, N. Fenton, and L. Nielsen, “Building Large-Scale Bayesian Networks,” The Knowledge Eng. Rev., vol. 15, pp. 257-284, 2000.
[43] B. Nixon, “Management of Performance Requirements for Information Systems,” IEEE Trans. Software Eng., vol. 26, pp. 1122-1146, 2000.
[44] D. Norman, The Psychology of Everyday Things. New York: MIT Press, 1988.
[45] J. Pearl, Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Information. San Francisco: Morgan Kaufmann, 1988.
[46] C. Potts, “ScenIC: A Strategy for Inquiry-Driven Requirements Determination,” Proc. Int'l Symp. Requirements Eng., 1999.
[47] C. Potts and A. Anton, “A Representational Framework for Scenarios of System Use,” Requirements Eng., vol. 3, pp. 219-241, 1998.
[48] C. Potts, K. Takahashi, and A. Anton, “Inquiry-Based Requirements Analysis,” IEEE Software, vol. 11, pp. 21-32, 1994.
[49] C. Potts, K. Takahashi, J. Smith, and K. Ota, “An Evaluation of Inquiry-Based Requirements Analysis for an Internet Service,” Proc. Second Int'l Symp. Requirements Eng., 1995.
[50] J. Rasmussen, “Human Error and the Problem of Causality in Analysis of Accidents,” Philosophical Trans. Royal Soc. of London Series B-Biological Sciences, vol. 327, pp. 449-462, 1990.
[51] J. Rasmussen, “Skills, Rules, Knowledge; Signals, Signs, and Symbols; and Other Distinctions in Human Performance Models,” IEEE Trans. Systems, Man, and Cybernetics, vol. 13, pp. 257-266, 1983.
[52] J. Reason, Human Error. New York: Cambridge Univ. Press, 1990.
[53] J. Reason, Managing the Risks of Organizational Accidents. Ashgate: Aldershot, 2000.
[54] C. Rolland, C. Souveyet, and C.B. Achour, “Guiding Goal Modeling Using Scenarios,” IEEE Trans. Software Eng., vol. 24, pp. 1055-1071, 1998.
[55] J. Ryser and M. Glinz, “A Scenario-Based Approach to Validating and Testing Software Systems Using Statecharts,” Proc. 12th Int'l Conf. Software and Systems Eng. and Their Applications (ICSSEA '99), 1999.
[56] N. Seyff, P. Grunbacher, N. Maiden, and A. Toscar, “Requirements Engineering Tools Go Mobile,” Proc. Int'l Conf. Software Eng. (ICSE 04), 2004.
[57] J.E. Shin, A. Sutcliffe, and A. Gregoriades, “Scenario Advisor Tool for Requirements Engineering,” Requirements Eng., http://www. m3tlwhruwl4u54qmhqvl&referrer=parent&backto= linkingpublicationresults,1:102830,1 , 2004.
[58] A.G. Sutcliffe, The Domain Theory: Patterns for Knowledge and Software Reuse. Mahwah N.J.: Lawrrence Erlbaum Assocs., 2002.
[59] A.G. Sutcliffe, J. Galliers, and S. Minocha, “Human Errors and System Requirements,” Proc. Fourth IEEE Int'l Symp. Requirements Eng., 1999.
[60] A.G. Sutcliffe and A. Gregoriades, “Validating Functional System Requirements with Scenarios,” Proc. First IEEE Joint Int'l Conf. Requirements Eng. (RE '02), Sept. 2002.
[61] A. Sutcliffe, N. Maiden, S. Minocha, and M. Darrel, “Supporting Scenario Based Requirements Engineering,” IEEE Trans. Software Eng., vol. 24 pp. 1072-1088, 1998.
[62] A.G. Sutcliffe and G. Rugg, “A Taxonomy of Error Types for Failure Analysis and Risk Assessment,” Int'l J. Human Computer Interaction, vol. 10, pp. 381-406, 1998.
[63] A.G. Sutcliffe and M. Ryan, “Assessing the Usability and Efficiency of Design Rationale,” Proc. Conf. Human Computer Interaction INTERACT-97, 1997.
[64] A.D. Swain and H. Guttmann, “Handbook of Human Reliability Analysis with Emphasis on Nuclear Power Plants Applications,” technical report, Nuclear Regulatory Commission, Washington, D.C., 1983.
[65] A. van Lamsweerde, “Goal-Oriented Requirements Engineering: A Guided Tour,” Proc. Fifth IEEE Int'l Symp. Requirements Eng. (RE '01), 2001.
[66] A. van Lamsweerde, “Goal-Oriented Requirements Engineering: A Roundtrip from Research to Practice,” Proc. Requirements Eng. Conf., 2004.
[67] A. van Lamsweerde and E. Letier, “Handling Obstacles in Goal-Oriented Requirements Engineering,” IEEE Trans. Software Eng., vol. 26, pp. 978-1005, 2000.
[68] M. Visser and P.A. Wieringa, “PREHEP: Human Error Probability Based Process Unit Selection,” IEEE Trans. Software Eng., vol. 31, pp. 1-15, 2001.
[69] D. Wright and K. Cai, “Representing Uncertainty for Safety Critical Systems,” PDCS2 Technical Report 135, Center for Software Reliability, London: City Univ., 1994.
[70] E. Yu and J. Mylopoulos, “Towards Modeling Strategic Actor Relationships for Information Systems Development, with Examples from Business Process Reengineering,” Proc. Fourth Workshop Information Technologies and Systems, 1994.
[71] H. Zhu and L. Jin, “Scenario Analysis in an Automated Tool for Requirements Engineering,” Requirements Eng., vol. 5, pp. 2-22, 2000.
[72] H. Ziv and D.J. Richardson, “Constructing Bayesian-Network Models of Software Testing and Maintenance Uncertainties,” Proc. Int'l Conf. Software Maintenance, Sept. 1997.

Index Terms:
Index Terms- Nonfunctional requirements validation, scenario-based testing, Bayesian belief networks, systems engineering.
Andreas Gregoriades, Alistair Sutcliffe, "Scenario-Based Assessment of Nonfunctional Requirements," IEEE Transactions on Software Engineering, vol. 31, no. 5, pp. 392-409, May 2005, doi:10.1109/TSE.2005.59
Usage of this product signifies your acceptance of the Terms of Use.