Testing
Testing is the process of examining the attributes of software to see if it meets both the defined expectations of the software developers and the actual needs of the users. It can include direct examination of the code (a “static” test) or execution of the code (a “dynamic” test) under circumstances that may or may not be predefined. The great preponderance of testing is dynamic, but some needs are met by static tests (e.g., an examination for hidden code that is a security compromise).
Testing is normally done in more than one level. Some examples of test levels (from ISO 12207) for new development are:
Each software unit and database
Integrated units and components
Tests for each software requirement
Software qualification testing for all requirement
System integration: aggregates of other software configuration items, hardware, manual operations and other systems
System qualification testing for system requirements
Testing to prove that the most recent life cycle step was completed successfully is commonly called “verification”. An example of verification is the unit test verifying that the code is correct. It is also characterized as “doing the job right”. Testing to assure that the system meets its original concept or the user’s actual needs is commonly called “validation”. An example of a validation test level is a user acceptance test at the end of the development life cycle just before going into production. It is also characterized as assuring that development succeeded in “doing the right job”.
The different levels are distinguished by the scope of the items under test (increasing with the achievement of progress), the test environment(s) necessary for the execution of the tests, and sometimes by the measures used to identify the desirable coverage. The term “coverage” refers to the amount of testing that is targeted and/or achieved. Tests are either “white box” (internal structure) or “black box” (data input range definitions) focused. An example of a white box coverage measure is the % of decision branch options exercised. A common goal is to set a 100% target for a new unit during unit testing, but not for a full system during qualification testing (it is too hard to set up an environment that will make all of the options happen). A common black box coverage measure is 100% of boundary values (minimum value(s), maximum value(s), just below the minimum(s), just above the maximum(s)).
A robust model for testing starts with early test planning to assure that the environments are available when needed, the system is itself testable in an optimal manner, and all resources are present at the time that they are required. Some organizations start with a Master Test Plan that identifies how many levels of test will be required, as well as the attributes that distinguish them. After the planning there is usually some kind of test design (documented or undocumented) where there is refinement of the approaches and establishment of the coverage goals. Specific test cases (the input and corresponding expected output) are selected and then executed. Actual results are recorded. Some organizations record all results, others record only the defects that are found. After each test level is executed, some assessment is made as to the success or failure of it. Retesting occurs as fixes are made.
