When tests are executed, the outputs of each test execution should be recorded in a test results file. These results are then assessed against criteria in the test specification to determine the overall outcome of a test.
Each test execution should also be noted in a test log. The test log will contain records of when each test has been executed, the outcome of each test execution, and may also include key observations made during test execution. Often a test log is not maintained for lower levels of testing (unit test and software integration test).
Test reports may be produced at various points during the testing process. A test report will summarizes the results of testing and document any analysis. An acceptance test report often forms a contractual document within which acceptance of software is agreed.
Software can be tested at various stages of the development and with various degrees of rigour. Developers should plan for between 30% and 70% of a projects effort to be expended on verification and validation activities, including software testing.
From an economics point of view, the level of testing appropriate to a particular organization and software application will depend on the potential consequences of undetected bugs. Such consequences can range from a minor inconvenience of having to find a work-round for a bug to multiple deaths. Often overlooked by software developers (but not by customers), is the long term damage to the credibility of an organization which delivers software to users with bugs in it, and the resulting negative impact on future business.
Conversely, a reputation for reliable software will help an organization to obtain future business.
Efficiency and quality are best served by testing software as early in the life cycle as practical, with full regression testing whenever changes are made.
The later a bug is found, the higher the cost of fixing it, so it is sound economics to identify and fix bugs as early as possible. Designing tests will help to identify bugs, even before the tests are Software Testing
executed, so designing tests as early as practical in a software development is a useful means of reducing the cost of identifying and correcting bugs.
In practice the design of each level of software testing will be developed through a number of layers, each adding more detail to the tests.Each level of tests should be designed before the implementation reaches a point, which could influence the design of tests in such a way as to be detrimental to the objectivity of the tests.
Remember: software should be tested against what it is specified to do, not against what it actually observed to do.
Selection of an appropriate testing strategy, good management of the testing process, and appropriate use of tools to support the testing process can maximize the effectiveness of testing effort. The net result will be an increase in quality and a decrease in costs, both of which can only be beneficial to a software developers business.
The following list provides some rules to follow as an aid to effective and beneficial software testing.
Always test against a specification. If tests are not developed from a specification, then it is not testing. Hence, testing is totally reliant upon adequate specification of software.
Document the testing process: specify tests and record test results. Test hierarchically against each level of specification. Finding more errors earlier will ultimately reduce costs.
Plan verification and validation activities, particularly testing.
Complement testing with techniques such as static analysis and dynamic analysis. Always test positively: that the software does what it should, but also negatively: that it doesn’t do what it shouldn’t.
Have the right attitude to testing – it should be a challenge, not the chore it so often becomes.