Testing Strategies

Overview

To frame our discussion, consider:

How do we control the amount of effort, and time needed to test the software?

Are there systematic model for testing software systems?

Are there properties of the system that suggest testing strategies?

OUTLINE

Introduction
Validation and Verification
A Problem of Economics
Unit Testing
Integration Testing
System Testing Strategies
References

Introduction

A testing strategy provides a process that describes for the developer, QA, and the customer the steps conducted as part of testing. Such a strategy includes

test planning,
test case design,
test execution,
data collection
and
effectiveness evaluation.

Validation involves checking that the program meets the expectations of users and customers. Validation is the set of activities that ensures the software has been built according to the customer's requirements.

Verification - "Are we building the product right?"

Verification refers to the set of activities that ensure the software correctly implements specific functions. Verification involves checking that a program conforms to its specification (from one phase to the next).

Software testing should be one element in the developer's quality assurance arsenal.

Others include:

configuration management
quality assurance methods
measurement
SE Methods
process definition and standards

From Software Research, Inc. http://www.soft.com/AppNotes/TestWorksIndex/filters.html

Technique	Life Cycle Stage	Cost to Buy	Cost to Apply	How Hard to Manage	Payoff: Defects Found	Payoff: Product Confidence	How Complete	Field Utilization Achieved	COMMENTS
Requirements Testing	Design & Develop	Moderate	Low	Moderate	High	High	Moderate	Beginning	Use Case Editor or FSM specification plus automated test case creation.
Code Inspection	Design & Develop	Lowest	Moderate	Moderate	High	Low	Low	Low	Human centric, seen by most as an annoyance.
Code Review	Design & Develop	Lowest	Low	Moderate	High	Low	Low	Low	More informal, viewed as less intrusive.
Static Analysis	Develop [Static]	Moderate	High	Very High	Moderate	Mode	Moderate	Lowest	Too many non-critical errors found.
Metric Analysis	Develop [Static]	Moderate	Moderate	Moderate	Low	Low	Low	Low	Controversy on meaning of the metrics.
Bounds Checking	Develop [Dynamic]	Low	Moderate	Low	Moderate	High	High	Moderate	Good dynamic checks with high coverage.
Unit Testing (Testbed w/Stubs)	Develop [Dynamic]	Moderate	Moderate	Low	Moderate	Moderate	Moderate	Moderate	Very few tools now available.
Regression Testing (BlackBox)	Develop QA/Test	Moderate	Moderate	High	Moderate	Moderate	Low	Moderate	Very sensitive to product changes.
Minimal WhiteBox Coverage (C0/C1?/S0)	Develop Integrate QA/Test	Free or Low	Moderate	Low	Moderate	Low	Low	Moderate	Better than nothing at all.
Advanced WhiteBox Path Coverage (Ct)	QA/Test	Moderate	High	Moderate	High	Very High	Very High	Very Rare	Almost like proof of correctness.
Load/Stress Testing	QA/Test	High	High	Moderate	Moderate	Moderate	Moderate	Low	Very specific experiment goals.
Capacity/Tuning Analysis	QA/Test	High	High	Moderate	Moderate	Moderate	High	Moderate	General analysis and "hotspot" reduction.
Defect Tracking	All Stages	Low to Moderate	Moderate	Moderate	High	Moderate	Moderate	Moderate	You must DO it!

A Problem of Economics

Industry Average (from around '86) - 30-85 defects/KLOC
Extensive Testing - .5-3 defects/KLOC

With large systems, it is almost always true that more testing will find more defects. The question is not whether all the defects have been found, but whether the cost of discovering the remaining defects can be justified. Hence, strategies for testing should be adopted that optimize the effort expended.

By connecting metrics from the testing effort with software reliabilty models, it is possible to develop a meaningful strategy for answering the when to stop question.

Unit Testing

interface
local data structures
boundary conditions
independent paths
error handling paths

Integration Testing

top-down integration
bottom-up integration
regression testing

System Testing Strategies

Recovery Testing

Some systems must be capable of encountering a fault and recovering, then resume processing within a specified time. Recovery testing is a system test that forces the software to fail in ways so an assessment can be made regarding recovery. For automatic recovery, the evaluators check reinitialization, data recovery, and restart. For people-based recovery, the time to repair is sometimes the measure

Security Testing

Tests to verify that protection mechanisms actually protect the system from improper access. The tester attempts to penetrate the system defenses using whatever means.

Stress Testing

Tests are designed to confront the program with abnormal situations. These tests executes the system in a manner that demands resources in abnormal quantity, frequency, and volume.

Performance Testing

Designs test cases to test the run-time performance of the software within the context of an integrated system. This type of testing is usually intended for a completely integrated system, though some performance issues related to individual modules can be assessed prior to full system performance.

Usage Testing

This type of testing attempts to provide some confidence that the typical uses the software will encounter does not have flaws. A usage probability distribution is specified. This specifies possible usage patterns and scenarios, including erroneous and unexpected usage. Test cases are derived from the distribution, such that every test represents actual usage.

Usability Testing

Go To Lecture [Outline] [Overview]

Go To [Course Outline]