To frame our discussion, consider:
What is the role of measurement in software engineering?
What are the pros and cons of various measurement systems?
How has the role of measurement in software engineering changed in the last five years?
How reliable will a software system be once it is installed?
How much more testing should I do?
How many defects can I expect to find?
How much will the testing cost?
How easy will it be to maintain a system?
How long will it take to conduct the integration for the system?
Is the use of peer reviews effective?
The movement to measurement in software engineering is derived from the development of the basic sciences. When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind: it may be the beginning of knowledge, but you have scarcely in your thoughts advanced to the stage of science. Lord Kelvin
Measurement is the process by which numbers or symbols are assigned to attributes of entities in the world according to a set of clearly defined rules.
| Abstract World | Empirical World | ||
| Theory | Proposition | Hypothesis | Data Analysis |
| Concept | Definition | Operational Definition | Measurement |
Measurement captures information about entities through their attributes. We describe an entity by defining characteristics that are important to us and that help distinguish one entity from another. We want to describe entities via their attributes.
Entities are objects in the world. These may be animate or inanimate, or they may be events.
An attribute is a characteristic of an entity. These characteristics are features or properties of the object.
| Entity | Attribute |
|---|---|
| Program | SLOC |
| Time in Testing | |
| Defects Reported | |
| Coding Phase | Design Defects Reported |
| Time Devoted to Testing | |
| Number of Reviews Conducted | |
Characteristics of the product are measured in an effort to increase its quality.
Measurement is a mapping from the empirical, observable world to the formal, relational world. A measure is the number or symbol assigned by the mapping that characterizes the attribute.
Definition: Let A be a set of physical or empirical objects. Let B be a set of formal objects. A measure m is a one-to-one mapping from A to B.
Domain: the real world
Range: mathematical system
In the real world we tend to understand the entities we observe through comparing and contrasting them. This activity is guided not by the formalities of mathematics, rather by the informal nature of our perceptions. To improve our skills we develop guidelines that assist us in making our comparisons. These comparisons define a relation between objects. We informally say program A was harder than program B, or program C is larger than program D. Given two items we can use these to provide commentary on the programs. If we have defined our informal rules sufficiently we have defined an empirical relation, a relation that allows us to verify the relation based on observation, or experimentation, of the objects.
Let R be an empirical relation on a collection of objects A. This says that, when x,y elements of A, xRy implies the relation holds for x and y. In mapping to the formal, mathematical world we require measures for the attributes of the objects as well as a numerical relation R'. The representation condition then says if xRy (world) then m(x)R'm(y) (formal world).
When is one relation system better than another?
Are we guaranteed every empirical relation has a numerical relation?
How do we choose between different numerical relations?
A Measurement Scale is (M, R, R') where M is the measurement mapping, R is the empirical relation and numerical relation.
For every operation defined on
the physical objects there is a corresponding operation defined on
the measures such that the result of measuring the combined objects
is the same as performing the operation on the measures of the
individual objects. Systems of this type are referred to as scales.
| Nominal scales simply give numeric names to objects. Any numbering is as good as any other, so any one-to-one mapping function is admissible. |
| Ordinal scales assign numbers to objects in a particular order, any numbers that maintain the order are equally good. |
| Interval scales assign numbers to objects in such a way that the interval between two measure values is meaningful throughout the range of values. |
| Ratio scales assign values in such a way that the ratio of two measures is meaningful. |
| Absolute scales have only one way of measuring objects. |
dynamic measure - derived from observations of the execution of the software
direct measures - size, effort, schedule, and quality
indirect measures - measures derived from direct measures, e.g., productivity (amount/unit of time)
Function Point
Line of code is typically correlated with effort. Boehm, Walston-Felix, and Halstead all show Effort as a function of lines of code.
| Attribute Class | Describes and distinguishes |
| Type of labor | direct and indirect labor: labor costs that can be charged directly to the project or contract, and those that cannot |
| hour information | regular or overtime work, and salaried or hourly workers |
| employment class | regular company employees, whether full-time or part-time, and employees brought in to work on a specific project task, such as consultants and subcontractors |
| labor class | workers by the types of work they do: managers at various levels, analysts, designers, programmers, documentation specialists, support staff, etc. |
| activity | software development activities and maintenance activities |
| product-level functions | functions of software development, such as design, coding, testing, and documentation; organized by major functional element, by customer release, and by system |
| Attribute Class | Describes and distinguishes |
| problem status | points in the problem analysis and correction process: open or closed; recognized, evaluated, or resolved |
| problem type | software defect or other kind of problem (hardware, operating system, user mistake, operations mistake, new requirement, enhancement); for a software defect, whether it is a defect in requirements, design, code, operational document, test case, etc. |
| uniqueness
| new and unique defect, or a duplicate of another reported defect |
| criticality | degree of disruption to a user when the problem is encountered |
| urgency | degree of importance given to the evaluation, resolution, and closure of the problem |
| finding activity | the activity that uncovered the problem, such as synthesis, inspection, formal review, testing, customer use |
| finding mode | operational or non-operational environment where defect was found |
Mean Time Between Failures
Mean Time to Repair
Availability
Go To Lecture [Outline] [Overview]
Go To [Course Outline]
Gary Ford (1993) Lecture Notes on Engineering Measurement for Software Engineers. Software Engineering Institute, Carnegie Mellon University, CMU/SEI-93-EM-9.
Fenton, N.& S. L. Pfleeger (1996) Software Metrics: A Rigorous and Practical Approach. Second Edition. International Thomson Computer Press