McGill University

Software Process Papers

El Emam, K., N. Moukheiber, & N. H. Madhavji (1993) Empirical Evaluation of the G/Q/M Method. Proceedings of the 1993 CAS Conference (CASCON'93), Toronto, Ontario, October 1993.

Abstract:Improvement of the software process is a major concern for many organizations. A critical part of such an endeavor is the definition of metrics. Despite the importance of metric definition, there have been no evaluations of existing methods for achieving this. It is generally taken for granted that a method with wide acceptance is suitable. A review of metric definition methods identifies Basili's G/Q/M as one of the most widely used. This paper reports on an evaluation of the G/Q/M method. The evaluation is based on an actual application of the method in a process improvement effort. The resultant metrics (and instrument) are evaluated with respect to the following criteria: interpretability, validity, reliability, effectiveness, and transportability. The causes of problems found are identified. These causes are problems with the G/Q/M method itself. The evaluation indicates that the metrics resultant from the application of the G/Q/M method do not appropriately meet the above criteria, and research on evolving G/Q/M or developing alternative methods is emphasized.

Höltje, D., N. H. Madhavji, T. Bruckhaus, & W. Hong (1994) Eliciting Formal Models of Software Engineering Processes. Proceedings 1994 CAS Conference (CASCON'94), Toronto, Ontario, Canada, October 1994.

Abstract: One of the first steps in improving a process is to understand the process, for example, by building descriptive models of the process. This paper addresses the elicitation of formal process models from real-world projects, using the Elicit method and tool, developed at McGill, and Statemate. In addition, the paper illustrates how formal models can make processes visible and how they can uncover inconsistencies and incompleteness in the understanding and the documentation. The described method and tools have been used in several real-world software projects, and the paper describes some conclusions drawn from this experience. Example process models are shown, and a tool demonstration is included in the CD-ROM that is part of this publication.

Madhavji, N. H., D. Höltje, W. Hong, & T. Bruckhaus (1994) Elicit: A Method for Eliciting Process Models. Proceedings 3rd International Conference on Software Process, Reston, VA, October 1994, IEEE Computer Society Press.

Abstract: Eliciting process models from software projects is a first significant step towards process improvement. In this paper, we present a method called Elicit , for eliciting software process models from industrial software environments. What is significant about this method is that it has evolved from an intuitive state - the state that defines the immaturity of current elicitation methods - to a formally defined, repeatable, effective and quantified state. Over the last two years of its usage, the method has been used to elicit models from three industrial-scale processes: preliminary analysis, requirements engineering, and product planning and dependency management. The example given in the paper focuses on the requirements engineering process.

El Emam, K. & D. R. Goldenson (1995) SPICE: An Empiricist's Perspective. Proceedings of the Second International Software Engineering Standards Symposium, Montrèal, Canada, August 1995.

Abstract: The SPICE project aims to deliver an international standard for software process assessment by the end of 1996. As part of this project there is an empirical trials phase whose purpose is to ascertain the effectiveness of the prospective SPICE standard. Two of the objectives of the trials phase are: (a) to determine the extent to which SPICE-conformant assessments are repeatable (i.e., reliability), and (b) to determine the extent to which SPICE-conformant assessments are really measuring best software process practices (i.e., validity). This paper introduces the theoretical foundations for evaluating the reliability and validity of measurement, suggests some empirical research methods for investigating them in SPICE, and discusses the constraints and limitations of these methods within the context of the SPICE project.

El Emam, K., N. H. Madhavji & K. Toubache (1993) Empirically Driven Improvement Of Generic Process Models. Proceedings of the 8th International Software Process Workshop, Schloss Dagstuhl, Wadern, Germany, March 1993

Madhavji, N. H., J. Botsford, T. F. W. Bruckhaus & K. El Emam (1993) Quantitative Measurements based on Process and Context Models. Proceedings of the International Workshop on Experimental Software Engineering Issues, Schloss Dagstuhl, Wadern, Germany, September 1993. 

Abstract: Measurement of a software development process is an important way to understand and improve processes. In this paper we propose that measurement programs should be based on a solid understanding of the process and the process context. One way to achieve this type of understanding is to first build formal process and context models. It is pointed out that process and context models can be used to systematically adapt, improve, reuse and standardise measurement programs. We conclude by describing briefly the currently conducted case studies that led to the views expressed in this position statement.

Sherdil, K. & N. H. Madhavji (1994) Personal 'Progress Functions' in the Software Process.In Proceedings of the 9th International Software Process Workshop, pp. 117-121, IEEE Computer Society Press.

Introduction: An individual can expect continuous improvement in productivity as a consequence of a growing stock of knowledge and experience gained by repeatedly doing some task [5]. Organizations have used this type of progress behavior in making managerial decisions regarding cost estimating and budgeting, production and labor scheduling, product pricing, etc. [4][12]. While considerable research on this topic has been done in industrial and manufacturing sectors [22], we found little such emphasis in the software process field, which is one of the most emerging and widely used ÔIndustryÕ of the present times. Furthermore, such research has remained almost exclusively focused on outcomes, such as units of items produced, rather than processes [2].

Our objective is to study the progress functions for an individual in the domain of software process. A progress function is a mathematical form for representing the improvement in performance, typically for some production activity. In popular form, the progress function represents the percentage decline in cost or labor requirement as the cumulative production output increases by one unit [12][16]. In our research, we shall study three types of progress functions: (i) labor productivity, (ii) product-quality and (iii) individual personal-skills, of which the latter two types have not been studied specifically before.

Progress functions differ from the widely used term learning curves because they also incorporate a second-order Learning mechanism [10]. Whereas first-order or autonomous (labor) learning is the improvement due to the experience which a person gains by repeatedly doing some task, second-order or induced (organizational) learning is due to technological and training programs injected by the organization [3][17]. Although the distinction between these two is often blurred [17], we intend to analyze them separately also. Such a distinction is useful for making managerial decisions regarding initiating formal training programs and making engineering technology changes [3].

Our position is that as in other areas, individual progress also takes place in the field of software development. Furthermore, in order to obtain significant and continuous improvement, the first-order learning alone is not sufficient; management must also officiate a strong second-order training/technical-knowledge program. To defend our position, we are carrying out an empirical study, which is described below.

See Sherdil 96 for a better presentation and the results of the experiment.

Sherdil, K. & N. H. Madhavji (1996) Human-Oriented Improvement in the Software Process. In Proceedings of the 5th European Workshop on Software Process Technology, Nancy, France, October 1996.

Abstract: By doing any task repeatedly, individuals can usually improve continuously due to the experience gained (called autonomous first-order learning). In addition, they can improve due to the injection of software development technology by the organization (called second-order learning). Organizations have studied such learning curves to make decisions regarding cost estimation and budgeting, production and labor scheduling, product pricing, etc. Such progress behavior was studied in a laboratory setting in an experiment involving a sample of 12 student software developers, who completed one small-sized project every week for ten weeks. A within-subject, repeated-measure, time-series quasi-experimental design was used as the research method. This also included the Goal/Question/Metric (GQM) paradigm with some additional validation techniques from Social Sciences/MIS/Software Engineering. Statistical tests showed that on average, progress takes place at a rate of about 20%, with technology injection (i.e., second-order learning) amounting to 13% improvement over autonomous learning alone. Such a distinction is useful for making personal decisions in software development and managerial decisions regarding training programs and making engineering technology changes. The study was replicated, twice, with samples of size 30 and 12. The average progress rate for the 54 subjects (in the three studies) was 18.51%.

Goldenson, D. R., K. El Emam, J. Herbsleb, & C. Deephouse (1996) Empirical Studies of Software Process Assessment Methods. In T. P. Rout (ed.), Software Process Assessment and Improvement, published by Computational Mechanics, 1996.

Introduction: There are now many methods for assessing the maturity and capabilities of software engineering organizations. Assessment scores are being used in making the contract award decision by the U.S. Navy [Ru93] and Air Force [SK95], as well as in commercial organizations [MC96]. Furthermore, conformance to process standards such as ISO 9001, as determined during an audit, is a necessity for doing business in many European countries. Software process assessments are also an essential element of the self-improvement cycle for many organizations (e.g., see[Ba96][Dy95]).

There has been a relative dearth of empirical investigations of the core premises of most contemporary assessment methods and their underlying models. Software organizations were being required and/or pressured to conform to certain standards (e.g., to be at Level 3 on the CMM) without adequate empirical evidence supporting the assumptions made by these standards. At least partly because of this, a certain amount of skepticisim and uncertainty exists about the accuracy and usefulness of software process assessments, and improvements based on them (e.g., see [BM91][Ba94][Ba95][Jo95]). The software community needs to be more confident that assessment results accurately reflect the capabilities of organizations being assessed, not simply the idiosyncrasies of those doing the assessments. We need a solid basis to better understand assessment methods, evaluate their basic premises, and inform decisions about their use and improvement. Similarly, more evidence is needed to justify investment in process improvement programs following the assessments.

Despite criticisms on lack of evidence, by now, a good number of empirical studies in fact do exist. Hence, the objectives of this chapter are twofold: (a) to demonstrate the different approaches for empirically studying software process assessment methods and the impact of software process on subsequent performance, and (b) to summarize the results of some empirical studies of software process assessment methods that have been conducted to date. Following this introduction, the chapter consists of three sections. Validity issues are addressed in Section 2, which examines the extent to which assessment methods are really measuring best software engineering practices. Reliability issues are addressed in Section 3, which examines the extent to which assessment scores and profiles are repeatable and consistent. Illustrative examples are drawn from recent empirical work conducted by the authors, including studies on the CMM, and from field trials of the emerging SPICE standard. We conclude in Section 4 with directions for future research.