Annotated Bibliography

Code Reading and Program Comprehension

Sources A - H I - P Q - Z

Abstract: Traceability ensures that software artifacts of subsequent phases of the development cycle are consistent. Few works have so far addressed the problem of automatically recovering traceability links between object-oriented (OO) design and code entities. Such a recovery process is required whenever there is no explicit support of traceability from the development process. The recovered information can drive the evolution of the available design so that it corresponds to the code, thus providing a still useful and updated high-level view of the system.

Automatic recovery of traceability links can be achieved by determining the similarity of paired elements from design and code. The choice of the properties involved in the similarity computation is crucial for the success of the recovery process. In fact, design and code objects are complex artifacts with several properties attached. The basic anchors of the recovered traceability links should be chosen as those properties (or property combinations) which are expected to be maintained during the transformation of design into code. This may depend on specific practices and/or the development environment, which should therefore be properly accounted for.

In this paper different categories of basic properties of design and code entities will be analyzed with respect to the contribution they give to traceability recovery. Several industrial software components will be employed as a benchmark on which the performances of the alternatives are measured.

Comments D&N: The authors argue that the emphasis on writing, rather than on reading programs has caused the visual (as opposed to logical) aspects of programs to be ignored. Attempts to program in a more visual way (so-called "visual programming") have not been successful, so the authors were led to examine program visualization. In particular, they have applied graphics design principles and techniques to the visual display of C programs. The result is enlightening and striking. Their programs use multiple typefaces and font sizes, elaborate spacing conventions, shading, color, and the like. Similar treatments could be given programs written in other languages. The authors justify their design choices and provide useful background information on program comprehension studies and the elements of typography.

Comments D&N:This book is similar in spirit to the work of Oman and Cook [Oman90a, Oman90b], though it is more radical in its suggestions. The book is most helpful in suggesting possibilities likely to be overlooked because we have accepted the technological limitations of the past as givens. Persistence and familiarity with C are prerequisites to getting through this substantial book.

Abstract: Many tools have been built to analyze source code. Most of these tools do not adequately support reengineering activities because they do not allow a software engineer to simultaneously perform queries about both the existing and the desired source structure.

This paper introduces the conceptual module approach that overcomes this limitation. A conceptual module is a set of lines of source that are treated as a logical unit. We show how the approach simplifies the gathering of source information for reengineering tasks, and describe how a tool to support the approach was built as a front-end to existing source analysis tools.

Abstract: We consider reading techniques a fundamental means of achieving high quality software. Due tothe lack of research in this area, we are experimenting with the application and comparison ofvarious reading techniques. This paper deals with our experiences with a family of readingtechniques known as Perspective-Based Reading (PBR), and its application to requirementsdocuments. The goal of PBR is to provide operational scenarios where members of a review team read a document from a particular perspective, e.g., tester, developer, user. Our assumptionis that the combination of different perspectives provides better coverage of the document, i.e., uncovers a wider range of defects, than the same number of readers using their usual technique.

To test the effectiveness of PBR, we conducted a controlled experiment with professionalsoftware developers from the National Aeronautics and Space Administration / Goddard SpaceFlight Center (NASA/GSFC) Software Engineering Laboratory (SEL). The subjects read twotypes of documents, one generic in nature and the other from the NASA domain, using tworeading techniques, a PBR technique and their usual technique. The results from these experiments, as well as the experimental design, are presented and analyzed. Teams applyingPBR are shown to achieve significantly better coverage of documents than teams that do notapply PBR.

We thoroughly discuss the threats to validity so that external replications can benefit from thelessons learned and improve the experimental design if the constraints are different from thoseposed by subjects borrowed from a development organization.

Abstract: This paper reports on an experiment in trying to understand an unfamiliar program of some complexity and to record the authors' understanding of it. The goal was to simulate a practicing programmer in a program maintenance environment using the techniques of program design adapted to program understanding and documentation; that is, given a program, a specification and correctness proof were developed for the program. The approach points out the value of correctness proof ideas in guiding the discovery process. Toward this end, a variety of techniques were used: direct cognition for smaller parts, discovering and verifying loop invariants for larger program parts, and functions determined by additional analysis for larger program parts. An indeterminate bounded variable was introduced into the program documentation to summarize the effect of several program variables and simplify the proof of correctness.

Comments D&N: The authors take a modest FORTRAN subroutine for finding roots of a function (ZEROIN, which is fewer than 150 lines long including comments) and reverse engineer it to produce a specification and correctness proof, documentation sufficient to answer several questions posed about the code and, presumably, adequate to meet future maintenance needs. This approach is consistent with the authors' belief that code and a correctness proof should be developed from a specification in the first place.

The code is first restructured into one composed of prime programs (see [Linger79]), a process requiring the duplication of several lines of code of the unstructured original. Hypotheses are generated about the resulting components, which are then confirmed by proving theorems. A table of data references is used to provide preliminary insights into the semantics of the program. Results of individual theorems are combined until statements can be proved about the entire subroutine.

This paper demonstrates something of a "brute force" approach to determining what a program does and writing down the findings. This is more important than it sounds; the goal of generating a correctness proof provides structure to the discovery process of learning what the program does. The strategy this imposes on the reader is certainly used by programmers, though in limited circumstances and with less formality. The authors are more prescriptive than descriptive, however, and are not suggesting that programmers actually follow their procedure.

It is too bad the authors offer only pages and pages of proofs as documentation and do not suggest what comments they might actually insert into the restructured code. It should be kept in mind, too, that the techniques demonstrated cannot capture certain pragmatic information about how the program was intended to be used if this information is not explicitly represented in the program. Teachers and students should read this paper. It demonstrates well how we can reason about code and is likely to provoke heated discussion about how much work is enough when reading programs.

Comments D&N: One of Bentley's "Programming Pearls" columns.

A nice description of Knuth's WEB system and his scheme of "literate programming." The idea here is to free programming from the arbitrary restrictions of programming languages and to produce essay-like texts for human consumption. A source file is prepared with ordinary English text, programming language statements-Knuth's system is Pascal-based -and imbedded TeX formatting commands. Compilers WEAVE and TANGLE produce, on the one hand, a human-readable program with commentary and, on the other, a thoroughly unreadable program for machine use. The system allows the programmer to introduce and annotate programming language statements in logical order, rather than some order dictated by a language standard. A two-page example program is included as a sidebar, and it communicates well the spirit of Knuth's programming method. No programmer should miss this column. Even if literate programming does not seem to you the wave of the future, Knuth's style of exposition is exceedingly thought-provoking. (See also [Bentley86b].)

Another "Programming Pearls" column.

Comments D&N: This column is a follow-up to [Bentley86a]. It contains a nearly 8-page literate program by Knuth, followed by a review (literary criticism, if you will) by Doug McIlroy. This is not essential reading, but it is very interesting-for its literate program, programming criticism, and use of a novel data structure.

Abstract: In the computer industry, expert programmers must often relearn parts of their craft as they retool themselves to new computer languages, programming environments, software frameworks and systems. Our study of consulting interactions between these apprentices and experts has given insights into this collaborative work practice and into the knowledge gaps of programmers in a new environment.

In this paper we characterize the apprenticeship interactions we observed, the skills experts use in collaborative problem solving, the hard-to-find information they emphasize, and the tutoring skills they exhibit. The observations also indirectly suggest the multi-faceted knowledge required for real-life programming expertise, and the knowledge and skills that make experts so much more effective in their daily work.

Comments UMaD: This article provides some insights into the differences between experts and non-experts. Notice the absence of the word novice. The non-experts were probably well-trained professionals but without the specific domain and/or system specific knowledge the experts had. One of the outcomes of this study is the discussion of the types of knowledge, and differences therein, that differentiates experts from these non-experts and how this may factor into productivity differences. Part of insights this work provides is in the interaction of the expert with the non-expert in mentoring situations. This illuminates some of the knowledge differences as a difference in kind. The expert has more refined and useful metacognitive strategies that seem to make navigating uncertain situations less stressful.

Abstract: This study compares comprehension of Lit style literate programs with that of traditional modular programs with both internal and external documentation. Literate programming (Knuth, 1984)* enhances a computer program by incorporating program text into a comprehensive design document. Although not previously well defined, we believe Knuth's concept has great intuitive appeal, fits in well with a multi-disciplinary approach to automating portions of the software engineering process, and can be adapted easily to the incorporation of empirically derived principles of program comprehension. The Lit system developed by Chris Bertholf employs many of Knuth's principles for literate style programs as well as several others; the program text is incorporated into a comprehensive design document which uses typographic cues and a book style presentation paradigm. A program description and information about design history, the task domain, and implementation are included in the program document. The table of contents provides information about the overall structure of the program. In addition, algorithms are documented in pseudo-code and documentation of anticipated modifications is included. Extensive documentation of the usage of variables, procedures, and functions is also included. Does this increased amount of documentation and the unique presentation format hinder or facilitate program comprehension? This study compared the comprehension results of 20 novice programmers randomly divided into two groups and given either a traditional modular FORTRAN program or an equivalent Lit style literate program to modify. Subjects performed the task of completing an incomplete program; all program modifications were made on paper, thus syntax errors were expected. The elapsed time to produce a solution was recorded, and several measures of comprehension were collected and analyzed. Completed programs were judged as completely correct, functionally correct with syntax errors, or incorrect. The overall result was that subjects given the literate programs found a solution more often than did subjects using the traditional modular programs. None of the subjects given the modular programs were able to produce even functionally correct solutions. In addition, none of the subjects given Lit style literate programs modified sections of code that were unrelated to the modification specification while all of the subjects given traditional modular programs modified sections of code which were unrelated to the modification specification. Similar results have also been obtained with advanced programmers in another related study. Although this study did not attempt to isolate the factors which aided in comprehension, it did show that the Lit style programs are useful for program maintenance tasks. Future research in this area should concentrate on isolating the factors that produced such a marked distinction in performance between the Lit style literate program group and the traditional program group.

Abstract: The authors discuss empirical research about the cognitive structures and strategies used by programmers during fault location. Empirical evidence indicates the cognitive processes involved in fault detection consist of a comprehension process and a fault location process. The two processes are distinct and separate. The comprehension process is extremely important and was found to be superior in experts due to the semantic encoding they utilize. The semantic representations used by experts consist of abstract hierarchies based on functional meaning. Fault location is less important and usually takes the form of hand simulation or causal reasoning. The fault locating strategies used by experts and novices were similar. The better debugging performance by experts is due to their superior abilities at comprehension. Research indicates that the semantic organizations used by experts can be successfully taught to novices and used by them to improve performance. The authors also examine two methods, slicing and team reviews, which seek to improve the debugging process. Each was found to affect comprehension and fault location differently. A review of slicing research revealed that it is performed during the fault location process, and does not apply to the comprehension process as some believe. Automating slicing was found to be a technique with potential benefits for debugging. The survey of the team dynamics during inspections and other reviews found them to be effective by enhancing the comprehension process, by improving fault location, and by providing more than one chance to catch each error.

Abstract:

Abstract: A number of claims have been made by the developers of program design methodologies, including the claim that the code produced by following the methodologies will be more understandable and more easily maintained than code produced in other ways. However, there has been little empirical research to test these claims. In this study, student and professional programmers were asked to make either simple or complex modifications to programs that had been generated using each of three different program structures. Data on the programmers' modification performance, cognitive representations formed of the programs and subjective reactions to the programs suggested that problem structure (as created by the different methodologies), problem content, complexity of modification, and programmer experience all play a crucial role in determining performance and the representation formed.

Abstract: This research examined program design methodologies which claim to improve the design process by providing strategies to programmers for structuring solutions to computer problems. In this experiment, professional programmers were provided with the specifications for each of three non-trivial problems and asked to produce pseudo-code for each specification according to the principles of a particular design methodology. The measures collected were the time to design and code, percent complete, and complexity, as measured by several metrics. These data were used to develop profiles of the solutions produced by different methodologies and to develop comparisons among the various methodologies. These differences are discussed in light of their impact on the comprehensibility, reliability, and maintainability of the programs produced.

Abstract: A theory is presented of how a programmer goes about understanding a program. The theory is based on a representation of knowledge about programs as a succession of knowledge domains which bridge between the problem domain and the executing program. A hypothesis and verify process is used by programmers to reconstruct these domains when they seek to understand a program.

The theory is useful in several ways in software engineering: It makes accurate predictions about the effectiveness of documentation; it can be used to systematically evaluate and critique other claims about documentation, and it may even be a useful guideline to a programmer in actually constructing documentation.

Comments D&N: In this paper, Brooks sets forth a model of program comprehension and relates it to the nature of programs and their documentation. His basic ideas reappear in [Brooks82] and in [Brooks83], where the process of comprehending a program receives greater emphasis. The author argues strongly against the notion put forth by Kernighan and Plauger that "the only reliable documentation of a computer program is the code itself" [Kernighan74]. He offers intuitively appealing arguments for his theory, as well a very brief description of a supporting experiment.

This paper is easy reading and likely to elicit a few "ahas" from anyone who takes programming seriously. The most assailable part of Brooks's theory, his severely top-down description of how people actually read programs, receives relatively little attention here.

Comments D&N: Brooks describes his theory of program comprehension and uses that theory to draw inferences about documentation. Brooks's most striking conclusion is that it is as important to document the problem domain as it is to document the program itself. He concludes that different programming languages require different kinds of documentation, and he asserts that multiple forms of documentation are beneficial when they convey different kinds of information. The description of Brooks's theory is better developed in [Brooks83], although this paper serves as a brief, readable introduction to it. This paper demonstrates the utility of the theory, although readers may find themselves wanting more details. Teachers may want to ask students to build on the inferences presented in the paper.

Abstract: A sufficiency theory is presented of the process by which a computer programmer attempts to comprehend a program. The theory is intended to explain four sources of variation in behavior on this task: the kind of computation the program performs, the intrinsic properties of the program text, such as language and documentation, the reason for which the documentation is needed, and differences among the individuals performing the task. The starting point for the theory is an analysis of the structure of the knowledge required when a program is comprehended which views the knowledge as being organized into distinct domains which bridge between the original problem and the final program. The program comprehension process is one of reconstructing knowledge about these domains and the relationship among them. This reconstruction process is theorized to be a top-down, hypothesis driven one in which an initially vague and general hypothesis is refined and elaborated based on information extracted from the program text and other documentation.

Comments D&N: This extended treatment of Brooks's ideas about program comprehension is intended to provide an adequate descriptive model of how programmers understand programs. According to this model, when one understands a program, one has constructed a mental model of successive knowledge domains bridging from the problem domain to the domain of the program in execution. Each of these domains consists of objects, properties, relations, and operations. The succession of domains may include the problem domain, the domain of a mathematical model of the problem, the algorithm domain, the programming language domain, etc. One must also understand the relationships that exist between adjacent domains. The process of reading a program to understand it is one of constructing a model of this sort or, depending upon one's reading objectives, constructing a part of one. According to Brooks, this process is largely top-down: the reader generates hypotheses about the program, which he then attempts to verify from the code and whatever other documentation is available. This verification task is aided by beacons, code features characteristic of recurring structures or operations. Hypotheses are generated hierarchically until hypotheses can be bound to particular code segments. In this process, hypotheses are frequently revised or replaced by more credible ones.

The paper concludes with a discussion of factors affecting comprehension, including the nature of the problem, available documentation, programmer knowledge (both of programming and of the problem domain), reading goals, and reading strategy.

Although not overtly based on empirical studies, Brooks's model seems both sensible and serviceable. The model is somewhat dogmatic about programs being read top-down, however; and although the author acknowledges the bottom-up strategies illustrated in [Basili82] and elsewhere, he dismisses them as less powerful and less important.

This is an important, well-written paper. It is particularly concerned with the process of comprehension, however, and the reader wishing to better understand what it means to understand a program should look at [Brooks78]. It may be worthwhile to ask students if they believe people actually read programs as Brooks describes.

Abstract This study analyzes object-oriented (OO) program comprehension by experts and novices. We examine the effect of expertise in three dimensions of comprehension strategies: the scope of the comprehension, the top-down versus bottom-up direction of the processes, and the guidance of the comprehension activity. Overall, subjects were similar in the scope of their comprehension, although the experts tended to consult more files. We found strong evidence of top-down, inference-driven behaviors, as well as multiple guidance in expert comprehension. We also found evidence of execution-based guidance and less use of top-down processes in novice comprehension. Guidance by inheritance and composition relationships in the OO program was not dominant, but nevertheless played a substantial role in expert program comprehension. However, these static relationships more closely tied to the OO nature of the program were exploited poorly by novices. To conclude, these results are discussed with respect to the literature on procedural program comprehension.

Abstract: Program understanding relates a computer program to the goals and requirements it is designed to accomplish. Understanding techniques that rely only on source code analysis are limited in their ability to derive this relationship. Application-domain analysis is another source of information that can aid program understanding by guiding the source analysis and providing structure to its results. This paper describes the application of a domain-based program understanding process, Synchronized Refinement, to the problem of reverse engineering the Mosaic World Wide Web browser software. It discusses the domain analysis undertaken, the corresponding source code analysis we plan to perform, and the strengths and limitations of available automated tools.

Abstract: This paper is concerned with the units of knowledge used in understanding programs. A pilot study was conducted wherein a short, but complex, program was examined looking for "knowledge atoms," the units from which program understanding is built. The resulting atoms were categorized along three orthogonal axes of knowledge type, design decision used, and the type of analysis required to uncover the atom. The results are discussed relative to several approaches to program understanding taken from the research literature.

Abstract: Technology to support system comprehension tends to reflect either a Òbottom-upÓ or Òtop-downÓ approach. Bot-tom- up approaches attempt to derive system models from source code, while top-down approaches attempt to map abstract ÒdomainÓ concepts to concrete system artifacts. While both approaches have merit in theory, in practice the top-down approach has not yielded scalable, cost-effective technology. One problem with the top-down approach is that it is very expensive to develop domain models, and it is difficult to develop models that are sufficiently general to be applied to multiple systems (and hence amortize the development cost). This paper describes the Gadfly, an approach for developing narrowly-focused, reusable domain models that can be integrated and (re)used to aid in the process of top-down system comprehension.

Abstract: Software maintenance represents the largest cost element in the life of a software system, and the process of understanding the software utilizes 50 percent of the time spent on software maintenance. Thus there is a need for tools to aid the program understanding task. The tool described in this paper-Program UNderstanding Support environment (PUNS)-provides the needed environment. Here the program understanding task is supported with multiple views of the program and a simple strategy for moving between views and exploring a particular view in depth. PUNS consists of a repository component that loads and manages a repository of information about the program to be understood and a user interface component that presents the information in the repository, utilizing graphics to emphasize the relationships and allowing the user to move among the pieces of information quickly and easily.

Comments D&N: This paper is a thorough description of the PUNS system developed at IBM. PUNS exists as a research prototype for IBM System/370 Assembly Language. The PUNS repository resides on an System/370 30XX computer, while the user interface can reside on a workstation running Microsoft Windows. PUNS supports the program understanding task by organizing and presenting the program from many different viewpoints: a call graph for a collection of procedures, a control flow graph for a single procedure, a graph that relates a file to the procedures that use it, a data flow graph, a use-definition chain for a variable. The tool uses static analysis techniques to detect low-level relationships that exist within the program. It consolidates and organizes these relationships and presents them in a user-friendly environment. Prior to using PUNS, the user is required to structure a data-base for the particular program to be investigated.

Several extensions are under way to make PUNS a more useful tool. Perhaps the most important one reported in the paper is called a dynamic information updating facility. As it stands now, PUNS requires the repository to be established before a user session can begin. However, there is much to be gained by allowing the user to update information during the session, since there are relationships that cannot be determined using the static analysis employed when the repository is set up. Of course, as is discussed in the paper, allowing for dynamic additions to the repository raises the question of accuracy of information retrieved from the system. Attempts to extend PUNS to operate on higher-level languages are not reported. PUNS provides the user with a powerful tool to aid in understanding a program. Although it is limited to assembly language programs, it suggests the kind of tool that may one day become commonplace for use with high-level languages.

Abstract: In the Program Understanding Project at IBM's Research Division, work began in late 1986 on tools which could help programmers in two key areas: static analysis (reading the code) and dynamic analysis (running the code). The work is reported in the companion papers by Cleveland [Cleveland89] and by Pazel [Pazel89] in this issue. The history and background which motivated and which led to the start of this research on tools to assist programmers in understanding code is reported here.

Comments D&N: The author makes a case for the study and development of program reading skills, both in the workplace and in the classroom. The paper quotes a large number of authors who, for the last two decades, have made predictions (some true, others not quite so) on software development and maintenance. Certainly aging software systems are an integral part of today's (and tomorrow's) software. Corbi stresses the point that program readers will be needed in the 1990s, but that, unfortunately, program reading instruction is missing from most computer science curricula. The paper suggests approaches to maintenance of existing systems and discusses current tools and their limitations. Finally, the author gives his own view on software maintenance for the Ô90s.

This paper is recommended as a reading assignment for upperclass undergraduates. Furthermore, anyone interested in program reading, both as a necessary professional skill and as an activity within computer-related curricula, will find this paper most helpful. Instructors can find plenty of motivational statements to share with colleagues and students alike. The paper contains a substantial bibliography.

Abstract: This study examines the direction and scope of comprehension-related activities of professional programmers carrying out several comprehension and maintenance activities over time. Procedural and object-oriented (OO) programmers studied a program and subsequently performed modifications during two sessions. Results showed that the OO programmers tended to use a strongly top-down approach to program understanding during an early phase of study of the program but increasingly used a bottom-up approach during the maintenance tasks. The procedural programmers used a more bottom-up orientation throughout all activities. The scope of the activities was greater for the procedural than for the OO programmers. However, regardless of paradigm, the programmers over time built a broad, rather than a localized, view of the program.

Abstract: This study examines the mental representations formed during program comprehension and maintenance by procedural and object-oriented (OO) experts. The programmer's mental representation reflects comprehension of a program and guides tasks carried out on the program, such as debugging and modifications. The goals of the research were three-fold: (1) to determine if and how the mental representations of procedural and OO experts differ, (2) to investigate the initial mental representation formed while comprehending a moderately large program and 3) to examine the evolution of the mental representations of procedural and OO experts over time as they carried out several modifications of the same program. Fifteen expert procedural programmers and 15 expert object-oriented programmers studied and then performed three program modifications during two sessions which were 7-10 days apart. They answered two question sets designed to elicit the categories of knowledge present in their mental representations at different times. The initial mental representation of the OO participants was dominated byÊproblem domain-based knowledge and contained relatively little detailed program information. The procedural participants' initial representation was more balanced, containing domain-based knowledge and also substantial program detail. After performing the modifications, the procedural participants' representations remained essentially the same, while those of the OO participants became more balance with respect to the program and domain elements. The results suggest that, regardless of paradigm, expert programmers build a mixed mental representation of a large prram, which includes detailed program knowledge as well as domain-based knowledge.

Abstract: Comprehension of computer programs involves identifying important program parts and inferring relationships between them. The ability to comprehend a computer program is a skill that begins its development in the novice programmer and reaches maturity in the expert programmer. This research examined the beginning of this process, that of comprehension of computer programs by novice programmers. The mental representations of the program text that novice form, which indicate the comprehension strategies being used, were examined. In the first study, 80 novice programmers were tested on their comprehension of short program segments. The results suggested that novices form detailed, concrete mental representations of the program text, supporting work that has previously been done with novice comprehension. Their mental representations were primarily procedural in nature, with little or no modeling using real-world referents. In a second study, the upper and lower quartile comprehenders from Study 1 were tested on their comprehension of a longer program. Results supported the conclusions from study 1 in that the novices tended towards detailed representations of the program text with little real-world reference. However, the comprehension strategies used by high comprehenders differed substantially from those used by low comprehenders. Results indicated that the more advanced novices were using more abstract concepts in their representations, although their abstractions were detailed in nature.

Comments D&N: The authors report on a study that examined eye movements of programmers reading a Pascal version of a binary search, as well as their eye movements while studying slides illustrating the operation of the algorithm. Nineteen low- and high-experience programmers, all but one a student, served as subjects. Subjects were given a pretest on the binary search, and cloze tests were used to measure comprehension. Fixation times and number of fixations were gathered. Most of the results reported are qualitative, however, and the experiment seems not to have addressed any particular hypothesis.

The authors claim support for the immediacy theory (that text is processed immediately, as opposed to being stored in a mental buffer for later cognitive processing), but they report no evidence for major systematic differences in reading strategy between novices and experts. High-experience subjects devoted more attention to meaningful areas of the code, however. Reading strategies differed in the relative attention given to comments and code, in the number of passes over the text, and in the degree to which subjects compared program elements to one another rather than reading left-to-right and top-to-bottom.

This paper is most interesting for its graphic analysis of the data to discover patterns in reading strategies. Although generalizations valid for really experienced programmers and for realistic blocks of code are not readily apparent, the paper nonetheless suggests interesting future work.

The paper is easy reading, though it is somewhat vague about details of the experimental protocol.

Abstract:

Abstract: A number of authors have proposed that the 'programming plan' be regarded as the major characteristic of programming expertise. Such plans are thought to represent the programmer's knowledge of generic and stereotypic fragments of programs that correspond to specific task goals or sub-goals. A range of empirical studies have been undertaken in order to provide support for the notion of the programming plan and to establish the relationship between such plans and expertise. Most of these studies, however, have been concerned with what might be characterised as a theory of plans rather than with a theory of 'planning' in programming. Such studies have tended to examine only the static elements of plans -- attempting to show merely, for example, that plans are related in some way to expertise or to the notation of a particular language, rather than providing a means of looking at the way in which plans might be combined or refined. In addition they have neglected to examine the question of whether plans are implemented differently in different circumstances or with respect to different skill levels. This paper considers the results of two experimental studies which suggest that the relationship between programming plans and expertise is by no means straightforward. This work highlights the need to examine strategic differences in plan generation and comprehension that exist at different skill levels. In conclusion some tentative requirements are proposed for a theory of 'planning' in programming.

Abstract: The notion of the programming plan as a description of one of the main types of strategy employed in the comprehension of programs is now widely accepted to form an adequate basis for an account of programming knowledge. Such plans are thought to be used universally in all programming languages by expert programmers. Recent work, however, has questioned the psychological reality of such plans and has suggested that they may be artifacts of the particular programming language used and the structure that it imposes on the programmer via the constraints of certain features of its notation. This paper considers the results of two experimental studies that suggest that the development and use of programming plans is strongly tied to the particular learning experience of the programmer. It is argued that programming plans cannot be considered solely to be natural strategies that evolve independently of teaching nor as mere artifacts or static properties of a particular programming language. Rather, such plans can be seen to be related to the expression of design-related skills. This has a number of important implications for our understanding of the nature and development of programming plans, and in particular, it appears that the notion of the programming plan provides too limited a view to adequately and straightforwardly explain the differences between novice and the expert's programming performance.

Abstract: The notion of the programming plan has been proposed as a mechanism through which one can explain the nature of expertise in programming. Soloway and Ehrlich (1984) suggest that such expertise is characterized by the existence and use of programming plans. However, studies in other complex problem-solving domains, notably text editing, suggest that expertise is characterized not only by the possession of plan-related structures but also by the development of appropriate selection rules which govern the implementation of plans in appropriate situations (Card et al. 1980, Kay and Black 1984, 1986). This paper presents an experimental study which examines the role of programming plans in the context of skill development in programming. The results of this study suggest that plan-based structures cannot be used in isolation to explain novice/expert differences. Indeed, such structures appear to prevail at intermediate levels of skill. The major characteristic of expertise in programming would appear to be strongly related to the development of appropriate selection rules and to so-called program discourse rules. This in turn suggests that current views on the role of plan-based structures in expert programming performance are too limited in their conception to provide an adequate basis for a thorough analysis of the problem-solving activity in the programming domain.

Abstract: Much of the literature concerned with understanding the nature of programming skill has focused explicitly upon the declarative aspects of programmers' knowledge. This literature has sought to describe the nature of stereotypical programming knowledge structures and their organization. However, one major limitation of many of these knowledge-based theories is that they often fail to consider the way in which knowledge is used or applied. Another strand of literature is less well represented. This literature deals with the strategic elements of programming skill and is directed towards an analysis of the strategies commonly employed by programmers in the generation and the comprehension of programs. In this paper an attempt is made to unify various analyses of programming strategy. This paper presents a review of the literature in this area, highlighting common themes and concerns, and proposes a model of strategy development which attempts to encompass the central findings of previous research in this area. It is suggested that many studies of programming strategy are descriptive and fail to explain why strategies take the form they do or to explain the typical strategy shifts which are observed during the transitions between different levels of skill. This paper suggests that what is needed is an explanation of programming skill that integrates ideas about knowledge representation with a strategic model, enabling one to make predictions about how changes in knowledge representation might give rise to particular strategies and to the strategy changes associated with developing expertise. This paper concludes by making a number of brief suggestions about the possible nature of this model and its implications for theories of programming expertise.

Abstract: This paper presents empirical evidence for differences in the nature of problem solver's information externalisation strategies. Two experiments concerned with programming behaviour are reported which suggest that experts tend to rely much more upon the use of external memory sources in situations where the device they use to construct the program hinders the utilisation of a display in the service of performance. Experts and novices also appear to externalise different kinds of information during problem solving. Hence, experts tend to externalise low level information, mainly to aid simulation, whereas novices develop higher level representations which might be characterised as transformations or re-representations of the problem state. Moreover in the case of experts, the nature of externalised information appears to depend upon whether they are generating a program as opposed to comprehending it. These results provide support for a display-based view of problem solving. Moreover these studies address strategic differences in the externalisation of information, which until now have remained unexplored in accounts of display-based behaviour. Finally, the paper suggests a number of implications for the design of tools intended to support the programming process and for systems aimed at teaching programming skills.

Abstract: The state of the art in psychological complexity measurement is currently at the same stage as weather forecasting was when early Europeans based their predictions on portents of change. Current direct measures of program characteristics such as operator and operand counts and control flow paths are not based on convincing indicators of complexity. This paper provides justification for using chunks as a basis for improved complexity measurement, describes approaches to identifying chunks, and proposes a chunk-based complexity measure.

Comments D&N:This paper focuses more on the uses of the abstraction operation of chunking as a means of measuring program complexity than on how to extract chunks from programs. The chunks Davis is concerned with are at the level of Letovsky's and Soloway's plans [Letovsky86b]. Some of the earliest studies of chunking examined chess players. Experiments show that master players can remember more than novices can from a quick scan of a chess board if the chess board represents a meaningful situation. If, however, the pieces are randomly arranged, non-master players do as well as chess masters. In the programming world, chunks can be thought of as patterns of statements that accomplish a particular task. For example, experienced programmers may recognize the familiar pattern of a sorting algorithm. Davis reports that the Raytheon Company found that about half the code in its inventory of COBOL programs was redundant, in the sense that similar code existed to perform essentially the same function.

The paper proposes two chunk-based complexity measurement models and reports on comprehension experiments aimed at validating proposed metrics. Davis points out that programmers often maintain the same piece of code over a long period of time. Comprehension experiments that present subjects with unfamiliar programs may therefore be less relevant to the maintenance task than it might at first appear.

The paper is recommended for students with some level of programming maturity. The results reported could be the basis of interesting class discussion in an advanced course.

Comments D&N: The authors review approaches to teaching the introductory programming course and conclude there is a need for a radically different approach, largely because students fail to grasp the nature of programs and of program development. Their solution is more analytical (as opposed to synthetic), consisting of four stages of instruction, in which students successively (1) experience programs as a user, (2) read and analyze programs and algorithms, (3) modify existing programs, and (4) design and implement new programs. Much of the discussion is concerned with the benefits of program reading.

This is a paper for teachers. The authors acknowledge that the lack of appropriate materials makes their approach difficult to implement.

Abstract: Evaluating program reading comprehension is a difficult task faced by both programming instructors and software psychologists. This paper offers a taxonomy of methods to measure comprehension and relates these techniques to a theory of comprehension. The classification should be useful for constructing test questions for both the classroom and laboratory. Deimel and Makoid present their classification of program reading comprehension questions that we have used in this report. The two-dimensional taxonomy is based on the Bloom taxonomy of educational objectives in the cognitive domain [Bloom56, Bloom71], and on Ruven Brooks's knowledge domains [Brooks83]. The authors present a list of question types by Bloom taxonomy level, a list refined in [Deimel85c].

Comments D&N: This paper is primarily for teachers, though it might be of interest to students needing to construct comprehension tests in experimental studies.

Abstract: A methodology for constructing program reading comprehension tests is discussed and illustrated. Emphasis is on multiple-choice tests used with realistic reading passages. Item writing employing a classification of question types developed by the authors and a program comprehension model developed by Ruven Brooks is recommended.

Comments D&N: The authors discuss constructing multiple-choice program reading comprehension tests within the framework described in [Deimel84]. The paper is illustrated with examples and provides a good deal of practical advice to computer science instructors unfamiliar with education literature. Most of the examples are taken from a reading passage and comprehension test reproduced completely in [Deimel85b].

This paper and the next are addressed to teachers, and may not be of much interest to students.

Abstract: Two experiments are described which compare the use of detailed comments placed either in line with the high-level language code or offset to the right. Given more than adequate time in Experiment 1 to read the program and answer comprehension questions, subjects given the two commenting treatments scored similarly on a comprehension test. When reading time was substantially reduced in Experiment 2, there were significant differences in comprehension between the commenting styles, favoring offset commenting. Further analysis revealed a significant time by comment placement interaction. Possible explanations and related questions are discussed.

Comments D&N: The commenting styles discussed and their possible effects on program readability are certainly of interest here, but perhaps of greater interest is the construction of the test by which program comprehension was measured. The test was constructed using the theory set forth in [Deimel84]. Most of the experimental materials are reproduced in the paper.

Teachers or experimenters needing to construct program comprehension tests will find this paper quite interesting.

Abstract: It is argued that program reading is an important programmer activity and that reading skill should be taught in programming courses. Possible teaching methods are suggested. The uses of program reading in test construction and as part of an overall teaching strategy is discussed. A classification of reading comprehension testing methods is provided in an appendix.

Comments D&N: This paper argues for the importance of program reading and contends that reading skills are not necessarily developed in students unless the students receive explicit instruction designed to develop these skills. The author contends that three components are needed to teach program reading-lecture, reading exercises, and program writing standards that are designed with the production of comprehensible programs in mind. He also suggests that program reading can be used both to teach and to evaluate general programming skills.

The appendix contains a revised version of the question classification introduced in [Deimel84]. This list is an improvement over the earlier one, although there are omissions and, perhaps, some misclassificationUniversity of Durham, DH1 3LE, UK.

This paper is primarily addressed to teachers.

Abstract: The ability to read and understand a computer program is a critical skill for the software developer, yet this skill is seldom developed in any systematic way in the education or training of software professionals. These materials discuss the importance of program reading, and review what is known about reading strategies and other factors affecting comprehension. These materials also include reading exercises for a modest Ada program and discuss how educators can structure additional exercises to enhance program reading skills.

Abstract: Studies of software comprehension often use short-term recall as a way to study comprehension. Experiments range from broad descriptions of program purpose to tests that require subject to 'fill in the gaps'. There is little doubt that human memory is a very important issue when it comes to understanding how software systems work. This paper addresses the topic of 'long term' comprehension, namely, the retention of programming knowledge over a period of months, years or even decades. Long-term retention of programming knowledge is thought to be an area within the psychology of programming literature that has received surprisingly little attention. To stimulate debate, methodological issues that may affect the study of long-term comprehension are detailed. Finally, a planned experiment based upon related psychology of programming studies is outlined.

Abstract: Large scale code testing can be made viable by determining and searching for the most probable faults in the system under examination. The results of an exploratory survey carried out to determine the common error factors for code written in C indicates that the program task and the programmer experience are important considerations. Using this information, testing can be directed efficiently towards the removal of prevalent coding faults. The results of the survey can also be useful in determining features of C which are likely to become the subject of corrective maintenance.

Abstract: To allow testers to know the types of faults they are looking for and to detect fault commonality and criticality, it is important to categorise code defects. The paper reviews testing techniques and taxonomies and considers fault clustering and isolation.

Abstract: The RECAST method (Reverse Engineering into CASE Technology) is a practical method that has been developed for reverse engineering COBOL systems. In applying the method the system maintainer derives a system representation which is equivalent to an SSADM logical system specification. The method has been developed for use in organisations with large COBOL legacy systems who have a requirement to redocument them in a well defined manner. This paper describes how the method's framework supports the maintainer during the reverse engineering process. This discussion is illustrated by reference to a case study carried out into the practical use of the method: where a "novice" maintainer was required to reverse engineer a small, commercial, COBOL system using RECAST.

Abstract: This paper discusses early experiences using two approaches to reverse engineering technology, Rigi and Software Refinery, to analyse the structure of product documentation written using BookMaster.

Abstract: In this paper, we study a synergy between object oriented design patterns and design methodologies. The design patterns describe design fragments, while the design methodologies describe the design process. We introduce design scenarios that employ design and transformation activities to introduce patterns. Design scenarios are pieces of design process that lead to design patterns. The example in the paper employs several patterns ("Composite", "Iterator", and "Adapter".) We discuss the benefits of this combined approach.

Abstract: In this paper, we describe the reengineering of a deterioratedobject-oriented industrial program written in C++.The main problem of the program was misplaced code,most often functions that were members of the wrongclass. In order to deal with this problem, we designed and implemented several restructuring tools and used them in specific reengineering scenarios. We also discuss how this set of tools could be enhanced in the future, and theimportance of restructuring for object-oriented software maintenance.

Abstract: This paper presents five abstact characteristics of the mental representation of comptuer programs: hierarchical structure, explicit mapping of code to goals, foundation on recognition of recurring patterns, connection of knowledge, and grounding in the program text. An experiment is reported in which expert and novice programmers studies a Pascal program for comprehension and then answered a series of questions about it designed to show these characteristics if they existed in the mental representations formed. Evidence for all of the abstract charcteristics was found in the mental representations of expert programmers. Novices' representations generally lacked the characteristics, but there was evidence that they had the beginnings, although poorly developed, of such characteristics.

Abstract: The paper describes a study that explores the relationship between program slicing and code understanding gained while debugging. The study consisted of an experiment that compared the program understanding abilities of two classes of debuggers: those who slice while debugging and those who do not. For debugging purposes, a slice can be thought of as a minimal subprogram of the original code that contains the program faults. Those who only examine statements within a slice for correctness are considered slicers; all others are considered non-slicers. Using accuracy of reconstruction as a measure of understanding; it was determined that slicers have a better understanding of the code after debugging.

Abstract: The paper describes a study that explored the relationship of program slicing to (1) code understanding gained while debugging, and to (2) a debugger's ability to localize the program fault area. The study included two experiments. The first experiment compared the program understanding abilities of two classes of debuggers: those who slice while debugging and those who do not. For debugging purposes, a slice can be thought of as a minimal subprogram of the original code that contains the program faults. Those who only examine statements within a slice for correctness are considered slicers; all others are considered non-slicers. Using accuracy of subprogram construction as a measure of understanding, it was determined that slicers have a better understanding of the code after debugging. The second experiment compared debugger fault localization abilities before and after a training session on how to use slicing in debugging. Using time as a measure of ability, it was shown that slicing while debugging improves a debugger's ability to localize the program fault area.

Abstract: Beacons are defined as sets of key features that typically indicate the presence of a particular data structure or operation. It has been claimed that programmers use them to help comprehend an unfamiliar program. However, despite their importance, beacons are not well-defined and very few have been identified. The present study investigated the importance of procedure and variable names as beacons during comprehension of short Pascal procedures. Ninety-six college seniors and graduate computer science students studied versions of a binary search and sorting procedures for one minute each and then provided a written description of the functions of each procedure. It was found that both meaningful procedure and variable names served as beacons to high-level comprehension. However, a strong code beacon, like the swap operation in a sorting routine, was found to be relied on more than the procedure name for high-level comprehension.

Abstract: Signaling refers to the addition of non-content information to a text in order to emphasize certain ideas and/or clarify the organization. There is increased interest in using typographic signaling, such as boldface type, different font sizes and ruled lines, to format computer program source code listings. However, little evidence exists to show that this type of formatting makes programs easier to read and understand. The present study reports on a controlled experiment which investigated the effectiveness of typographic signaling, header comments before modules, and mnemonic module names as ways to aid comprehension. We found that while all three factors helped experienced programmers confirm hypotheses about a 913-line program, only the header comments and mnemonic names helped the programmers locate material in the program. We also found only modest improvements in comprehension when typographic signaling was used with header comments and mnemonic names.

Comments D&N: This paper describes how the facilities of the Smalltalk-80 environment support program reading, a particularly important function in what the author calls an "exploratory environment," in which much programming is accomplished by modifying and reusing existing application and system code. Goldberg describes the Smalltalk-80 system in terms of four levels (user interface, functionality, structure, and language/implementation) and lists important comprehension questions for each level.

This paper offers an unusual argument for needing to read programs, an argument tied quite directly to program writing. It suggests ways in which future environments may provide support for both activities.

Abstract: It is proposed that hypertext interfaces can address some of the problems associated with program comprehension and coding. The benefits may be particularly striking for object-oriented programming environments. We review the literature on hypertext code tools and examine the implications of recent research of cognitive aspects of programmers' tasks. Finally, we propose several user interface solutions based on ongoing research into programmers' tasks, information retrieval and hypertext.

Comments UMaD: The paper makes more claims that it delivers. The review of hypertext code tools is minimal. The recent research in cognitive aspects of programmer's tasks isn't connected very well to the solution space of hpertext. The author provides little support, other than other authors' anecdotes, to support the claim that OO program may be more important to support via hypertext tools.

Abstract: Two experiments investigated expert-0novice differences in debugging computer programs. Debugging was done on programs provided to the subject, and were run on a microcomputer. The programs were in LOGO in Exp..1 and Pascal in Exp. 2. Experts debugged more quickly and accurately, largely because they generated high quality hypotheses on the basis of less study of the code than novices. Further, novices frequently added bugs to the program during the course of tying to find the original one. At least for these simple programs, experts superior debugging performance seemed to be due primarily to their superior ability to comprehend the program.

Abstract: Program slicing is a technique for program simplification based upon the deletion of statements which cannot affect the values of a chosen set of variables. Because slicing extracts a subcomponent of the program concerned with some specific computation on a set of variables, it can be used to assist program comprehension, allowing a programmer to remodularise a program according to arbitrarily selected slicing criteria.

In this paper it is shown that the simplification power of slicing can be improved if the syntactic restriction to statement deletion is removed, allowing slices to be constructed using any simplifying transformation which preserves the effect of the original program upon the set of variables of interest. It is also shown that quasi static slicing, first proposed by Venkatesh (and defined here in a slightly more general form), is the most suitable slicing paradigm for program comprehension. The various forms of slice are formally defined, an algorithm, based upon transformation, symbolic execution and conventional slicing is introduced for computing syntactically unrestricted, quasi static slices. A worked example is used to show how this approach supports program comprehension by case analysis and simplification.

Abstract: Recently, the first two in a series of planned comprehension experiments were performed to measure the effect of the control structure diagram (CSD) on program comprehensibility. Upper- and lower-division computer science and software engineering students were asked to respond to questions regarding the structure and execution of one source code module of a public domain graphics library. The time taken for each response and the correctness of each response was recorded. Statistical analysis of the data collected from these two experiments revealed that the CSD was highly significant in enhancing the subjects' performance in this program comprehension task. The results of these initial experiments promise to shed light on fundamental questions regarding the effect of software visualizations on program comprehensibility.

Abstract: This research examined programmers' cognitive representations of software. In this study, student and professional programmers were asked to make either simple or complex modifications to the program that had been generated using each of three different design methodologies: in-line code functional decomposition, and a form of object-oriented design. The programmers' mental models of the programs they had studied were elicited and then scored in several different ways. The results suggest that problem structure, problem type and ease of modification may affect the mental models formed. Specifically, the data suggest that while the mental models of professional programmers were affected primarily by modification difficulties, the mental models of student programmers were primarily affected by the structure and content of the programs. Performance differences between the two groups of programmers were small because the experience variables which were most strongly related to performance were nearly equal in the two groups, and the experiences variables which were very different between the two groups were not related to performance. Across the two groups, the primary aspect of the mental model which was correlated with performance variables was the width and breadth of the mental model structure. The implications of the results for the application of program design methodologies are discussed.

Sources A - H I - P Q - Z

Comments should be sent to

This document
Created: March 8, 1996
by RLU

Modified: May 8, 2002