There is widespread agreement that a primary factor in producing quality software is the excellence of its designers [1][2][3]. The unsolved problem is to identify what characterizes proficiency in design and how SEE can promote the development of such skill. More attention has been paid to the identification of characteristics of good programs such as encapsulation, information hiding,and modularity. Such principles are useful and easily incorporated into curriculua, but are not sufficient to produce good designers because they do not focus on the act of designing [4][5].
Several different strategies to improve the quality of designers has been suggested. One approach is to identify top designers as early as possible, then nurture these individuals over time. Other strategies focus on helping designers to acquire the knowledge and skills they need. Freeman [4] suggests that the key to success of this strategy is for experts to identify the essential knowledge of design, and then to teach the distillate to students so that they can most efficiently acquire the knowledge they need. Such an approach assumes that what differentiates excellent designers from others is what they know. The imperative for SEE that follows from this strategy is to find better ways to present the information students need to know. A second, popular approach to helping students develop design skills is to provide an environment in which students can practice the skills they will use in the "real world" in as realistic a way as possible [6]. This typically results in some form of project or practicum. This approach acknowledges that the skills needed to be effective software engineers are not limited to declarative knowledge or to programming skills per se, but it does not identify what those skills may be. It assumes that practice is sufficient to develop these unanalyzed skills. The imperative for educators is to develop projects that simulate real-life situations. A third approach is to identify the processes needed to do excellent design and to teach those skills specifically [7][8]. To implement this educators must first understand what the processes of software designers are and then how to help students acquire those skills.
Research on the nature of expertise and its acquisition has matured sufficiently to provide a solid base on which to evaluate various approaches to SEE, and to identify paths for future action. In this paper we will first evaluate the approaches just described in terms of the research on expertise, then we will give a description of the nature of expertise and how instructional techniques can help students develop expertise. Finally, we will describe a number of recent instructional projects that exemplify the use of this knowledge to promote SEE.
Brooks [2] advocates identifying top designers as early as possible, then nurturing them over time. The expertise research suggests that this approach will not work because it assumes that early talent predicts later accomplishment. Ericsson et al. [9] presented convincing evidence from a variety of fields that the characteristics that distinguish outstanding from less skilled novices are not the same as those characteristics that distinguish the outstanding expert.
Although their review did not include software design or programming, it seems a reasonable conclusion here as well. What seems to distinguish outstanding computer science students is their facility at writing small-scale programs and their ability to learn content information for tests. Yet, what characterizes mature experts is their ability to plan, organize, and manage their development activity during the course of large, complex software projects, not their programming skill per se.. When designing, professionals use their programming knowledge that a function can be implemented, but do not focus on how to implement it [10]. Furthermore, others on the software team usually write the code; it is the latter who need the outstanding programming ability. Finally, it can not be assumed that outstanding programming skill develops into large-frame designing skills. Another problem with the early identification strategy is that it requires the existence of some reliable mechanism for rating designers, so those with superior skill can be singled out. Neither in the field of software engineering nor in the broader world of personnel evaluation have ability measures such as programming skill or IQ been found to be strongly predictive of job performance. The review by Ericsson et. al. [9], suggests only a weak connection between psychometric tests and occupational success. Measures of initial ability predict entry level performance, but do not predict long-term job success. Amount of experience is a stronger predictor of long-term effectiveness, which is consistent with the notion that expertise is a result of intensive immersion in a field rather than pre-existing talent.
Other approaches to improving SEE focus on what needs to be learned by students to become effective designers. They disagree in their assessment of what SEE is failing to provide students. The most common approach, exemplified by Freeman [4], is to claim that we need to do a better job teaching what is known about internal design (the implementation of program functions). Although it is obviously important to undertand this arena, it is not sufficient. As discussed earlier, software engineers need a wide variety of skills beyond programming.
The realization that designing software requires something beyond "book learning" has driven most of the other approaches to SEE. The project has become standard fare in innovative SEE programs. Behind these project endeavors is the situated cognition principle (by which we mean learning within an authentic context). Students or student groups are given "real" or realistic problems, possibly having actual customers. In particular, the focus has been to identify situations where students would encounter the myriad problems of progamming-in-the-large. Instructionally the task has been to construct problems that require that the students 1) design something, 2) make decisions regarding options and/or resources, and 3) consider real-world and technical issues. If students engage in this type of development context, they should gain an appreciation of the processes of large-scale software development. Typically, however, instructors focus on the experiences with little regard to what is actually learned from those experiences (a criticism leveled at design education in traditional engineering by Dixon [12]).
Engaging in large-scale projects may indeed introduce students to the realities of the software engineering industry, but such an approach used as the only practice-oriented part of a curriculum is doomed to failure for three reasons.
First, it is predicated on the assumption that experience alone is sufficient to develop expertise. The expertise literature is very clear in this regard. Experience alone is a weak predictor of performance. Practice alone is sufficient to develop a certain level of competence, but, for most people, will result in an asymptotic level below that of virtuoso. To develop beyond this asymptotic level, deliberate practice techniques are necessary. Deliberate practice techniques, such as strategies for remembering things, muscle-specific training and drills in sports, and exercises to improve specific skills in music, have been shown to provide the tools novices and intermediates need to get beyond the practice-produced asymptote [13]. Although the effect of practice alone versus deliberate practice has not been studied in software engineering, it seems likely that deliberate practice techniques would improve outcomes here as well..
Second, the expertise research indicates that novices' understanding is tied to the context in which they learn something. Thus they are able to apply what they have learned only to very similar situations. In contrast, experts are able to apply what they know to a broad range of problems [14]. Experts evidently developed abstracted (context-independent) representations as a result of repeated experiences in different contexts. . Hence, a single project is an insufficient basis on which to develop an adequate representation of the necessary knowledge, and we should not expect much transfer from the project activity to other design activities unless they are very similar.
Third, participating in one or two large-scale projects does not lend itself to deliberate practice techniques, such as scaffolding and fading, which have been found effective in similar instructional contexts (e. g., [15]). They require multiple experiences, because the coach provides maximum support in the first instance, but gradually withdraws that support in later experiences.
Researchers are slowly unraveling the mystery of what constitutes expertise. Clearly, experts know more about their domains than novices. Moreover, their knowledge is differentially organized or represented. For example, what differentiates chess experts from novices is their ability to recognize legitimate configurations of chessboards and to know whether particular configurations will lead to a win or a loss [16]. Physicists represent physics problems in terms of underlying constructs, such as mass and acceleration, whereas novices represent such problems in terms of surface features, such as weights and pulleys [14]. Programmers remember more of a structured program than do novices, but do not remember more of a program presented in random order [17].
A second way experts differ from novices is in their how-to knowledge. Experts evidently have a script that guides the conduct of their work. For example, Schraagen [18] has shown that, although experts with specific knowledge of the domain create better experimental designs in psychology than design experts unfamiliar with the specific domain, all experts followed the same overall plan of attack. In software design, breadth-first with successive refinement is the prescribed script; designers tend to follow that prescription when they are highly familiar with the domain, but engage in depth-first or opportunistic design when the design task is unfamiliar [19]. Furthermore, experts are able to evaluate their work and their progress, know when they need more information, when to go on to the next step. Acquiring such metacognitive control mechanisms is part of developing expertise.
This research suggests that students need to learn how to go about a field as well as what to know about a field. Traditional lecture courses can disseminate knowledge about a field, but they leave the acquisition of how-to knowledge to chance. Fortunately, several instructional research projects have demonstrated that explicitly teaching how-to knowledge and skills is effective in promoting the acquisition how-to skills in college-level mathematical problem-solving, writing, and reading-to-understand. The techniques these projects sharedhas been named the cognitive apprenticeship model [15. Its steps are:
The model provides a framework for dealing with the complexities of design education by supporting strategies to engage students in what may be perceived as the right kind of thinking. This approach has been applied successfully to teach software design [20][21].
Each of the projects to be described in this section have taken as their focus teaching students how to go about developing a design intelligently. Linn and Clancy [22] developed a series of case studies for CS1 which included 1) a narrative description of an expert's solution written so the student can understand the approach, 2) the expert's coded solution, and 3) a series of study questions to guide the student's analysis of the program. The results indicated that students using these elaborated models began to reuse the expert's programming solution templates that had been presented. The software engineering apprentice project at Cal Poly (http://phoenix.calpoly.edu/~jdalbey/swebrief.html) applies this case study approach to CS1 and CS2.
Software engineering educators at Georgia Tech (http://www.cc.gatech.edu/computing/classes/RWL/process/docs/Table_of_Contents.html) and the University of Texas [23] used a level 2 Capability Maturity Model (CMM) process in support of student projects. Through expert modeling, they demonstrate how experts track project costs, schedules, and functionality, how they baseline work, how they control integrity of the product through a defined process, how they define software project standards, and how they ensure standards are followed.
Kolodner et. al. [7] developed an interdisciplinary design course for first year engineering students that focuses on these component subprocesses. Their pedagogical strategy is to have students practice, reflect on, and discuss each of these subprocesses, so that they understand the need for these skills and how they fit into the design process. Making the learning experience active and contextualized, students maximize their development of abstract representations of the how-to knowledge. Kolodner's group is developing software tools to support their curricular approach.
We are currently using the cognitive apprenticeship approach to integrate activities in software process early in the undergraduate curriculum [25]. First and foremost, we integrate certain best practices [26] into the development activities of the students. We again address both aspects of how-to knowledge--developing appropriate scripts for how to go about developing software and learning how to regulate the execution of that script. Students are asked to plan how they will proceed in their project and to document the product as they do the work. After completing their product, other students conduct a formal review of their product that includes defect identification and tracking. The student developers then discuss the criticisms with the reviewers. Finally all students complete a postmortem. In the postmortem they document the development process for that project and collect measurement data. Thus, the students have size, effort, and defect data and can use these data, along with self-reflection, to assess their development process, and establish goals for the next project. The results of each postmortem are stored electronically in a design notebook available via a standard web browser. By maintaining records for each project the students can review their progress over the course of the semester. A culminating event then is a personal assessment of what they gained during the course of the semester. This project supports the acquisition of how-to knowledge in several ways. First, through expert modeling and practice students learn to create plans for how to complete their products. Second, through systematic measurement and qualitative postmortems they learn effective self-regulation techniques. Reviewing the work of others helps them to adopt attitudes of constructive criticism that they can then apply to themselves. These techniques improve on the basic cognitive apprenticeship model because they systematize and make necessary the articulation and reflection that should occur in small-group work. Those who have closely observed students working in small groups know that not all students actively engage in articulation and reflection even though the situation encourages it.. By requiring that all students engage in systematic planning, systematically measure, and reflect in reviews and postmortems, no tudent can escape that difficult process.
Just as software design can be improved by following particular methodologies, SEE can effectively use some of the same techniques. Prototyping, user-based design has been found to be more effective than the waterfall methodology in software design and we have applied it successfully in SEE [20][21]. This requires producing prototypes, conducting formative evaluations, producing new prototypes, and conducting summative evaluations once the product stabilizes. The purpose of formative evaluations is to improve the product. Since our goal is to discover deliberate practice techniques that will help students overcome their difficulties, a major goal in formative evaluations is to identify (a) areas in which the prototype helps students overcome some difficulty (e. g., critiquing their own designs), (b) places where students still get stuck, and (c) places where the prototype can be improved. Students are often very insightful about how and why an instructional technique is not working. Formative evaluations also assess curricular effectiveness in an informal way, that is, not requiring the rigor of scientific experimentation. Finally, we measure user (student) satisfaction. This is fundamentally important, because, if it takes 10 years of dedicated practice to become an expert, the students must be motivated. We find that students enjoy hard work and disciplined approaches to learning when it is obvious to them that they can succeed, improve, and progress toward their goals.
Once the prototype becomes a curricular product, it behooves the developers to conduct formal, summative evaluations. The software engineering literature reminds us that the path to process improvement involves measurement and data driven change. That should be one of the working principles guiding SEE development.
Implementing an SEE program with the features suggested here and by the recommended techniques will be difficult within the current state of higher education. The mentoring and coaching implied by a cognitive apprenticeship approach does not fare well with current staff loading formulas and the reward structure typical in higher education, or industry for that matter. Yet, to dismiss the approach based on this difficulty seems ill-advised.