Evaluation will be the responsibility of Sims-Knight, who is a cognitive developmental psychologist with 25 years research experience. Over the course of her career she has created evaluation tools of all sorts, both formative and summative including objective paper-and-pencil, structured interview, and thinking-aloud protocols. Furthermore, she and Upchurch have collaborated on a project exploring the potential of developing a curriculum to teach high-level design to a variety of students from high school through senior computer science majors (see Results from Prior NSF Support Section). In this prior project they used the cognitive apprenticeship model as their teaching technique and used the principles of user-based design to develop the pedagogical procedures. Sims-Knight was responsible for the extensive evaluations, both formative and summative, that user-based design requires. In this project she will again take that role, while Upchurch and Green develop the instructional activities. Thus, her independence from the PIs will be assured. Sims-Knight's understanding of cognitive development and her understanding of the software industry and software design, developed through her activities as Co-Director of Academic Computing and her research activities in computer-related domains provides her a unique perspective from which to give general advice on the project's concept and conduct.
Formative evaluation is an essential feature of this proposal. The investigators model curriculum development on user-based design principles. We develop a prototype, build in informal evaluation and feedback in preliminary trials of the prototype, and revise. Thus, for each laboratory component (typically 4-6 weeks of activities), we will develop informal assessments. They will take various forms, but will always include 1) feedback gathered in coaching during instruction, 2) a test of declarative knowledge, 3) an assessment of skill in accomplishing the tasks, and 4) written or verbal interviews with the students to tap their feelings about the success or failure of the activities (see Results of Prior NSF Support for and example of user-based design in curricular development).
Summative assessment of students' skills will be conducted at the end of each of the five software-oriented courses. We will collect three kinds of data: tests of declarative knowledge of process and design issues, students' projects, particularly their legacy documents (collected in their portfolios), and an assessment of their design skills. The last is necessary because we are making a strong claim that this integrated curriculum will allow students to acquire expertise in a way not possible otherwise. Therefore, we need to document how students think about developing software at various points in the curriculum. We will track students' progress in designing software at the end of every course. All students in the class will be given a specific problem to solve. Five students randomly selected from those who earn a C or better will participate in a thinking aloud protocol while solving this problem. Thus we will be able to evaluate the product of all students and the processes of a constant select few.
Control groups will be formed from students who take these courses before the laboratory component is added. Students in the courses after the laboratory components are added will serve as experimental groups. This comparison is, of course, not methodologically pure, because the experimental and control groups are different cohorts and because the students in the more advanced courses will have received prototype versions of the laboratories in the earlier courses. Nonetheless, the comparison will give us preliminary indication of whether the laboratories are working. Furthermore, the most important comparison is the outcome of the fifth class. Even though these students will have had prototypes throughout their career, their participation in the laboratories will provide the class with the portfolio and previous exposure to the software process issues that will make a difference.
It should be evident from the discussion of the evaluation plan provided that we wish to critically assess not only the existence of change in our students behaviors in project develop, but the nature of the changes we engender. We see this issue as critical for a project of this scope, therefore to facilitate the ability to make such judgments we must begin the data collection, and thus, the data collection portion of the project prior to any formal award. You will see in the evaluation plan below the beginnings of the data collection activity following the structure provided in the student evaluation section above.
Thus, the plan of evaluations is as follows:
The anticipated results of the formative evaluations will, of course, be reflected primarily in improvements in the laboratory activities. There will be three cycles for the first two courses, which we expect to be sufficient to develop an effective, finished set of activities. For the final three courses, the products may still be works in progress, but if so, the formative evaluations will suggest the directions in which to move.
Two kinds of summative evaluations can be made, one longitudinal and one cross-sectional. The longitudinal control group can serve as control for both sets of experimental groups, because they consist of a single cohort of students who will be taking the five-course sequence without exposure to the laboratories. The cross-sectional experimental groups will have the most advanced versions of the laboratories and so will be the best test of the effectiveness of each single laboratory. We expect that the students who took the laboratories in Year 4 of the project will show better understanding of process concepts, including design methodologies, and more effective design skills than the control group taking the same course. These findings will demonstrate that students in traditional computer science curricula do not acquire process knowledge automatically by learning basic software concepts. The comparison of students at the end of their five-course sequence with or without the laboratories tests the hypothesis that the effect of the laboratories and their integration into the software engineering course is cumulative. This is the best test of the main hypothesis of this proposal--that students must explicitly learn to reflect upon software development and must actively practice the skills, and that such skills are acquired slowly over the course of several classes.
Sims-Knight will develop and administer the survey of department faculty's knowledge of software process and their assessment of its importance in computer science education. This will have to be a structured interview because the interviewer has to make sure that the faculty member understands the process issues before that faculty member can be expected to give informed opinions.
The results of our curriculum development activities will be made available in a number of ways. We plan to submit papers to the Software Engineering Institute's Conference on Software Engineering Education and ACM SIGCSE conference. We are interested in whether the strategies we use would be useful in a non-educational setting, therefore we are considering paper submissions to the International Conference on Software Engineering and the International Conference on Software Process. We are looking for research workshops at these conferences that are directly related to the role of individuals in learning and contributing to software process activities. We have started to form a collection of internet sites where material and practitioners can be found, and are currently in contact with some of these individuals. There is some likelihood of a research workshop, birds-of-a-feather session at the Conference on Software Engineering Education related to this topic.
All materials and reports will be made available at the Computer Science Department world-wide web site maintained by the one of the PIs at http://www2.umassd.edu/. The problem book is certainly open to all, but we will have to explore the issues related to student portfolio access before making it available.
Of particular importance is the faculty training component developed during our activity. We will plan a faculty training workshop for computer science faculty in the University of Massachusetts system for the third year of the project as part of a system initiative to coordinate between the different universities. From that workshop we should gain significant feedback regarding both our approach and the potential for porting the outcomes of this endeavor to other institutions.