Teaching and Learning Forum 98 [ Contents ]

Curriculum evaluation: Considering the interests of all the stake holders

Karen Glaister, Robyn Davis, Alan Tulloch and Carol Piercey
School of Nursing
Curtin University of Technology
Evaluating any curriculum can be frustrating and challenging. This is especially so when there are multiple stake holders from diverse backgrounds and philosophical positions. An evaluation of the Bachelor of Science (Nursing) curriculum, was undertaken 1995-1996. The science stream, however, was not completely reviewed. This created a need to evaluate the science units separately from other units in the program.

The subsequent evaluation of the science units involved a multidisciplinary approach. In order to preserve the credibility of the process, it was necessary that procedures were acceptable to all teaching staff. This presented an interesting array of challenges requiring ingenuity, patience and dedication to the task.

Stake holders were recognised as the students enrolled in the undergraduate program, teaching staff from both the School of Nursing and other schools within the Division of Health Sciences. The evaluation was conducted by members of the nursing staff involved in the science stream, with resources being provided by the School of Nursing. The methodology took both a qualitative and quantitative approach utilising questionnaires and focus groups.

This paper describes the process and outcomes of the evaluation and discusses the implications of the findings for other disciplines, confronted with similar program reviews.


Introduction

The aim of this study was to evaluate the biological and physical science stream within the Bachelor of Science (Nursing) curriculum. The focus was to identify strengths and weaknesses and to determine the effectiveness of course objectives within the philosophy of the School of Nursing. Fundamental to this study were the objectives of the previous evaluation which omitted the units in the science stream. The main issues were the concerns of the staff and students with regards to the implementation of the 'New Curriculum'. The science stream included eight units spread over five semesters of study. Three of the units were coordinated and taught by nursing academics, the remaining five were taught entirely by staff from disciplines other than nursing.

Methodology

In order to be consistent with the methodological strategies used in the original curriculum evaluation, this study used both a qualitative and quantitative approach. Triangulation of methods, data sources and analysis were employed to improve validity and reliability of the study.

This study differed from the previous evaluation in that the stakeholders were not only the School of Nursing and nursing students, but also the service teaching schools in the Division of Health Sciences.

As part of the collaborative approach between stakeholders, the study commenced with a staff workshop. The purpose of this was to identify nursing academic perceptions of the strengths and weaknesses of the science stream. Similar meetings were held with the relevant teaching staff of all service schools to elicit their cooperation and involvement in the study. This progressed to the collection of student derived data, using a questionnaire and focus groups. The final phase of the study will be to inform and share with all relevant schools the outcomes and recommendations.

Research tools

A questionnaire was used to collect data from the large student body. Throughout the process of selecting a suitable questionnaire, stakeholders (excluding the students) were consulted together with the Nursing Research Committee, Teaching and Learning Group and the University Quality Office. A systematic search and comparative analysis of available questionnaires resulted in the adoption of the 1994 version of the 'Course Experience Questionnaire' (CEQ). This instrument which was designed to collect recent graduate's perceptions of their course, has been widely adopted with recognised reliability and validity (Johnson, 1996).

The only modification required was a change in the focus of the questionnaire from "course" to "unit" . The questionnaire was multi-dimensional, composed of both quantitative and qualitative inquiry. Quantitative data was collected using a thirty item Likert-type scale, ranging from strong disagreement to strong agreement. The items were randomly ordered, with eight items being reversed. Two open ended questions were included to gather qualitative data. These questions asked for comments regarding the best aspects of the course and those aspects most in need of improvement. Open ended questions allowed the generation of a wide range of replies and provided the researchers with some insight into prevailing attitudes. The comments from these were later coded into 6 categories : teaching and learning, unit material/resources, unit structure, assessment, student issues and staff issues.

Sample

The key informants for this study were most of the nursing students in the defined units. First semester students were exempted as they had no previous experience with the course.

Data Collection

Questionnaire

The questionnaire was administered in week two, second semester, 1997. All students who attended a science related teaching and learning session were asked to complete the questionnaire. Appropriate ethical protocols and the purpose of the study were discussed with the students before administering the instrument. Data collected from each group of students was retrospectively oriented, requiring students to comment on those units that they had successfully completed in the preceding semester. Semester seven students were asked to comment on the entire science stream and it's articulation with other units in the curriculum.

Focus groups

Focus groups were conducted with students from each semester. A random sample of 12 students from each semester was selected. This method of sample selection allowed some control over the potential bias that mature age students and unit repeaters could have posed.

Questions for the interview protocol were planned to clarify the themes that emerged from the open ended responses on the questionnaire. Items from the CEQ analysis indicating student disagreement were rephrased to form open ended questions and redirected to the students for more specific comment and clarification. The interview protocol was semi-structured with broad questions preceding specific questions. It was used as an outline and guide for the facilitator to follow during the focus group discussion, as strict adherence to the questions and changing topics may have stifled spontaneous comments and reduced facilitator flexibility. The flow of discussion was maintained by the facilitator prompting and probing statements and comments and keeping students on track. This drew out a range of perceptions and beliefs within group. Using this method the facilitator endeavored to gain a broader understanding of the student responses and the reasons behind their attitudes and behaviors. Sessions were recorded on audiotape in a sound studio. Each focus group continued until the interview protocol was completed and discussion was exhausted, these sessions took up to one hour for each group.

Analysis

Computer generated results used standard measures adopted by CEQ from the quantitative component of the questionnaire. Data were grouped into five standard scales. These being, Good teaching scale, Clear goals and standards scale, Appropriate workload scale, Appropriate assessment scale and the Generic skills scale.

The values of the Likert scale were altered in the calculation of the mean to where 1 = -100; 2 = -50; 3 = 0; 4 = +50 and 5 = +100. Scoring for the negatively worded items was reversed. Values ranged between +100 and -100. A score of -100 represented strongly disagree and +100 for strongly agree. Negative scores were interpreted as disagreement and positive scores indicated agreement with the item. Percentages provided an indicator of overall agreement or disagreement.

The written qualitative data was examined for recurring patterns. These patterns were categorised and used to provided direction for focus groups. Transcription of the focus groups have yet to be analysed using a thematic content analysis (Burnard, 1991).

Results and discussion

The results of the CEQ suggest, that students perceived inconsistencies within and between the science stream units. The following table illustrates the variability within the subscales.

SubscalesRangeStream average

Good teaching-27 to 28-4
Generic-28 to 13-10
Clear goals and standards-27 to 3410
Appropriate workload-50 to 40-19
Appropriate assessment-41 to 23-4
Satisfaction-23 to 4213

The results from the analysis of the questionnaire were examined identifying common themes and trends. The score for each unit on each of the subscales was compared to the stream average for that subscale and those units which scored poorly were noted. A unit was highlighted if the score was negative or if the score was markedly less positive than the stream average. Using this method, significant inconsistencies in the subscales were identified. The analysis showed a marked difference in the perception of the students with regard to individual units. Significant variation was found on the "Good Teaching" scale. Those units which were taught by nursing academics ranging from +24 to +28, while service taught units scored between - 4 and -27. The stream average for this subscale was -4. Scores for other subscales were more balanced and did not demonstrate such a wide variation.

In general, the open ended questions from the questionnaire posed a dichotomy between the nursing and service taught units. Some of the most salient negative responses which were outside the scope of CEQ were attributed to the service taught units. There was a strong student perception that academic staff involved in these units lacked commitment to the unit and to nursing students in general. Students listed a lack of support, derogatory comments against the Nursing profession and in some cases open hostility towards them as a significant reason for their impaired learning. In some cases failure or withdrawal from the unit was attributed to this problem. Other issues identified were, lack of articulation of service units with the rest of the curriculum, lack of direct application (focus) and relevance of content in the service units to the practice of nursing. Inappropriate and inconsistent assessment practices were also noted. Some service units did not follow the policies and philosophies of the School of Nursing. Many of these comments, in particular those relating to relevance, content and sequencing, were also raised as points of issue by nursing academics in the workshop held at the commencement of the study. While there were many negative comments, students generally recognised the relevance of the science units to their clinical practice. This perception strengthened as the students proceeded through their course.

Analysis and examination of the data gathered in the focus groups has yet to be completed. Initial inspection of the transcripts, however, suggests similarities with the outcomes from the questionnaire.

Implications

Although the study is still in progress, it is possible to identify areas of importance from the results available.

There are two district areas which need to be considered in light of the findings to date; academic issues within the process of evaluating service taught units and the potential impact of service teaching on student learning outcomes

Issues for academics

The use of service teaching as a method of cost containment and maximising the use of available expertise is widespread within the university system. These units are often taught by academic staff who are experts in their field, but have no basis in the professional discipline in which the students are studying. This may mean that the service teaching staff and the school (in which the student is enrolled) come from different educational and philosophical perspectives.

In a project of this nature, reservations may be held by some team members with respect to the selection of the research process. These reservations may be based on the ideological orientation and personal experiences of the individual academic. The use of an evaluation process which has rigor, validity and reliability will encourage acceptance of the outcomes.

The need for a service taught unit to be professionally relevant was identified by both nursing academics and the students as vitally important. The data indicated that both groups felt that the content taught within the service unit must reflect the rest of the course, both in depth and focus. Nursing academics expressed concern that they had "lost control" of what was being taught within the service unit. It was felt that there was a risk that content of the unit may not reflect the intent and focus of the curriculum if the unit was funded and staffed by a school other than Nursing. This is particularly important if the unit forms part of a Common First Year as these concerns may equally apply to other professionally based courses .

The collection of data relating to student perceptions and opinion of teaching practice can a sensitive issue. Much of the data personal and must be held in the strictest confidence, while being available to the individual so they can evaluate their own teaching style. Academic accountability for acting on the information which impacts on the course content and student learning, rests with the Curriculum Committee of the school in which the student is enrolled. Protocols for the distribution of this type of information must be established in the planning phase of an evaluation project.

Student learning outcomes

In professionally based courses the primary learning outcome is the application of theoretical knowledge to professional practice. It is therefore vital that the course content is directly related to practice. It is also important that students are exposed to professional role models from within their chosen career and that this exposure is continuous throughout their course of study. First Year Experience research (McInnis & James (1995) indicates that staff commitment to student support and the relevance of learning to their chosen profession enhances student participation. These factors also influence failure and drop-out rates. Student and staff data indicated that most service taught units failed to provide this professional link.

The students reported a perception that the lecturers who did not have the appropriate professional base failed to provide the same level of commitment, support and respect that was provided by nursing lecturers. Some students felt that this restricted their learning and in some cases the student did not attend lectures or tutorials because they felt "put down" by the lecturer. Service units which are usually taught in the early semesters need to be designed carefully to meet these professional needs.

Conclusion

This paper has presented an overview of a curriculum evaluation project incorporating qualitative and quantitative approaches. The results of the project have ramifications for curriculum planning, not only to the School of Nursing, but to other schools and disciplines with similar course structures.

Curriculum design should be founded upon sound educational practices. There is no place for a curriculum which is purely economically driven. Any curriculum with service taught units must have well established protocols which will ensure quality teaching staff, validity of content and appropriate teaching strategies.

Reference list

Burnard, P. (1991). A method of analysing interview transcripts in qualitative research. Nurse Education Today, 11, 461-466.

Johnson, T. (1997). The 1996 Course Experience Questionnaire: A report prepared for the Graduate Careers Council of Australia. Australia: Graduate Careers Council of Australia.

McInnis C. & James, R. (1995). First year on campus. Committee for the Advancement of University Education

Please cite as: Glaister, K., Davis, R., Tulloch, A. and Piercey, C. (1998). Curriculum evaluation: Considering the interests of all the stake holders. In Black, B. and Stanley, N. (Eds), Teaching and Learning in Changing Times, 110-114. Proceedings of the 7th Annual Teaching Learning Forum, The University of Western Australia, February 1998. Perth: UWA. http://lsn.curtin.edu.au/tlf/tlf1998/glaister.html


[ TL Forum 1998 Proceedings Contents ] [ TL Forums Index ]
HTML: Roger Atkinson, Teaching and Learning Centre, Murdoch University [rjatkinson@bigpond.com]
This URL: http://lsn.curtin.edu.au/tlf/tlf1998/glaister.html
Last revision: 5 Mar 2002. The University of Western Australia
Previous URL 29 Dec 1997 to 5 Mar 2002 http://cleo.murdoch.edu.au/asu/pubs/tlf/tlf98/glaister.html