Teaching and Learning Forum 2000 [ Proceedings Contents ]

Student questionnaires and teaching evaluation: Cutting the cloth to fit the purpose

A panel discussion by
Elizabeth Santhanam
Centre for Staff Development, University of Western Australia

Christina Ballantyne
Teaching and Learning Centre, Murdoch University

Denise Mulligan, Barbara de la Harpe
Centre for Educational Advancement, Curtin University of Technology
and
Rod Ellis
Educational Development Unit, Edith Cowan University
    There are almost as many formats for obtaining student views on teaching as there are universities in Australia. The drive for improving teaching quality in higher education in the last two decades has resulted in universities developing their own questionnaires or adapting questionnaires developed at other institutions to suit their purposes. Among the stakeholders of student evaluation of teaching are students, academic staff and heads of departments/ schools/ centres. Each party has its own agenda, and the instrument that is used is expected to serve all their interests. Given the divergent interests of some stakeholders, as well as changing economic and policy trends, the system to evaluate teaching using student feedback tends to evolve over time.

    In this discussion, the panel members will describe the current system of student evaluation at each of their institutions, namely The University of Western Australia, Murdoch University, Curtin University of Technology and Edith Cowan University, and provide the reasons for adopting their particular system. They will discuss the respective policies that govern student evaluations of teaching, and how the various stakeholders' interests are met. The open discussion to follow will further clarify some of the issues, and the panel members will attempt to dispel some of the frequently stated myths regarding student evaluation of teaching.

Teaching and Learning Forum 2000 Home Page


Introduction

A number of parties have vested interests in maintaining the quality of university education in Australia, and university administrators and policy makers generally try to ensure that the various stakeholders are satisfied. Australian universities, like tertiary institutions elsewhere, usually have policies in place regarding quality assurance and evaluation of teaching. One way of determining that the education provided by an institution meets the expected quality standards and answers the needs of students is to ask the students themselves.

Typically, this probing of student views is through questionnaires. Questionnaires can be used to provide an overall view of teaching to administrators (including heads of departments/ schools/centres), to diagnose individual academics' teaching for further development, and to provide students with the opportunity to indicate their needs to academics/administrators. Universities have either developed their own questionnaires or adapted/adopted a questionnaire developed elsewhere. Often the questionnaire undergoes many changes before a general consensus is reached regarding its effectiveness. Some aspects of making questionnaires fit the purposes of student evaluations of teaching and units can be seen in the way four universities in Western Australia currently facilitate the process.

Student feedback on teaching at UWA

The University of Western Australia has provided a centralised service for obtaining student feedback on teaching for about ten years. The instrument used to collect feedback has undergone changes and is currently known as Student Perceptions of Teaching (SPOT). The primary purpose of SPOT is formative evaluation of teaching, but it can be used for obtaining summative information. It is based on research evidence and past experiences in UWA relating to student feedback. The questionnaire was modelled loosely on the 'Cafeteria system' developed at Purdue University. In this system, instructors select from a pool of items, and there is no one fixed instrument of measurement. Thus SPOT is highly flexible. In addition to the items selected by academics, three generic items (global items) are typically included. Students indicate their level of agreement or disagreement on a five point scale for each item on the questionnaire. There is also an opportunity for including open ended questions. SPOT can be used for both teaching evaluation and unit evaluation.

SPOT evaluation is voluntary and confidential. The SPOT policy stipulates that in a group teaching situation evaluation of teaching should be carried out with the permission of all those involved in teaching. An academic whose teaching is being evaluated should be absent during the administration of the survey. Usually a colleague in the department administers the survey. Completed responses are enclosed in a sealed and signed envelope, and forwarded to the SPOT Office at the Centre for Staff Development. In order to minimise the real or perceived threat students may feel when giving honest opinions, SPOT is anonymous. A teaching survey report is sent to the academic whose teaching was evaluated. A unit survey report is currently sent to all academics teaching in the unit, but this practice in being reviewed. Heads of departments are sent SPOT reports only when the lecturers/tutors have agreed in writing to release their reports.

A standard survey report produced by the SPOT Office consists of a detailed breakdown of student responses for each item i.e. number of student responses for each response option, and a mean value of the ratings; ratings are based on a scale of 1 (Strongly Disagree) to 5 (Strongly Agree). For ease of interpretation, bar charts showing percentage responses for each option are also plotted. A summary sheet accompanies these details, listing the items and their mean scores. Rating Interpretation Guides, which provide comparative data for the three global items, are available on the Web. Comparative data for the other items are not provided, since each questionnaire is unique.

Student feedback on teaching at Murdoch

In 1994 Murdoch University introduced mandatory student surveys of units on a three year cycle. The original purpose of the surveys was for academic staff development and curriculum improvement in individual units. Over the years several changes have occurred in the uses of the data collected; as one of the university's performance indicators, in the calculation of a Teaching Quantum, in the determination of Teaching Excellence Awards, for promotions and, of course, for the overall improvement of the unit. Thus what was initially a formative tool for improvement at the level of the individual teacher and unit, is now also being used to collect summative information for personnel and management decisions. By 1998 these changes in the use of the information, together with the development of 'Key Quality Standards' for units by the university [see http://cleo.murdoch.edu.au/evaluation/survey/unit.html#kqs] indicated the need to move to annual surveys using a standard questionnaire for all units.

The coordinator of each coursework unit with more than ten enrolled students is sent questionnaires for distribution in the last two weeks of semester. Instructions are provided for a student to collect and return the results for analysis. The standard questionnaire for units addresses the Key Quality Standards, and provides feedback to the unit coordinator on whether they are being met. This questionnaire can be seen at [http://cleo.murdoch.edu.au/evaluation/survey/draft.html]. Results from individual unit surveys are sent not only to the individual unit coordinator, but also to the Dean and the head of the Academic Organisational Unit. Aggregated results at Divisional and University level are available on the web at [http://cleo.murdoch.edu.au/evaluation/survey/results.html].

With the changeover to this new system of annual surveys of units, there was a need also to formulate a means of collecting student feedback on an individual's teaching. A standard questionnaire was constructed with questions relating to the criteria for assessing items of evidence of teaching that promotes effective learning, as identified in Table 2 of the Guidelines for the Presentation of a Teaching Portfolio [see http://wwwadmin.murdoch.edu.au/hr/traindev/teachportfolio.html]. Teaching surveys are currently by request only and the results are returned in confidence to the teacher concerned. To avoid the need for students to complete two separate surveys at the end of the semester, it is recommended that teaching surveys are undertaken in the middle teaching block - weeks 5-9 - of the semester. Results are provided by week ten to allow staff to respond to the students concerns, giving the teaching surveys a more formative aspect. The teaching questionnaire can be seen at [http://cleo.murdoch.edu.au/evaluation/survey/teachdraft.html]

Student feedback on teaching at Curtin

The Student Evaluation of Educational Quality (SEEQ) is the officially endorsed instrument for student evaluation of teaching at Curtin. The SEEQ has been developed, systematically validated and adapted for use in Australasian settings by Marsh and Roche (1993; 1994). The SEEQ forms part of an integrated package to obtain student feedback on teaching and to develop teaching quality through reflective practice. It is used at Curtin for both reflective self development and promotional purposes.

After extensive trials in 1995 and 1996 (Teaching Learning Group, 1997) the Teaching and Learning Advisory Committee and University Academic Board (UAB) decided to adopt SEEQ over a number of other student appraisal systems for the following reasons:

The SEEQ package currently comprises: Apart from the requirement for staff to produce their SEEQ results for the purposes of promotion, the use of SEEQ is voluntary. SEEQ must be administered by a person other than the staff member seeking feedback and results are only made available to the staff member seeking feedback.

Following administration of the survey, the staff member is provided with:

Since its official adoption by UAB in 1997, the standard SEEQ package has been modified to better meet the needs of Curtin teaching staff and continues to be developed and refined.

Student feedback on teaching at ECU

Edith Cowan University has a formal unit evaluation procedure that is centrally managed. The procedure is outlined below: A University Unit Evaluation policy document underpins all procedures and processes relating to the above mentioned steps. This document was formulated after much discussion and debate by the wider university community.

The instrument has been used now for about 5 years and has undergone several 'improvements'. The current process specifically targets unit design, unit content, workload, presentation (teaching), assessment and feedback. It was designed more for unit feedback than teaching effectiveness feedback although these obviously overlap.

The University has recently emphasised teaching and learning as a key strategic issue and has recognised that a more comprehensive 'teaching effectiveness' process is required. One of the driving forces is the need to have reliable and valid evidence of teaching effectiveness for staff promotions and appointments.

To this end a working party is currently devising a second instrument that deals specifically with the performance of people who contribute to the teaching of a unit of study. Such an instrument will be able to be used at any time in the running of a unit and not just as the 'normal' end of unit evaluation exercise. The results would be made available only to the lecturer requesting. One of the most difficult issues is to design a universal instrument that copes with the many varied ways in which staff work with students in the presentation of a unit of study.

Conclusion

There are similarities and differences among the instruments used by these four Western Australian universities to collect student feedback on teaching and/or units. Some of the common features are that the evaluation of individual academics' teaching is (or will be in the case of ECU) confidential and voluntary, and that the instruments have both rating types of items and open ended questions. The main difference is in the degree of flexibility made available and whether or not unit evaluation is mandatory. The instruments range from cafeteria systems, where the academics determine the number and type of items to be included in the questionnaire, to a standardised instruments. It is also evident that instruments in each institution have undergone changes to cater to the needs of the community they serve.

The advantages and disadvantages of these instruments, as well as other aspects of student evaluations of teaching, can be discussed during the forum:

References

Marsh, H. W., & Roche, L. (1993). The use of students' evaluations and an individually structured intervention to enhance university teaching effectiveness. American Educational Research Journal, 30(1), 217-251.

Marsh, H. W., & Roche, L. A. (1994). The use of students' evaluations of university teaching to improve teaching effectiveness. Final project report for the Evaluations and Investigations Program of the Department of Employment and Education. Canberra: Australian Government Printing Service.

Teaching Learning Group. (1997). Student evaluation of teaching at Curtin University: Piloting the Student Evaluation of Educational Quality (SEEQ). A report on a Quality Funded Project to the Teaching and Learning Advisory Committee and the Promotions Review Committee. Perth, W.A.: Curtin University of Technology.

Please cite as: Santhanam, E., Ballantyne, C., Mulligan, D., de la Harpe, B. and Ellis, R. (2000). Student questionnaires and teaching evaluation: Cutting the cloth to fit the purpose. In A. Herrmann and M.M. Kulski (Eds), Flexible Futures in Tertiary Teaching. Proceedings of the 9th Annual Teaching Learning Forum, 2-4 February 2000. Perth: Curtin University of Technology. http://lsn.curtin.edu.au/tlf/tlf2000/santhanam2.html


[ TL Forum 2000 Proceedings Contents ] [ TL Forums Index ]
HTML: Roger Atkinson, Teaching and Learning Centre, Murdoch University [rjatkinson@bigpond.com]
This URL: http://lsn.curtin.edu.au/tlf/tlf2000/santhanam2.html
Last revision: 20 Feb 2002. Curtin University of Technology
Previous URL 29 Dec 1999 to 20 Feb 2002 http://cleo.murdoch.edu.au/confs/tlf/tlf2000/santhanam2.html