|Teaching and Learning Forum 2000 [ Proceedings Contents ]
Student questionnaires and teaching evaluation: Cutting the cloth to fit the purpose
A panel discussion by
Centre for Staff Development,
University of Western Australia
Teaching and Learning Centre,
Denise Mulligan, Barbara de la Harpe
Centre for Educational Advancement,
Curtin University of Technology
Educational Development Unit,
Edith Cowan University
There are almost as many formats for obtaining student views on teaching as there are universities in Australia. The drive for improving teaching quality in higher education in the last two decades has resulted in universities developing their own questionnaires or adapting questionnaires developed at other institutions to suit their purposes. Among the stakeholders of student evaluation of teaching are students, academic staff and heads of departments/ schools/ centres. Each party has its own agenda, and the instrument that is used is expected to serve all their interests. Given the divergent interests of some stakeholders, as well as changing economic and policy trends, the system to evaluate teaching using student feedback tends to evolve over time.
In this discussion, the panel members will describe the current system of student evaluation at each of their institutions, namely The University of Western Australia, Murdoch University, Curtin University of Technology and Edith Cowan University, and provide the reasons for adopting their particular system. They will discuss the respective policies that govern student evaluations of teaching, and how the various stakeholders' interests are met. The open discussion to follow will further clarify some of the issues, and the panel members will attempt to dispel some of the frequently stated myths regarding student evaluation of teaching.
A number of parties have vested interests in maintaining the quality of university education in Australia, and university administrators and policy makers generally try to ensure that the various stakeholders are satisfied. Australian universities, like tertiary institutions elsewhere, usually have policies in place regarding quality assurance and evaluation of teaching. One way of determining that the education provided by an institution meets the expected quality standards and answers the needs of students is to ask the students themselves.
Typically, this probing of student views is through questionnaires. Questionnaires can be used to provide an overall view of teaching to administrators (including heads of departments/ schools/centres), to diagnose individual academics' teaching for further development, and to provide students with the opportunity to indicate their needs to academics/administrators. Universities have either developed their own questionnaires or adapted/adopted a questionnaire developed elsewhere. Often the questionnaire undergoes many changes before a general consensus is reached regarding its effectiveness. Some aspects of making questionnaires fit the purposes of student evaluations of teaching and units can be seen in the way four universities in Western Australia currently facilitate the process.
Student feedback on teaching at UWA
The University of Western Australia has provided a centralised service for obtaining student feedback on teaching for about ten years. The instrument used to collect feedback has undergone changes and is currently known as Student Perceptions of Teaching (SPOT). The primary purpose of SPOT is formative evaluation of teaching, but it can be used for obtaining summative information. It is based on research evidence and past experiences in UWA relating to student feedback. The questionnaire was modelled loosely on the 'Cafeteria system' developed at Purdue University. In this system, instructors select from a pool of items, and there is no one fixed instrument of measurement. Thus SPOT is highly flexible. In addition to the items selected by academics, three generic items (global items) are typically included. Students indicate their level of agreement or disagreement on a five point scale for each item on the questionnaire. There is also an opportunity for including open ended questions. SPOT can be used for both teaching evaluation and unit evaluation.
SPOT evaluation is voluntary and confidential. The SPOT policy stipulates that in a group teaching situation evaluation of teaching should be carried out with the permission of all those involved in teaching. An academic whose teaching is being evaluated should be absent during the administration of the survey. Usually a colleague in the department administers the survey. Completed responses are enclosed in a sealed and signed envelope, and forwarded to the SPOT Office at the Centre for Staff Development. In order to minimise the real or perceived threat students may feel when giving honest opinions, SPOT is anonymous. A teaching survey report is sent to the academic whose teaching was evaluated. A unit survey report is currently sent to all academics teaching in the unit, but this practice in being reviewed. Heads of departments are sent SPOT reports only when the lecturers/tutors have agreed in writing to release their reports.
A standard survey report produced by the SPOT Office consists of a detailed breakdown of student responses for each item i.e. number of student responses for each response option, and a mean value of the ratings; ratings are based on a scale of 1 (Strongly Disagree) to 5 (Strongly Agree). For ease of interpretation, bar charts showing percentage responses for each option are also plotted. A summary sheet accompanies these details, listing the items and their mean scores. Rating Interpretation Guides, which provide comparative data for the three global items, are available on the Web. Comparative data for the other items are not provided, since each questionnaire is unique.
Student feedback on teaching at Murdoch
In 1994 Murdoch University introduced mandatory student surveys of units on a three year cycle. The original purpose of the surveys was for academic staff development and curriculum improvement in individual units. Over the years several changes have occurred in the uses of the data collected; as one of the university's performance indicators, in the calculation of a Teaching Quantum, in the determination of Teaching Excellence Awards, for promotions and, of course, for the overall improvement of the unit. Thus what was initially a formative tool for improvement at the level of the individual teacher and unit, is now also being used to collect summative information for personnel and management decisions. By 1998 these changes in the use of the information, together with the development of 'Key Quality Standards' for units by the university [see http://cleo.murdoch.edu.au/evaluation/survey/unit.html#kqs] indicated the need to move to annual surveys using a standard questionnaire for all units.
The coordinator of each coursework unit with more than ten enrolled students is sent questionnaires for distribution in the last two weeks of semester. Instructions are provided for a student to collect and return the results for analysis. The standard questionnaire for units addresses the Key Quality Standards, and provides feedback to the unit coordinator on whether they are being met. This questionnaire can be seen at [http://cleo.murdoch.edu.au/evaluation/survey/draft.html]. Results from individual unit surveys are sent not only to the individual unit coordinator, but also to the Dean and the head of the Academic Organisational Unit. Aggregated results at Divisional and University level are available on the web at [http://cleo.murdoch.edu.au/evaluation/survey/results.html].
With the changeover to this new system of annual surveys of units, there was a need also to formulate a means of collecting student feedback on an individual's teaching. A standard questionnaire was constructed with questions relating to the criteria for assessing items of evidence of teaching that promotes effective learning, as identified in Table 2 of the Guidelines for the Presentation of a Teaching Portfolio [see http://wwwadmin.murdoch.edu.au/hr/traindev/teachportfolio.html]. Teaching surveys are currently by request only and the results are returned in confidence to the teacher concerned. To avoid the need for students to complete two separate surveys at the end of the semester, it is recommended that teaching surveys are undertaken in the middle teaching block - weeks 5-9 - of the semester. Results are provided by week ten to allow staff to respond to the students concerns, giving the teaching surveys a more formative aspect. The teaching questionnaire can be seen at [http://cleo.murdoch.edu.au/evaluation/survey/teachdraft.html]
Student feedback on teaching at Curtin
The Student Evaluation of Educational Quality (SEEQ) is the officially endorsed instrument for student evaluation of teaching at Curtin. The SEEQ has been developed, systematically validated and adapted for use in Australasian settings by Marsh and Roche (1993; 1994). The SEEQ forms part of an integrated package to obtain student feedback on teaching and to develop teaching quality through reflective practice. It is used at Curtin for both reflective self development and promotional purposes.
After extensive trials in 1995 and 1996 (Teaching Learning Group, 1997) the Teaching and Learning Advisory Committee and University Academic Board (UAB) decided to adopt SEEQ over a number of other student appraisal systems for the following reasons:
The SEEQ package currently comprises:
- There is substantial research evidence that SEEQ is a valid and reliable instrument for measuring students' perceptions of teaching, that is relatively unaffected by variables that may bias responses (e.g. class size, level of unit);
- SEEQ is specifically designed as a self development package, thus having the potential to contribute to Objective 4 of the Curtin University Teaching and Learning Strategic Plan i.e. To encourage reflective practice by staff; and
- Professor Marsh was willing to collaborate to further develop SEEQ to suit Curtin's needs.
Apart from the requirement for staff to produce their SEEQ results for the purposes of promotion, the use of SEEQ is voluntary. SEEQ must be administered by a person other than the staff member seeking feedback and results are only made available to the staff member seeking feedback.
- SEEQ closed response form, consisting of 33 items that combine to provide indicators of quality in nine key areas, namely learning and academic value, instructor enthusiasm, organisation and clarity, group interaction, individual rapport, breadth of coverage, examinations and grading, assignments and readings, and workload and difficulty, plus an overall rating;
- SEEQ open response form where students provide written comments;
- supplementary Rating Items developed by Curtin staff to provide feedback on a range of teaching modes not covered by the standard SEEQ form;
- provision for lecturers to formulate up to 40 of their own questions in addition to the standard SEEQ items;
- SEEQ Lecturer Self Rating version of the form; and
- support materials in the form of Targeted Teaching Strategies booklets.
Following administration of the survey, the staff member is provided with:
Since its official adoption by UAB in 1997, the standard SEEQ package has been modified to better meet the needs of Curtin teaching staff and continues to be developed and refined.
- a Summary Report that includes mean scores and percentile graphs for the nine SEEQ factors and the overall rating, and for each of the 33 items. For comparative purposes only, university norms for the nine SEEQ factors and overall rating are also included;
- an interpretation guide (Your SEEQ Report) that includes general advice on how to interpret SEEQ scores and how to seek follow up advice; and
- the URL [http://cea.curtin.edu/seeq/follow.html#tts] for access to the SEEQ Targeted Teaching Strategies booklets online - these aim to help staff to improve their teaching and relate directly to the feedback provided by the SEEQ survey.
Student feedback on teaching at ECU
Edith Cowan University has a formal unit evaluation procedure that is centrally managed. The procedure is outlined below:
A University Unit Evaluation policy document underpins all procedures and processes relating to the above mentioned steps. This document was formulated after much discussion and debate by the wider university community.
- Every unit of study at ECU is evaluated using the centralised Unit Evaluation process once every 3 years (more often if requested by a Head of School or Unit Coordinator).
- The instrument consists of 16 core questions that must be included, two open ended questions and up to 14 questions that can be selected from a bank of 350 questions grouped in 20 different categories. The requestor can also design their own questions as part of the 14 alternative questions.
- The instrument is administered by 'people' independent of the presentation of the unit and the forms are collected and returned to the Programme Review staff directly for computer processing.
- The forms are processed and the results are made available to Heads of School and the Unit coordinator (core questions only) and the individual lecturer (all questions). The Head of School and Unit coordinator are not permitted to forward data to Deans or the University Executive or to other groups such as promotion panels, etc.
- Aggregate Unit Evaluation data is computed from all returns and the results are returned to Faculty Deans (anonymously) for unit and courses comparisons. The same data is also made available on the web for general interest.
The instrument has been used now for about 5 years and has undergone several 'improvements'. The current process specifically targets unit design, unit content, workload, presentation (teaching), assessment and feedback. It was designed more for unit feedback than teaching effectiveness feedback although these obviously overlap.
The University has recently emphasised teaching and learning as a key strategic issue and has recognised that a more comprehensive 'teaching effectiveness' process is required. One of the driving forces is the need to have reliable and valid evidence of teaching effectiveness for staff promotions and appointments.
To this end a working party is currently devising a second instrument that deals specifically with the performance of people who contribute to the teaching of a unit of study. Such an instrument will be able to be used at any time in the running of a unit and not just as the 'normal' end of unit evaluation exercise. The results would be made available only to the lecturer requesting. One of the most difficult issues is to design a universal instrument that copes with the many varied ways in which staff work with students in the presentation of a unit of study.
There are similarities and differences among the instruments used by these four Western Australian universities to collect student feedback on teaching and/or units. Some of the common features are that the evaluation of individual academics' teaching is (or will be in the case of ECU) confidential and voluntary, and that the instruments have both rating types of items and open ended questions. The main difference is in the degree of flexibility made available and whether or not unit evaluation is mandatory. The instruments range from cafeteria systems, where the academics determine the number and type of items to be included in the questionnaire, to a standardised instruments. It is also evident that instruments in each institution have undergone changes to cater to the needs of the community they serve.
The advantages and disadvantages of these instruments, as well as other aspects of student evaluations of teaching, can be discussed during the forum:
- What fears/frustrations do academics and students have with the existing systems?
- What are the benefits for different stakeholders?
- What should be included/excluded/changed?
Marsh, H. W., & Roche, L. (1993). The use of students' evaluations and an individually structured intervention to enhance university teaching effectiveness. American Educational Research Journal, 30(1), 217-251.
Marsh, H. W., & Roche, L. A. (1994). The use of students' evaluations of university teaching to improve teaching effectiveness. Final project report for the Evaluations and Investigations Program of the Department of Employment and Education. Canberra: Australian Government Printing Service.
Teaching Learning Group. (1997). Student evaluation of teaching at Curtin University: Piloting the Student Evaluation of Educational Quality (SEEQ). A report on a Quality Funded Project to the Teaching and Learning Advisory Committee and the Promotions Review Committee. Perth, W.A.: Curtin University of Technology.
|Please cite as: Santhanam, E., Ballantyne, C., Mulligan, D., de la Harpe, B. and Ellis, R. (2000). Student questionnaires and teaching evaluation: Cutting the cloth to fit the purpose. In A. Herrmann and M.M. Kulski (Eds), Flexible Futures in Tertiary Teaching. Proceedings of the 9th Annual Teaching Learning Forum, 2-4 February 2000. Perth: Curtin University of Technology.
[ TL Forum 2000 Proceedings Contents ]
[ TL Forums Index ]
HTML: Roger Atkinson, Teaching and Learning Centre, Murdoch University [email@example.com]
This URL: http://lsn.curtin.edu.au/tlf/tlf2000/santhanam2.html
Last revision: 20 Feb 2002. © Curtin University of Technology
Previous URL 29 Dec 1999 to 20 Feb 2002 http://cleo.murdoch.edu.au/confs/tlf/tlf2000/santhanam2.html