Teaching and Learning Forum 2013 Home Page

Category: Research

Teaching and Learning Forum 2013 [ Refereed papers ]
Learning from evaluations: Probing the reality

Dorothy Spiller and Trudy Harris
Teaching Development Unit, University of Waikato, New Zealand
Email: dorothy@waikato.ac.nz, scooby@waikato.ac.nz

This paper reports on findings from a major New Zealand research project around staff perceptions of student evaluations of teaching. The main focus of this discussion is the insights that the research afforded into staff engagement with and use of student evaluations to inform their teaching practice and to improve student learning. The research data indicates that there is a gap between academics' relatively positive views of evaluation and their actual engagement with the process. A high percentage of academics, particularly at NZ universities, do not report engagement in dialogue or professional development activities around student evaluation. The use of evaluations in teaching tends to be individual, often isolated and unsystematic. This paper recommends some ideas for moving forward and argues for a cultural change that endorses, supports and rewards the systematic integration of student evaluations into teaching practice.


Introduction

The current quality environment has led to a proliferation of surveys which, it may be claimed, is evidence of a focus on improving learners' experiences in higher education. However, cynics may be tempted to argue that the growing mountain of data collected through such quality processes is in danger of burying, or, at best, not influencing, the actual calibre of teaching and student learning. It was concern about one of these quality processes that provided the initial impetus for a New Zealand research study which was undertaken by a cross-institutional collaborative team that included the authors of this current paper. The research project looked at the engagement of academics with student feedback from formal institutional student evaluations. While the research team recognised that there are many other avenues for academics' professional development, the research team was concerned about the possibility of elaborate rituals of data collection that made little difference to the students' learning experiences. Equally they were concerned that students could become cynical and disaffected should they feel that their voices were not being heard. The study probed the perceptions that academics held of student evaluations and the degree to which they engaged with them in order to enhance student learning. The research was a nationally funded collaborative study undertaken by two New Zealand universities and one polytechnic. The findings of the research confirmed that, in keeping with the literature, academics were relatively positively disposed towards student evaluations, accepting them to varied degrees as a feature of contemporary academic life. The caveats that were raised also replicated the literature and included reservations about students' capacity to judge, reliability of evaluation instruments, fear of manipulation by academics and suspicion of institutional use of evaluations. Despite these qualms, the reported hostility towards student evaluations was not nearly as widespread and intense as the researchers had anticipated. However this finding was not really a cause for celebration, because acceptance of evaluations did not necessarily translate into engagement with feedback and its use for a process of continuous improvement of student learning.

This paper shares the findings of the New Zealand research and focuses on teachers' engagement with student feedback and the ways in which they incorporate student feedback into the teaching and learning cycle. Overall, our study suggested that there is not a culture of conversation, reflection or action that acknowledges the centrality of student feedback in the continuous improvement of teaching and learning. The evidence from the New Zealand research in these respects will be presented and discussed in this paper. Correspondingly, this paper also begins to articulate the next set of questions engendered by the New Zealand evaluations study. These questions open up the conversation about the kinds of changes that are necessary if student evaluation feedback is to play a pivotal role in enhancing the quality of higher education teaching and learning.

Formal student evaluations in New Zealand: Our reality

To begin with it is useful to document some of the distinguishing features of the NZ tertiary climate which frame this paper. In terms of national quality standards around teaching and learning within universities there is no controlling body equivalent to TEQSA (Tertiary Education Quality and Standards Agency) that sets standards for compliance. The only national measures of performance which is monitored by the Tertiary Education Commission (TEC) are the basic indicators of retention and completion. For the polytechnic sector the New Zealand Qualifications Authority (NZQA) establishes standards and monitors quality.

Correspondingly, higher education institutions in NZ have varied expectations around teaching development and student evaluations. In some instances, such as at the University of Waikato there is no formal probation system and no compulsory professional development, although there is a formal requirement to conduct student evaluations every two years and or for promotion purposes (as the sole indicator of teaching quality). The University of Otago has a probation process, no compulsory professional development, and student evaluations are required within a portfolio for promotion. By contrast, Otago Polytechnic requires all staff members to have teaching credentials and to undertake a student evaluation for all courses.

Myths or reality: The literature

While the current literature indicates relatively positive attitudes towards student evaluations, which appears to contradict the more commonly reported academic hostility towards the instruments, Beran and Rokosh (2009) and Burden (2008) remind us that the acceptance of the student evaluation scheme does not correlate with perceptions of their usefulness to enhance teaching, or with actual usage of the instrument for teaching changes. Beran and Rokosh (2009) speculate that "since instructors find ratings to be of little practical value, their seemingly positive attitudes regarding student ratings actually reflects a neutral view point or a passive acceptance of the ratings in general" (p. 183). Similarly, Smith (2008) notes that there is "little published evidence that they [evaluations] are systematically used for developing and improving their teaching" (p. 518). As Ory and Ryan (2001) contend, unless we can understand this failure to use evaluations to inform teaching and develop processes that help to remedy this, student evaluations will be little more than a ritual that both teachers and students participate in because it is compulsory.

The literature suggests that one of the barriers to teachers using student evaluation feedback are the difficulties associated with interpreting typical questionnaire data. Related to this concern is the fact that the evaluations process is often an isolated exercise and institutions generally provide very little guidance to staff in relation to the interpretation of student evaluations (Arthur, 2009). Penny and Coe (2004) cite the findings of Cohen's (1981) review which argues that students' ratings on their own were not enough to facilitate teaching improvements and that they needed to be supplemented by a consultation process. Penny and Coe (2004) outline key elements that they believe should be part of a consultation process. They propose that consultation with teachers around student evaluations and possible teaching improvements work best when other sources of evaluation are incorporated into the discussion. Specifically, they highlight the usefulness of incorporating self-ratings and peer feedback on teaching into the consultations. Another approach is mooted by Smith (2008), who suggests that an interpretative guidance system could help to combat the weak correlation between student evaluations and subsequent improvements to teaching.

The literature reports that along with difficulties in interpreting evaluations feedback, many academics indicate that they do not know how to act on the feedback provided in student evaluations. For example, in trying to account for the disparity between positive attitudes towards evaluations and limited use to inform teaching and learning, Centra (1993) argues that the most significant impediment to teaching improvement is that teachers do not know how to make the appropriate changes to their practice. In order to convert student evaluations into an integral and normal element of professional development, appropriate supporting institutional systems need to be put in place. Smith (2008) argues that for institutional purposes, managers generally focus on aggregate data, but improvements that really enhance the students' learning experience need to happen at the individual level. This is supported by the comments of Penny and Coe (2004) who argued that supporting consultation needs to be context and person specific. Bovill (2011) discusses the importance of thinking about evaluation for learning rather than of learning in her study, and argued for greater and more meaningful involvement of students in evaluation processes. Indeed the New Zealand study argues that:

The models of Ballantyne et al. (2006) and Smith (2008) both provide academic staff members with a degree of personal agency in the interpretation of evaluation results and associated professional development. Ownership is arguably a key component for any system that is designed to integrate student evaluation and professional development more effectively. As Arthur (2009) suggested, academics are "less likely to act on the findings of student feedback if it is collected and analysed centrally (for performativity purposes) because this divorces the findings from the context of teaching and learning" (p. 443). Arthur argued that a "performativity culture" reduces academics' sense of control and influence, and lecturers are more likely to see student evaluations as an imposed ritual. According to the typology he developed, academics in this culture are more likely to respond to negative evaluations with a "blame the students" reaction. This is because, Arthur contends, a performativity culture removes academics from agency for their own teaching. By contrast, according to Arthur, in cultures that emphasise academic professionalism, academics are more likely to modify aspects of their practice (Ôtame') or re-evaluate it. Discussions about interpretation of, and appropriate support for, subsequent development must therefore be conducted within the broader framework of the institutional use of evaluations data. (Stein et al. 2012, p.17)

Probing the reality: Findings

The first set of data on engagement with student evaluations is based on an online survey that included both quantitative and qualitative components This survey was completed by 1,065 (44%) academic staff members across the three institutions the University of Otago, the University of Waikato and Otago Polytechnic. A thematic analysis on the qualitative comments from the questionnaire provided insight into the quantitative data. The questionnaire findings and the literature were used to design the interview questions and to identify possible interviewees, twenty at each institution. A sample of interviewees was selected to provide a good range of participants in terms of academic disciplines, career stage and level of seniority. Thematic analysis was also used to analysis these findings.

Question 4 in the questionnaire specifically asked about teacher behaviours after receiving student evaluation feedback. At first glance, the questionnaire findings in the research suggest that those academics who find the student evaluations useful, see them as a way of informing their teaching. Of those who considered evaluations useful (73%), the most commonly cited reason for their usefulness was claimed to be "to inform teacher and course development" (19%), followed by the view that evaluations helped in "identifying students' learning needs" (19%). This apparently strong link between evaluation feedback and teaching and learning is more dubious when academics' actual use of evaluations information is investigated. A high percentage of academics report that they spend time reading the students' comments (95%), however, the percentages decline sharply in response to questions that investigate the practice of continuing conversation with colleagues (47%), students (16%) or undertaking professional development (12%) in relation to the feedback.

Interestingly, from learning and development perspective, the majority of teachers at all three institutions appear to deal with their evaluation data in isolation. Even for those teachers who use their evaluations systematically, the responses to Question 4 indicate that there appears to be less discussion with colleagues and/or their teaching teams about evaluation data.

Data from this question also shows that the lowest ranking is given to "seeking assistance with interpreting the results from others" (12%), a finding which suggests that relatively few academics see the evaluations as a springboard for subsequent professional development. Alarmingly, these findings indicate that for many academics the student evaluations are a discreet entity for private perusal and not central either to classroom practice, professional development or departmental cultures.

From the thematic analysis of the qualitative comments there was a recurrent theme concerning the interpretation of evaluation data and the need for tools and support systems to help with interpretation. This also link to an identified need for professional development in relation to student evaluations. Respondents did make suggestions about how the evaluations system could be used to enhance their usefulness for professional development. These include timing, the use of multiple forms of evaluation, enhanced flexibility and contextual appropriateness of the evaluation instruments. Additionally, suggestions were offered about the need for guidance with interpretation and use of feedback to enhance teaching, and for integrating education around evaluation into the professional development of academics. Examples of comments include the following:

A more collegial/academic mentor and 'professional development' model needs devising, in which student feedback is constructively sought from which teachers would be taught about.

The one thing I would do is include evaluation education sessions in our whole staff training days. I don't think all staff know how to access the evaluations or that they can modify them to evaluate specific things. I think it would also be good to educate staff about the professional development aspects of evaluations as I think lots of people perceive them as punitive things. I think that any punitive aspects should be played down - not helpful for anyone!

The engagement with student evaluation also stalls in terms of communication back to the students. Questionnaire respondents claimed that one of the main factors for not communicating with students about their feedback is that of timing. When most evaluation data becomes available to staff, the cohort of students who provided the feedback has moved on. For example:
Once students have completed an evaluation for my course, they move on and are not taught in this course again. I do talk with subsequent students about evaluation responses in general terms, but it is not a direct feedback loop to the same students who did the evaluation.
The interviews provided an opportunity to examine some of the themes that emerged from the survey in more depth. As with the survey, the interviews suggest that a substantial number of teachers take an interest in the implications of evaluations for the teaching and learning they provide. At both universities, over half of the interviewees report that they use student evaluations to inform their teaching to some degree. While this is a positive trend, there is a considerable range in the ways in which university interviewees spoke about how they engage with evaluation results and their processes vary from deliberate and systematic usage to a more cursory Ônod' to the student feedback. Only five interviewees at the University of Otago and four at the University of Waikato spoke about deliberate and relatively systematic responses they make to the student evaluation feedback they gathered. Examples of comments include:
I type up a list of the general comments that students make and act on those if I can... I'll often ask them to comment on things I'm trying or testing out, and I want to know if they're working.

I take their comments and use them as objectives for myself of things that I need to change and adjust...I ask myself, how can I transform this into a teaching objective?

The comments of the others who responded affirmatively to this question indicate a more piecemeal approach to using student feedback, although in this group there is also considerable variation in approaches. Comments include:
Looks at them reflectively and in comparison with past results ... it is just one little thing in amongst the huge amounts of teaching and administration and everything else I'm trying to do.

If several students come up with the same idea, then I try to incorporate it or if one student comes up with a brilliant idea, I try to incorporate it.

The respondents at the universities indicated a lack of engagement with evaluations feedback that ranged from apathy to negativity. Comments include:
Needs encouraging to go over them.

Doesn't use them to improve. Nothing very valuable in them. Vacuous. Written in ten minutes what you have thought about for years.

There was no strong evidence of a pattern of closing the loop around evaluations with very patchy reference to discussion with colleagues and feedback to students. Seven interviewees at the University of Otago and ten at the University of Waikato reported some degree of discussion with colleagues, but this is not a systematic and deliberate process. It would appear that peer conversation around student evaluations is not a normative and expected part of the culture. Comments include:
Informal meeting/discussion with colleagues when looking at the programme as a whole.

Sometimes shares information with other teachers-only informally.

But I tend to say the feedback was better or worse than last year, don't go into specifics that much.

Sometimes/informal/in the staffroom but not in meetings

The university interviewees' comments about feedback were in a similar vein to the questionnaire responses. Six interviewees at the University of Otago and six at the University of Waikato reported some conversation with their students about formal evaluations feedback. This feedback ranges from deliberate purposeful communication to informal chats. Two other points should be noted here. Many of those who report on feeding back to students do so with the next cohort of students. There was also another group who give ongoing feedback to students through regular informal evaluation. This group was particularly noticeable at the University of Waikato. Timing of the formal evaluations is again commonly reported as a drawback. Comments include:
Important that students believe the process is worthwhile. And that involves them knowing why they are doing it, what might happen as a result of it, and what that might mean to them.

Does let students know about changes in response to evaluations but not systematically or regularly.

I do discuss mid-semester informal appraisals. I present it to them graphed and categorised and say what we will do about it. I think that's most valuable because of the time it comes.

If there's stuff that's interesting and relevant, I usually discuss with incoming classes what I've learned from previous classes.

I get ongoing feedback just in the normal course of a lecture.. I make it clear to them that we do value feedback, that we're looking at ways to evaluate and improve their learning

There is a clear difference in the extent to which Otago Polytechnic interviewees say they engage with student feedback as compared with respondents from the two universities. Seventeen reported that they took account of the evaluation feedback provided by students and used it to modify their teaching. While the degree of engagement with student feedback and its incorporation into changes in course design and delivery varied considerably across the interviewees, there was a general recognition of this process as a necessary and constructive routine. Examples of comments include:
I use feedback to look for themes about learning styles and teaching methods. I use feedback to think about and adjust teaching to engage students in learning.

They are really helpful... useful... I actually will adapt my classes early on to meet as many of these styles that I can.

I see it as quite a continual process... if they are done earlier then there is better feedback and understanding between the students and the teacher.

In spite of the high number of interviewees who reported that they found student feedback useful for their teaching, only five of the Otago Polytechnic interviewees reported discussing their response to student feedback with the students. The relatively small number of academics who saw it as important to complete the feedback loop in this way was similar to the responses from the two universities. Furthermore, even those who did say they discuss their responses with students, did so primarily on the evaluation of courses, as opposed to the evaluation of teaching. Comments include:
If I get an evaluation that needs to be actioned, I will tell them this has occurred and this is what we are going to change.

On the whole I don't discuss them. The paper that I co-ordinate is in Semester 2. They have moved on to the next course.

As in the case of the universities, the timing of evaluations is cited as a significant reason for not communicating responses to feedback to students.

In conclusion, it appears that most teachers engage with the evaluations process, but to varying degrees. Many believe that collecting evaluation data is worthwhile, mainly for ongoing course refinements, and to receive feedback on the students' learning experiences. However, there is a small number of academic staff members who believe that the data that students provide can be biased, and based on poor judgment. There appears to be little feedback of evaluation information to students, mainly because of the timing of the process, but many staff do indicate that they use other forms of evaluation throughout their teaching.

In terms of professional development, many teachers do not actively seek help with using evaluation data, even though a small group of teachers spoke of problems with interpretation of the data. From the qualitative questionnaire comments, issues were identified with follow-up professional development processes, educating and supporting students around the evaluations process, and also staff engagement being reduced as a consequence of institutional restraints and requirements. It would appear that many teachers deal with their evaluation results in a somewhat isolated and haphazard way.

What is quite clear is the difference in the way that teachers from the polytechnic and the universities engage with the evaluations. The polytechnic's use of evaluations is part of a structured quality assurance process which focuses mainly on the provision of good teaching practice. The universities, meanwhile, have not had such well-defined structures or processes, possibly because their focus is spread more widely across different activities, particularly research.

Redefining the reality

The research findings around teacher engagement with evaluations at the universities indicate that staff members are generally accepting of formal student evaluations and many can see their potential for course and teaching improvements. However, the reality is that a culture of engaging with student evaluations for professional development and enhancing student learning is not embedded. This gap was highlighted by the contrast with the polytechnic where the majority of interviewees saw reference to the student evaluations as a normal and integral part of the teaching and learning cycle. In spite of this important distinction, the percentage of teachers at the polytechnic who report back to students on their response to evaluation feedback remains small. While listening to the students' voice appears to be more widely seen as important to the polytechnic teachers in this study, this is not matched by an institutional norm of reporting back to students.

The findings around engagement with student feedback in the New Zealand study have raised many questions about how the formal evaluations system can be made a more integral part of teaching and student learning, particularly in the universities. Questions have arisen about how instruments, processes and cultural changes could transform evaluations into a core teaching and learning tool. These changes need to occur at all levels.

At the institutional level, there needs to be a well-articulated statement of norms and expectations about the student evaluations and their use for professional development. These norms need to be communicated widely and regularly in all institutional forums and teaching and learning spaces. This research has indicated that without such clarity, individuals tend to do their own thing around evaluations. While there may be some very good individual practices, there are too many uncertainties which individuals tend to deal with in different ways and which are often influenced by personal history or higher education folklore. Complementing this institutional transparency and clarity about and support for the use of evaluations in professional development, institutions should make it obligatory for academics to demonstrate how they use student feedback and how this is communicated to students. Institutions should reward academics through the promotion processes for documenting how the respond to student evaluations for professional development purposes and demonstrating the accompanying changes they make in their practice. In keeping with this commitment to and highly public acknowledgement of the professional development purposes of student evaluations, institutions need to recognise and reward multiple forms of getting student feedback. Such changes are not only important for enhancement of individual practices, but can provide the essential framework for the continuous improvement of teaching and learning in the institution.

The need for the institution to clarify and communicate the expectation that evaluations be used for formative purposes has to be complemented by robust evaluation systems and processes that support this goal. These include addressing the timing issue, recognising other modes of evaluation initiated by academics and reviewing evaluation instruments for their pertinence, helpfulness and usefulness for enhancing the student learning experience. In terms of engagement, it is arguable that the review of the evaluation instruments should include consultation with both staff and students, who will thus have a greater sense of ownership of the evaluations process.

The literature and the research findings suggest that interpretation of evaluations is often a random, ad hoc affair. To make this stage of the teaching and learning cycle as productive for learning as possible a number of interventions could help. In line with Smith (2008), a set of interpretative guidelines could be developed to aid this stage of the process and more institutional resources dedicated to providing help with interpretation and the implementation. The comments of university academics suggest that education around the learning possibilities offered by student evaluations is necessary. More generally, it could be argued that compulsory professional development around teaching (that could occur in different ways, such as peer coaching or teaching conversational circles) could help to equip academics with the pedagogical language and tools to make better use of student feedback. These education processes can never be stand alone remedies and the institutional commitment to the use of evaluations to inform teaching and learning needs to be matched at all levels of the organisation. The institution should require discussion and documentation of the use of student evaluation feedback at the department, course and programme level. Such a requirement could help to place student evaluations at the centre of reviewing and planning for teaching and learning.

Corresponding to staff education around evaluations, the students, the partners in the dialogue need to be given ongoing education about ways of providing feedback and coaching in the process of meaningful evaluation. As critical evaluation and making judgments are widely stated as generic goals of higher education, specific guidance around these attributes should already be embedded into most curricula. Coaching students in the application of these skills and dispositions to the evaluation process should not be difficult. However, the best way of engaging students in thoughtful evaluation is by demonstrating that their views are taken account of in the teaching and learning.

Conclusion

These suggestions may seem grandiose and a new phase of research is required to examine the best possible strategies for creating these different components and communicating their value to academic staff. The goal is to build a supportive framework around the currently somewhat tenuous link between evaluations and the quality of the student learning experience. Each part of this suggested framework has potential significance, but all the elements need to work together to build a structure that works towards the common goal of enhanced teaching and learning.

References

Arthur, L. (2009). From performativity to professionalism: Lecturers' responses to student feedback. Teaching in Higher Education, 14(4), 441-454. http://dx.doi.org/10.1080/13562510903050228

Ballantyne, R., Borthwick, J. & Packer, J. (2000). Beyond student evaluation of teaching: Identifying and addressing academic staff development needs. Assessment and Evaluation in Higher Education, 25(3), 221-236. http://dx.doi.org/10.1080/713611430

Beran, T. N. & Rokosh, J. L. (2009). Instructors' perspectives on the utility of student ratings of instruction. Instructional Science, 37(2), 171-184. http://dx.doi.org/10.1007/s11251-007-9045-2

Bovill, C. (2011). Sharing responsibility for learning through formative evaluation: moving to evaluation as learning. Practice and Evidence of the Scholarship of Teaching and Learning in Higher Education, 6(2), 96-109. http://www.pestlhe.org.uk/index.php/pestlhe/article/view/112/222

Burden, P. (2008). Does the use of end of semester evaluation forms represent teachers' views of teaching in a tertiary education context in Japan? Teaching and Teacher Education, 24(6), 1463-1475. http://dx.doi.org/10.1016/j.tate.2007.11.012

Centra, J. A. (1993). Reflective faculty evaluation. San Francisco: Jossey-Bass.

Ory, J. C. & Ryan, K. (2001). How do student ratings measure up to a new validity framework? New Directions for Institutional Research, 109, 27-44. http://dx.doi.org/10.1002/ir.2

Penny, A. & Coe, R. (2004). Effectiveness of consultation on student ratings feedback: A meta-analysis. Review of Educational Research, 74(2), 215-253. http://dx.doi.org/10.3102/00346543074002215

Smith, C. (2008). Building effectiveness in teaching through targeted evaluation and response: Connecting evaluation to teaching improvement in higher education. Assessment and Evaluation in Higher Education, 33(5), 517-533. http://dx.doi.org/10.1080/02602930701698942

Stein, S. J., Spiller, D., Terry, S., Deaker, L., Harris, T. K. & Kennedy, J. (2012). Unlocking the impact of tertiary teacher's perceptions of student evaluation. Ako Aotearoa, New Zealand. http://akoaotearoa.ac.nz/project/impact-student-evaluations-teaching-behaviour/resources/files/unlocking-impact-tertiary-teachers-perceptions-student-eva

Authors: Dorothy Spiller, Senior Lecturer
Teaching Development Unit, Centre for Tertiary Teaching and Learning
University of Waikato, Hamilton, New Zealand

Dr Trudy Harris, Senior Academic Staff Member (Mechanical Engineering)
School of Engineering Science and Primary Industries
Waikato Institute of Technology, Hamilton, New Zealand
(formerly University of Waikato)

Please cite as: Spiller, D. & Harris, T. (2013). Learning from evaluations: Probing the reality. In Design, develop, evaluate: The core of the learning environment. Proceedings of the 22nd Annual Teaching Learning Forum, 7-8 February 2013. Perth: Murdoch University. http://ctl.curtin.edu.au/professional_development/conferences/tlf/tlf2013/refereed/spiller.html

Copyright 2013 Dorothy Spiller and Trudy Harris. The authors assign to the TL Forum and not for profit educational institutions a non-exclusive licence to reproduce this article for personal use or for institutional teaching and learning purposes, in any format, provided that the article is used and cited in accordance with the usual academic conventions.


[PDF version] [Refereed papers] [Contents - All Presentations] [Home Page]
This URL: http://ctl.curtin.edu.au/professional_development/conferences/tlf/tlf2013/refereed/spiller.html
Created 7 Feb 2013. Last revision: 13 Feb 2013.