|Teaching and Learning Forum 2009 [ Refereed papers ]|
Lina Pelliccione, Kathryn Dixon, Lou Siragusa, Christine Howitt, Bill Atweh, Alma Dender, Jillian Swaine
Curtin University of Technology
University of Technology Sydney
Government pressure for accountability in matters relating to comprehensive student evaluation of learning is now more overt with funding incentives for compliance included within these reform agendas. Gauging the quality of teaching on student perception alone is a cause for concern. This paper introduces the notion of 'peer review' as an additional mechanism to help gauge the quality of teaching with the aim of improving the learning environment and thus making the learning experiences more valuable to each student. The authors of this paper are part of a larger Australian Learning and Teaching Council (ALTC) Funded Project examining peer review of learning and teaching in blended learning environments. The paper will report on the progress of one institutional team, in particular their individual rationale for participating in such a project. Even though these are early days, there are a number of positive indicators which signal the success of the peer review process in blended learning environments.
According to the former Federal Minister of Education, Science and Training, Dr. Brendan Nelson (DEST, 2004), Australian higher education needs to improve quality, enhance competitiveness in the field while facilitating critical thinking and research. One of Dr Nelson's proposals was for universities to establish systematic student evaluation of teaching and learning which should be made publicly available via the World Wide Web.
Government pressure for accountability in matters relating to comprehensive student evaluation of learning is now more overt with funding incentives for compliance included within these reform agendas. As Ramsden (2003, p. 211) stated "[e]valuation for accountability has become an essential part of today's university". Increasingly, universities are being encouraged to focus on students' learning rather than on teaching alone, with many exploring feedback mechanisms which focus more on students' perception of the learning that occurred and student development rather than what the teacher/lecturer did. The environment for tertiary education is one in which "prospective students are focusing on course quality and likely employment outcomes in making their selection" (Ronayne, 1999, p. 8). Gauging the quality of teaching on student perception alone is a cause for concern as close examination of some of these evaluation tools reveals the ambiguity of certain items, therefore making the data difficult to interpret accurately. This paper introduces the notion of 'peer review' as an additional mechanism to help gauge the quality of teaching with the aim of improving the learning environment and thus making the learning experiences more valuable to each student.
The technique of peer review is also referred to as 'peer observation', 'peer coaching', 'peer learning' or 'peer pairing'. The issues of quality assurance and accountability, once again, are driving the agenda (Marshall, 2004; McMahon, Barrett & O'Neill, 2007). However, a number of authors believe that the greatest value of peer review is the potential to improve teaching and learning practices (Bennett & Barp, 2008; Hammersley-Fletcher & Orsmond; 2004). Lomas and Nicholls (2005) make a clear distinction between engaging in the peer review process for quality enhancement and engaging in the process for quality assurance. The research conducted by Shortland (2004) as well as McMahon, Barrett and O'Neill (2007) reveals that teaching staff engage more actively with the peer review process when the focus is solely developmental.
Gosling (2002) identified three models of peer observation of teaching (Table 1) as a way of recognising that "When someone observes another teaching there are many factors which influence the success of the activity (p.1)".
|Characteristic||Evaluation model||Development model||Peer review model|
|Who does it and to whom?||Senior staff observe other staff||Educational developers observe practitioners; or expert teachers observe others in department||Teachers observe each other|
|Purpose||Identify under-performance, confirm probation, appraisal, promotion, quality assurance, assessment||Demonstrate competency/improve teaching competencies; assessment||Engagement in discussion about teaching; self and mutual reflection|
|Outcome||Report/judgement||Report/action plan; pass/fail PGCert||Analysis, discussion, wider experience of teaching methods|
|Status of evidence||Authority||Expert diagnosis||Peer shared perception|
|Relationship of obs-erver to observed||Power||Expertise||Equality/mutuality|
|Confidentiality||Between manager, observer and staff observed||Between observer and the observed, examiner||Between observer and the observed - shared within learning set|
|Inclusion||Selected staff||Selected/ sample||All|
|Judgement||Pass/fail, score, quality assessment, worthy/unworthy||How to improve; pass/fail||Non-judgemental, constructive feedback|
|What is observed?||Teaching performance||Teaching performance, class, learning materials||Teaching performance, class, learning materials|
|Who benefits?||Institution||The observed||Mutual between peers|
|Conditions for success||Embedded management processes||Effective central unit||Teaching is valued, discussed|
|Risks||Alienation, lack of co-operation, opposition||No shared ownership, lack of impact||Complacency, conservatism, unfocused|
According to Gosling (2002) some of the contributing factors to the success of peer reviews are housed under the following key terms: peer; observation; and teaching. A peer within any given setting can include a number of different relationships (colleagues from the same/different departments, of similar or unequal status). Depending on the purpose of the judgement being made the social relationship of power and authority of the peer will impact on the interaction and ultimately on the level of learning derived from the peer review experience (Gosling, 2002, p. 2). The impact of the interaction will further be exacerbated when the peer adopts the role as an expert and the observer does not fully trust the expert. Confidentiality must also be guaranteed. Gosling (2002) also advocates shifting away from the 'one-way model' as identified in the Evaluation and Developmental models to ensure that all involved learn from the experience.
The process of observation forces the participants to focus on what is visible and thus, missing many of the key elements of what is not visible. Understandably, the experience and knowledge of the observer strongly influences their observations. Observation can be recorded using a number of methods, such as checklists, observation schedules, interactions, or verbal cues. For developmental purposes, Gosling advocates informal recording, which means simply recording what actually occurs. Placing an observer within any given context will naturally impact the learning environment and alter what is being observed. Gosling (2002, p. 3) summarises the concerns regarding the use of observation by stating, "observing is not a neutral process, it is influenced by circumstances, the method of observation as well as what the observer brings to the event".
Finally, Gosling (2002) looks at how the observer's conceptions and assumptions about teaching influence the process. The observation focuses on the act of performing. However, there is a great deal to teaching that is not easy to observe. He also notes that other evidence such as assessment processes, student feedback and learning outcomes need to be collected in order to make an informed decision. The peer review process needs to go beyond what is observable to encompass curriculum design, support and assessment. The review process needs to encourage greater dialogue between colleagues into the perceptions and beliefs of what quality teaching really looks like.
The project has been funded for two years and is using co-productive approaches involving a core project team of one academic staff member from each of the five ATNs, with a cross-institutional group of six members within each University (see Figure 1 on the following page). Each team will develop, trial and analyse a framework for evaluating evidence of how and what students learn and how teaching supports this learning in blended learning environments. Case studies will illustrate the framework in different disciplines and teams will pilot and evaluate the framework using peer reviews within and across institutions (McKenzie, 2007).
Figure 1: Project teams
They then paired up with colleagues they were very familiar with, in the same field as themselves and ultimately who they trusted to be their peer reviewer. This approach was invaluable as the literature indicates the peer review process can be very challenging and confronting. A total of six academic staff became members of the institutional team: two staff from Humanities (1 female, 1 male); two from Science and Engineering (1 female, 1 male); and two from Health (2 females).
During the first institutional team meeting a number of key benefits for participating in the project were highlighted. These were identified by the project team leader and the project officer from the University of Technology Sydney, and were distributed to all participants. The anticipated benefits were: exchanging ideas with others within and across universities; contributing to understandings of good teaching and learning in blended learning environments; improving students' learning; support for documenting and communicating good practices; creating evidence for recognition and reward; development and recognition of contributions to leadership in teaching and learning; and opportunities for presentations and publications (Bernstein, Burnett, Goodburn & Savory, 2006).
The first few meetings involved familiarising everyone with the project brief and making clear the expectations and involvement in the project, getting to know each other and discussing the definitions and dimensions of peer review and blended learning. Once this was established the focus was turned to the peer review framework. This framework is underpinned by the six broad standards of scholarly work developed through the Carnegie Foundation (Glassick, Huber & Maeroff, 1997): clear goals; adequate preparation; appropriate methods and their implementation; effective communication; significant results; and reflective critique. This framework is currently being trialled by the institutional team to provide feedback during the peer review process. In addition, prior to the actual peer review, each team member completed a case study overview which provided a context and purpose for the review being conducted. The case study overview contained the following dimensions: unit title; unit description; unit outcomes; focus of the peer review; intentions for teaching in the subject; overview; and using a peer review as evidence. Unit plans and guides were also included. In order to facilitate this process the team were asked to document their general purpose for participating in this peer review project. The next section reveals their individual responses.
I have been very interested to gain useful feedback from a teaching colleague regarding my teaching in the online unit ***. The major reason for this is that so few of my colleagues in the School of *** teach in fully online modes and because of this my opportunities to discuss my methods and approaches to teaching and designing instructional materials for online delivery are extremely limited. Student feedback on teaching methods and the overall design of the unit is collected every semester through the systemic feedback mechanism used in this university known as - eVALUate. This of course provides reasonable information regarding the achievement of outcomes and the perceived quality of the contents and assessment protocols used in the unit from the student perspective, however as the students are not experts in instructional design for online environments the overall quality of the feedback is limited. Through the peer review process I am hoping to gain significant feedback from not only a respected peer but also from a recognised researcher in the field of online instructional design. This will be a valuable experience for me as my colleague is experienced in online teaching himself as well as an active researcher. He is aware of the difficulties and drawbacks of teaching in online environments as well as the multitude of benefits this mode of delivery offers students, in particular adult learners. His feedback will provide me with a multi-layered view of the design of the unit, the opportunities I have provided for the students to communicate within the environment, the clarity of language and explanations used in the unit and insights into the success or otherwise of the students' motivation to engage in and complete the assessment requirements. Colleague1: Senior Lecturer
With my increasing use of blended learning environments in higher education, it is important to ensure that the teaching and learning strategies developed and facilitated for these environments are appropriate for effective student learning. I am wishing, therefore, to be a part of the Peer Review Project in order to critically examine my use of these environments and to reflect upon the feedback received from peers. It is planned that this examination and reflection process will result in the revision of my teaching practices to ensure that they are appropriate for blended learning environments in order to enhance student learning and motivation. The Peer Review Project will also assist with determining the possible applicability of blended learning approaches to other learning situations. Colleague 2: Lecturer
I relocated from *** to Australia a year ago to lecture at Curtin University in the School of ***. The language and cultural differences between *** and Australia may not appear outwardly disparate but they are profound in many ways. This is more pronounced in the area of ***, specifically, ***. This project provides me with an opportunity to receive structured feedback from an Australian colleague who has immense experience and skill in teaching *** in Australia. In addition, I am able to learn a new peer review model by which I can provide feedback to my colleagues. I have a keen interest in life long learning, especially in how to teach more effectively to meet the needs of students. Peer review is another tool to use to share teaching strategies between colleagues. Colleague 3: Lecturer
The unit has been taught in this manner for two years and the eVALUate suggests that while many students appreciate the blended learning environment, some do not. The purpose therefore of the review is to examine the materials used, specifically the i-lecture and the integrated workbook to examine its efficacy in providing stimulating and enriched learning in a way that suits students of the X and Y Generations. In this way anecdotal evidence can be supported with more objective means, in conjunction to the comments made in eVALUate. ... The review would also evidence reflective teaching practice which I believe is critical. The review is three fold in that it gives feedback from another informed and experienced academic's point of view, especially where the reviewer's knowledge of the subject matter is well-established. The second point is that the review itself shows that as a teacher I am open and willing to have critical appraisal of my teaching and my knowledge of a unit which is foundational to many areas of *** practice. Thirdly in these days of heightened awareness of accountability to the profession, the students and the institution, as well as ultimately to our clients and consumers, a review may make some impact on keeping the unit relevant. Colleague 4: Lecturer
This unit has been designed based on a blended learning environment where external and internal students have equal access to all material and pedagogy. I have designed this unit in 2007 and it is in its second trial this year. A feedback on alignment of aims, content, pedagogy and assessment will be very useful in further development of the unit as well as providing evidence for curriculum design in the Centre where the majority of students are at a distance. ... Very infrequently do academics have a chance to obtain feedback from their colleagues on their curriculum design, teaching and assessment. I have been lecturing in *** for over 20 years now. Although I have always sought feedback from students and modified my courses accordingly, only once I sought feedback on my unit designs from a colleague. I realise that this is an important contribution to my professional development as a lecturer in ***. Often I spoke with school teachers about peer relations as a source of improvement of their teaching as a professional development. I believe in this project I will have a chance to learn from the experience and demonstrate its effectiveness. Colleague 5: Associate ProfessorTable 2 provides a summary of the key reasons identified by the institutional team regarding their rationale for participating in this peer review of blended learning project.
As a reflective practitioner I am continually searching for ways to enhance my student learning. I consider student learning the overarching goal of all education, at any level. However, hand-in-hand with this goal is the search for ways to improve the quality of my teaching. I see this peer review process as a third lens, alongside personal reflection and student feedback, from which to obtain constructive, collegial and professional feedback on my teaching and subsequently student learning. Colleague 6: Lecturer
|Key factor||Individual responses per key factor|
|Obtain feedback from a respected and experienced source||gain significant feedback from not only a respected peer but also from a recognised researcher in the field of online instructional design. ... my colleague is experienced in online teaching himself (1)|
opportunity to receive structured feedback from an Australian colleague who has immense experience and skill in teaching (3)
feedback from another informed and experienced academic's point of view, especially where the reviewer's knowledge of the subject matter is well-established (4)
Very infrequently do academics have a chance to obtain feedback from their colleagues on their curriculum design, teaching and assessment (5)
Obtain constructive, collegial and professional feedback (6)
|Improve quality of teaching and student learning||the revision of my teaching practices to ensure that they are appropriate for blended learning environments in order to enhance student learning and motivation (2)|
teach more effectively to meet the needs of students. (3)
Improve the quality of my teaching (6)
Enhance student learning (6)
|Obtain additional feedback to feedback given by students||...however as the students are not experts in instructional design for online environments the overall quality of the feedback is limited (1)|
anecdotal evidence can be supported with more objective means, in conjunction to the comments made in eVALUate (4)
I have always sought feedback from students and modified my courses accordingly, only once I sought feedback on my unit designs from a colleague (5)
Third lens, alongside personal reflection and student feedback (6)
|Reflective practice||reflect upon the feedback received from peers. (2)|
also evidence of reflective teaching practice (4)
to learn from the experience and demonstrate its effectiveness (5)
|Critically examine the use of blended learning environments||I have been very interested to gain useful feedback from a teaching colleague regarding my teaching in the online unit (1)|
...teaching and learning strategies developed and facilitated for these environments are appropriate for effective student learning (2)
examine the materials used, specifically the i-lecture and the integrated workbook to examine its efficacy in providing stimulating and enriched learning in a way that suits students of the X and Y Generations(4)
|Share teaching strategies||Peer review is another tool to use to share teaching strategies between colleagues (3)|
|Provide feedback to colleagues||learn a new peer review model by which I can provide feedback to my colleagues (3)|
|Professional development||open and willing to have critical appraisal (4)|
Contribute to Professional development (5)
|Keeping current||A review may make some impact on keeping the unit relevant.(4)|
Even though these are early days in the peer review project there are a number of positive signs which signal the success of the peer review process in blended learning environments. The institutional team are participating in this project on a voluntary basis and have selected peers they trust and whose judgement and expertise they respect which alleviates many of the concerns voiced by Gosling (2002) regarding unequal status and power and authority. Each pair will be engaged in mutual reviews and therefore they will share a respect for their partner's privacy with regard to the outcomes of the peer review process. The data presented earlier revealed that each team member has a very clear purpose for participating in the peer review process, with common themes revolving around improved teaching and student learning. The peer review framework and the case study overview will enable each reviewer the opportunity to gather a range of evidence as it is the belief of this project team, that the review process needs to extend beyond the observable and performance aspect of teaching to include curriculum design, support and assessment. Gosling (2002) notes that one of the conditions for successful implementation of peer review, is that teaching and learning is valued and openly discussed and debated. It would appear that the institutional team involved in this project are suitably prepared for the next stage of the peer review project as they have clearly demonstrated their commitment and enthusiasm to improve the quality of their teaching and subsequent student learning.
Bernstein, D., Burnett, A., Goodburn, A. & Savory, P. (2006). Making teaching and learning visible: Course Portfolios and the peer review of teaching. Bolton, Mass. USA: Anker Publishing Co. Ltd.
DEST (n.d.). Our universities: Backing Australia's future, Minister's Message. [viewed 31 Jan 2004] http://www.backingaustraliasfuture.gov.au/ministers_message.htm
Glassick, C., Huber, M. & Maeroff, G. (1997). Scholarship assessed: Evaluation of the professoriate (A special report). San Francisco, US: The Carnegie Foundation for the Advancement of Teaching.
Gosling, D. (2002). Models of peer observation of teaching. LTSN Generic Centre. [viewed 21 Oct 2008] http://www.heacademy.ac.uk/resources/detail/id200_Models_of_Peer_Observation_of_Teaching
Hammersley-Fletcher, L. & Orsmond, P. (2004). Evaluating our peers: Is peer observation a meaningful process? Studies in Higher Education, 29(4), 489-503.
Lomas, L., & Nicholls, G. (2005). Enhancing teaching quality through peer review of teaching, Quality in Higher Education, 11(2), 137-149.
Marhsall, B. (1994). Learning from the academy: From peer observation of teaching to peer enhancement of learning and teaching. Journal of Adult Theological Education, 1(2), 185-204.
McKenzie, J. (2007). Embedding peer review of learning and teaching in e-learning and blended learning environments. Carrick Institute, Grants Scheme 2007, Application Form.
McMahon, T., Barrett, T. & O'Neill, G. (2007). Using observation of teaching to improve quality: Finding your way through the muddle of competing conceptions, confusion of practice and mutually exclusive intentions. Teaching in Higher Education, 12(4), 499-511.
Ramsden, P. 2003. Learning to teach in higher education, 2nd ed. London: Routledge Falmer.
Ronayne, J. 1999. The environment for tertiary education and training (Message from the Vice Chancellor and President) Victoria University of Technology, Australia.
Shortland, S. (2004). Peer observation: A tool for staff development or compliance? Journal of Further and Higher Education, 28(2), 219-28.
|Authors: Lina Pelliccione, Kathryn Dixon, Lou Siragusa, Christine Howitt, Bill Atweh, Alma Dender, Jillian Swaine|
Curtin University of Technology
Jo McKenzie, University of Technology Sydney
Email: email@example.com, firstname.lastname@example.org, email@example.com, firstname.lastname@example.org, email@example.com, firstname.lastname@example.org, email@example.com, Jo.McKenzie@uts.edu.au
Please cite as: Pelliccione, L., Dixon, K., Siragusa, L., Howitt, C., Atweh, B., Dender, A., Swaine, J. & McKenzie, J. (2009). Academic peer review: Enhancing learning environments for global graduates. In Teaching and learning for global graduates. Proceedings of the 18th Annual Teaching Learning Forum, 29-30 January 2009. Perth: Curtin University of Technology. http://otl.curtin.edu.au/tlf/tlf2009/refereed/pelliccione.html
Copyright 2009 Lina Pelliccione, Kathryn Dixon, Lou Siragusa, Christine Howitt, Bill Atweh, Alma Dender, Jillian Swaine and Jo McKenzie. The authors assign to the TL Forum and not for profit educational institutions a non-exclusive licence to reproduce this article for personal use or for institutional teaching and learning purposes, in any format (including website mirrors), provided that the article is used and cited in accordance with the usual academic conventions.