Teaching and Learning Forum 2009 Home Page

Category: Research
Teaching and Learning Forum 2009 [ Refereed papers ]
Academic peer review: Enhancing learning environments for global graduates

Lina Pelliccione, Kathryn Dixon, Lou Siragusa, Christine Howitt, Bill Atweh, Alma Dender, Jillian Swaine
Curtin University of Technology
Jo McKenzie
University of Technology Sydney

Government pressure for accountability in matters relating to comprehensive student evaluation of learning is now more overt with funding incentives for compliance included within these reform agendas. Gauging the quality of teaching on student perception alone is a cause for concern. This paper introduces the notion of 'peer review' as an additional mechanism to help gauge the quality of teaching with the aim of improving the learning environment and thus making the learning experiences more valuable to each student. The authors of this paper are part of a larger Australian Learning and Teaching Council (ALTC) Funded Project examining peer review of learning and teaching in blended learning environments. The paper will report on the progress of one institutional team, in particular their individual rationale for participating in such a project. Even though these are early days, there are a number of positive indicators which signal the success of the peer review process in blended learning environments.


Introduction

Since the late 1980s Australian universities have been grappling with the ongoing problems associated with measuring and making effective judgements relating to the quality of teaching. Performance indicators in higher education in relation to quality teaching began to be developed in the early 1990s and it was recognised that assessing teaching quality was a matter of judgement which involved a complex pattern of behaviour.

According to the former Federal Minister of Education, Science and Training, Dr. Brendan Nelson (DEST, 2004), Australian higher education needs to improve quality, enhance competitiveness in the field while facilitating critical thinking and research. One of Dr Nelson's proposals was for universities to establish systematic student evaluation of teaching and learning which should be made publicly available via the World Wide Web.

Government pressure for accountability in matters relating to comprehensive student evaluation of learning is now more overt with funding incentives for compliance included within these reform agendas. As Ramsden (2003, p. 211) stated "[e]valuation for accountability has become an essential part of today's university". Increasingly, universities are being encouraged to focus on students' learning rather than on teaching alone, with many exploring feedback mechanisms which focus more on students' perception of the learning that occurred and student development rather than what the teacher/lecturer did. The environment for tertiary education is one in which "prospective students are focusing on course quality and likely employment outcomes in making their selection" (Ronayne, 1999, p. 8). Gauging the quality of teaching on student perception alone is a cause for concern as close examination of some of these evaluation tools reveals the ambiguity of certain items, therefore making the data difficult to interpret accurately. This paper introduces the notion of 'peer review' as an additional mechanism to help gauge the quality of teaching with the aim of improving the learning environment and thus making the learning experiences more valuable to each student.

Peer review in blended learning environments

Peer review in simple terms is the process of making judgements about the quality of learning and teaching which usually involves a colleague observing/examining a learning experience and learning environment and providing feedback. This feedback leads to reflection and discussion, with the ultimate aim of improving student learning. Even though peer review is becoming more common practice in higher education, more so in the UK, there has been very little application to e-learning or blended learning (a mix between online and face to face) environments. The application of peer review has typically been applied within the boundaries of traditional classrooms. Marshall (2004) advocates the need to recognise the changing nature of learning environments and the various roles academic staff have within such environments.

The technique of peer review is also referred to as 'peer observation', 'peer coaching', 'peer learning' or 'peer pairing'. The issues of quality assurance and accountability, once again, are driving the agenda (Marshall, 2004; McMahon, Barrett & O'Neill, 2007). However, a number of authors believe that the greatest value of peer review is the potential to improve teaching and learning practices (Bennett & Barp, 2008; Hammersley-Fletcher & Orsmond; 2004). Lomas and Nicholls (2005) make a clear distinction between engaging in the peer review process for quality enhancement and engaging in the process for quality assurance. The research conducted by Shortland (2004) as well as McMahon, Barrett and O'Neill (2007) reveals that teaching staff engage more actively with the peer review process when the focus is solely developmental.

Gosling (2002) identified three models of peer observation of teaching (Table 1) as a way of recognising that "When someone observes another teaching there are many factors which influence the success of the activity (p.1)".

Table 1: Models of peer observation of teaching (Gosling, 2002, p. 5)

CharacteristicEvaluation modelDevelopment modelPeer review model
Who does it and to whom?Senior staff observe other staffEducational developers observe practitioners; or expert teachers observe others in departmentTeachers observe each other
PurposeIdentify under-performance, confirm probation, appraisal, promotion, quality assurance, assessmentDemonstrate competency/improve teaching competencies; assessmentEngagement in discussion about teaching; self and mutual reflection
OutcomeReport/judgementReport/action plan; pass/fail PGCertAnalysis, discussion, wider experience of teaching methods
Status of evidenceAuthorityExpert diagnosisPeer shared perception
Relationship of obs-erver to observedPowerExpertiseEquality/mutuality
ConfidentialityBetween manager, observer and staff observedBetween observer and the observed, examinerBetween observer and the observed - shared within learning set
InclusionSelected staffSelected/ sampleAll
JudgementPass/fail, score, quality assessment, worthy/unworthyHow to improve; pass/failNon-judgemental, constructive feedback
What is observed?Teaching performanceTeaching performance, class, learning materialsTeaching performance, class, learning materials
Who benefits?InstitutionThe observedMutual between peers
Conditions for successEmbedded management processesEffective central unitTeaching is valued, discussed
RisksAlienation, lack of co-operation, oppositionNo shared ownership, lack of impactComplacency, conservatism, unfocused

According to Gosling (2002) some of the contributing factors to the success of peer reviews are housed under the following key terms: peer; observation; and teaching. A peer within any given setting can include a number of different relationships (colleagues from the same/different departments, of similar or unequal status). Depending on the purpose of the judgement being made the social relationship of power and authority of the peer will impact on the interaction and ultimately on the level of learning derived from the peer review experience (Gosling, 2002, p. 2). The impact of the interaction will further be exacerbated when the peer adopts the role as an expert and the observer does not fully trust the expert. Confidentiality must also be guaranteed. Gosling (2002) also advocates shifting away from the 'one-way model' as identified in the Evaluation and Developmental models to ensure that all involved learn from the experience.

The process of observation forces the participants to focus on what is visible and thus, missing many of the key elements of what is not visible. Understandably, the experience and knowledge of the observer strongly influences their observations. Observation can be recorded using a number of methods, such as checklists, observation schedules, interactions, or verbal cues. For developmental purposes, Gosling advocates informal recording, which means simply recording what actually occurs. Placing an observer within any given context will naturally impact the learning environment and alter what is being observed. Gosling (2002, p. 3) summarises the concerns regarding the use of observation by stating, "observing is not a neutral process, it is influenced by circumstances, the method of observation as well as what the observer brings to the event".

Finally, Gosling (2002) looks at how the observer's conceptions and assumptions about teaching influence the process. The observation focuses on the act of performing. However, there is a great deal to teaching that is not easy to observe. He also notes that other evidence such as assessment processes, student feedback and learning outcomes need to be collected in order to make an informed decision. The peer review process needs to go beyond what is observable to encompass curriculum design, support and assessment. The review process needs to encourage greater dialogue between colleagues into the perceptions and beliefs of what quality teaching really looks like.

Methodology

The authors of this paper are part of a larger Australian Learning and Teaching Council (ALTC) Funded Project titled "Embedding peer review of learning and teaching in e-learning and blended learning environments". The aim of the larger Project is to develop, implement, evaluate and embed a scholarly framework, processes and resources for peer review of learning and teaching in blended learning environments, and embed a peer review process for recognising and rewarding good teaching across the Australian Technology Network (ATN) universities. The project is intended to make valuable contributions to improving learning and teaching in e-learning environments, informing decisions about academic performance, promotion and teaching awards for those who teach in these environments, and enabling benchmarking of processes and outcomes across institutions (McKenzie, 2007, p. 6).

The project has been funded for two years and is using co-productive approaches involving a core project team of one academic staff member from each of the five ATNs, with a cross-institutional group of six members within each University (see Figure 1 on the following page). Each team will develop, trial and analyse a framework for evaluating evidence of how and what students learn and how teaching supports this learning in blended learning environments. Case studies will illustrate the framework in different disciplines and teams will pilot and evaluate the framework using peer reviews within and across institutions (McKenzie, 2007).

Selection of participants

Encouraging participation in this Peer Review Project initially appeared daunting. However, after a few emails to key people who were well known for their interest in teaching and learning an institutional team was formed at Curtin University of Technology. Primarily, only three people were approached and asked whether they had a colleague in their department who would also be interested in the project and who they would like to work with.

Figure 1

Figure 1: Project teams

They then paired up with colleagues they were very familiar with, in the same field as themselves and ultimately who they trusted to be their peer reviewer. This approach was invaluable as the literature indicates the peer review process can be very challenging and confronting. A total of six academic staff became members of the institutional team: two staff from Humanities (1 female, 1 male); two from Science and Engineering (1 female, 1 male); and two from Health (2 females).

During the first institutional team meeting a number of key benefits for participating in the project were highlighted. These were identified by the project team leader and the project officer from the University of Technology Sydney, and were distributed to all participants. The anticipated benefits were: exchanging ideas with others within and across universities; contributing to understandings of good teaching and learning in blended learning environments; improving students' learning; support for documenting and communicating good practices; creating evidence for recognition and reward; development and recognition of contributions to leadership in teaching and learning; and opportunities for presentations and publications (Bernstein, Burnett, Goodburn & Savory, 2006).

The first few meetings involved familiarising everyone with the project brief and making clear the expectations and involvement in the project, getting to know each other and discussing the definitions and dimensions of peer review and blended learning. Once this was established the focus was turned to the peer review framework. This framework is underpinned by the six broad standards of scholarly work developed through the Carnegie Foundation (Glassick, Huber & Maeroff, 1997): clear goals; adequate preparation; appropriate methods and their implementation; effective communication; significant results; and reflective critique. This framework is currently being trialled by the institutional team to provide feedback during the peer review process. In addition, prior to the actual peer review, each team member completed a case study overview which provided a context and purpose for the review being conducted. The case study overview contained the following dimensions: unit title; unit description; unit outcomes; focus of the peer review; intentions for teaching in the subject; overview; and using a peer review as evidence. Unit plans and guides were also included. In order to facilitate this process the team were asked to document their general purpose for participating in this peer review project. The next section reveals their individual responses.

Motivating factors for academic involvement

Colleagues 1 and 2: Humanities

I have been very interested to gain useful feedback from a teaching colleague regarding my teaching in the online unit ***. The major reason for this is that so few of my colleagues in the School of *** teach in fully online modes and because of this my opportunities to discuss my methods and approaches to teaching and designing instructional materials for online delivery are extremely limited. Student feedback on teaching methods and the overall design of the unit is collected every semester through the systemic feedback mechanism used in this university known as - eVALUate. This of course provides reasonable information regarding the achievement of outcomes and the perceived quality of the contents and assessment protocols used in the unit from the student perspective, however as the students are not experts in instructional design for online environments the overall quality of the feedback is limited. Through the peer review process I am hoping to gain significant feedback from not only a respected peer but also from a recognised researcher in the field of online instructional design. This will be a valuable experience for me as my colleague is experienced in online teaching himself as well as an active researcher. He is aware of the difficulties and drawbacks of teaching in online environments as well as the multitude of benefits this mode of delivery offers students, in particular adult learners. His feedback will provide me with a multi-layered view of the design of the unit, the opportunities I have provided for the students to communicate within the environment, the clarity of language and explanations used in the unit and insights into the success or otherwise of the students' motivation to engage in and complete the assessment requirements. Colleague1: Senior Lecturer

With my increasing use of blended learning environments in higher education, it is important to ensure that the teaching and learning strategies developed and facilitated for these environments are appropriate for effective student learning. I am wishing, therefore, to be a part of the Peer Review Project in order to critically examine my use of these environments and to reflect upon the feedback received from peers. It is planned that this examination and reflection process will result in the revision of my teaching practices to ensure that they are appropriate for blended learning environments in order to enhance student learning and motivation. The Peer Review Project will also assist with determining the possible applicability of blended learning approaches to other learning situations. Colleague 2: Lecturer

Colleagues 3 and 4: Health

I relocated from *** to Australia a year ago to lecture at Curtin University in the School of ***. The language and cultural differences between *** and Australia may not appear outwardly disparate but they are profound in many ways. This is more pronounced in the area of ***, specifically, ***. This project provides me with an opportunity to receive structured feedback from an Australian colleague who has immense experience and skill in teaching *** in Australia. In addition, I am able to learn a new peer review model by which I can provide feedback to my colleagues. I have a keen interest in life long learning, especially in how to teach more effectively to meet the needs of students. Peer review is another tool to use to share teaching strategies between colleagues. Colleague 3: Lecturer

The unit has been taught in this manner for two years and the eVALUate suggests that while many students appreciate the blended learning environment, some do not. The purpose therefore of the review is to examine the materials used, specifically the i-lecture and the integrated workbook to examine its efficacy in providing stimulating and enriched learning in a way that suits students of the X and Y Generations. In this way anecdotal evidence can be supported with more objective means, in conjunction to the comments made in eVALUate. ... The review would also evidence reflective teaching practice which I believe is critical. The review is three fold in that it gives feedback from another informed and experienced academic's point of view, especially where the reviewer's knowledge of the subject matter is well-established. The second point is that the review itself shows that as a teacher I am open and willing to have critical appraisal of my teaching and my knowledge of a unit which is foundational to many areas of *** practice. Thirdly in these days of heightened awareness of accountability to the profession, the students and the institution, as well as ultimately to our clients and consumers, a review may make some impact on keeping the unit relevant. Colleague 4: Lecturer

Colleagues 5 and 6: Science and Engineering

This unit has been designed based on a blended learning environment where external and internal students have equal access to all material and pedagogy. I have designed this unit in 2007 and it is in its second trial this year. A feedback on alignment of aims, content, pedagogy and assessment will be very useful in further development of the unit as well as providing evidence for curriculum design in the Centre where the majority of students are at a distance. ... Very infrequently do academics have a chance to obtain feedback from their colleagues on their curriculum design, teaching and assessment. I have been lecturing in *** for over 20 years now. Although I have always sought feedback from students and modified my courses accordingly, only once I sought feedback on my unit designs from a colleague. I realise that this is an important contribution to my professional development as a lecturer in ***. Often I spoke with school teachers about peer relations as a source of improvement of their teaching as a professional development. I believe in this project I will have a chance to learn from the experience and demonstrate its effectiveness. Colleague 5: Associate Professor

As a reflective practitioner I am continually searching for ways to enhance my student learning. I consider student learning the overarching goal of all education, at any level. However, hand-in-hand with this goal is the search for ways to improve the quality of my teaching. I see this peer review process as a third lens, alongside personal reflection and student feedback, from which to obtain constructive, collegial and professional feedback on my teaching and subsequently student learning. Colleague 6: Lecturer

Table 2 provides a summary of the key reasons identified by the institutional team regarding their rationale for participating in this peer review of blended learning project.

Table 2: Summary of key factors

Key factorIndividual responses per key factor
Obtain feedback from a respected and experienced sourcegain significant feedback from not only a respected peer but also from a recognised researcher in the field of online instructional design. ... my colleague is experienced in online teaching himself (1)
opportunity to receive structured feedback from an Australian colleague who has immense experience and skill in teaching (3)
feedback from another informed and experienced academic's point of view, especially where the reviewer's knowledge of the subject matter is well-established (4)
Very infrequently do academics have a chance to obtain feedback from their colleagues on their curriculum design, teaching and assessment (5)
Obtain constructive, collegial and professional feedback (6)
Improve quality of teaching and student learningthe revision of my teaching practices to ensure that they are appropriate for blended learning environments in order to enhance student learning and motivation (2)
teach more effectively to meet the needs of students. (3)
Improve the quality of my teaching (6)
Enhance student learning (6)
Obtain additional feedback to feedback given by students...however as the students are not experts in instructional design for online environments the overall quality of the feedback is limited (1)
anecdotal evidence can be supported with more objective means, in conjunction to the comments made in eVALUate (4)
I have always sought feedback from students and modified my courses accordingly, only once I sought feedback on my unit designs from a colleague (5)
Third lens, alongside personal reflection and student feedback (6)
Reflective practicereflect upon the feedback received from peers. (2)
also evidence of reflective teaching practice (4)
to learn from the experience and demonstrate its effectiveness (5)
Critically examine the use of blended learning environmentsI have been very interested to gain useful feedback from a teaching colleague regarding my teaching in the online unit (1)
...teaching and learning strategies developed and facilitated for these environments are appropriate for effective student learning (2)
examine the materials used, specifically the i-lecture and the integrated workbook to examine its efficacy in providing stimulating and enriched learning in a way that suits students of the X and Y Generations(4)
Share teaching strategiesPeer review is another tool to use to share teaching strategies between colleagues (3)
Provide feedback to colleagueslearn a new peer review model by which I can provide feedback to my colleagues (3)
Professional developmentopen and willing to have critical appraisal (4)
Contribute to Professional development (5)
Keeping currentA review may make some impact on keeping the unit relevant.(4)

Concluding comments

Not surprisingly the key factors have been attributed to obtaining feedback from a respected and experienced peer for the ultimate purpose of improving the quality of their teaching and thus impacting on student learning outcomes. Some of the institutional team members also noted the need to gather additional feedback from sources other than their students. Colleague 1 made it very clear that the students were unable to provide the level of feedback she required on her online unit for her to determine whether she had designed the best possible learning environment. She noted that an instructional design expert would be the best possible person to judge the quality of her teaching in such an environment. Interestingly, only one person commented on the opportunity to provide feedback to their peers.

Even though these are early days in the peer review project there are a number of positive signs which signal the success of the peer review process in blended learning environments. The institutional team are participating in this project on a voluntary basis and have selected peers they trust and whose judgement and expertise they respect which alleviates many of the concerns voiced by Gosling (2002) regarding unequal status and power and authority. Each pair will be engaged in mutual reviews and therefore they will share a respect for their partner's privacy with regard to the outcomes of the peer review process. The data presented earlier revealed that each team member has a very clear purpose for participating in the peer review process, with common themes revolving around improved teaching and student learning. The peer review framework and the case study overview will enable each reviewer the opportunity to gather a range of evidence as it is the belief of this project team, that the review process needs to extend beyond the observable and performance aspect of teaching to include curriculum design, support and assessment. Gosling (2002) notes that one of the conditions for successful implementation of peer review, is that teaching and learning is valued and openly discussed and debated. It would appear that the institutional team involved in this project are suitably prepared for the next stage of the peer review project as they have clearly demonstrated their commitment and enthusiasm to improve the quality of their teaching and subsequent student learning.

References

Bennett, S. & Barp, D. (2008). Peer observation - a case for doing it online. Teaching in Higher Education, 13, (5), 559-570.

Bernstein, D., Burnett, A., Goodburn, A. & Savory, P. (2006). Making teaching and learning visible: Course Portfolios and the peer review of teaching. Bolton, Mass. USA: Anker Publishing Co. Ltd.

DEST (n.d.). Our universities: Backing Australia's future, Minister's Message. [viewed 31 Jan 2004] http://www.backingaustraliasfuture.gov.au/ministers_message.htm

Glassick, C., Huber, M. & Maeroff, G. (1997). Scholarship assessed: Evaluation of the professoriate (A special report). San Francisco, US: The Carnegie Foundation for the Advancement of Teaching.

Gosling, D. (2002). Models of peer observation of teaching. LTSN Generic Centre. [viewed 21 Oct 2008] http://www.heacademy.ac.uk/resources/detail/id200_Models_of_Peer_Observation_of_Teaching

Hammersley-Fletcher, L. & Orsmond, P. (2004). Evaluating our peers: Is peer observation a meaningful process? Studies in Higher Education, 29(4), 489-503.

Lomas, L., & Nicholls, G. (2005). Enhancing teaching quality through peer review of teaching, Quality in Higher Education, 11(2), 137-149.

Marhsall, B. (1994). Learning from the academy: From peer observation of teaching to peer enhancement of learning and teaching. Journal of Adult Theological Education, 1(2), 185-204.

McKenzie, J. (2007). Embedding peer review of learning and teaching in e-learning and blended learning environments. Carrick Institute, Grants Scheme 2007, Application Form.

McMahon, T., Barrett, T. & O'Neill, G. (2007). Using observation of teaching to improve quality: Finding your way through the muddle of competing conceptions, confusion of practice and mutually exclusive intentions. Teaching in Higher Education, 12(4), 499-511.

Ramsden, P. 2003. Learning to teach in higher education, 2nd ed. London: Routledge Falmer.

Ronayne, J. 1999. The environment for tertiary education and training (Message from the Vice Chancellor and President) Victoria University of Technology, Australia.

Shortland, S. (2004). Peer observation: A tool for staff development or compliance? Journal of Further and Higher Education, 28(2), 219-28.

Authors: Lina Pelliccione, Kathryn Dixon, Lou Siragusa, Christine Howitt, Bill Atweh, Alma Dender, Jillian Swaine
Curtin University of Technology
Jo McKenzie, University of Technology Sydney
Email: l.pelliccione@curtin.edu.au, k.dixon@curtin.edu.au, l.siragusa@curtin.edu.au, c.howitt@curtin.edu.au, b.atweh@curtin.edu.au, a.dender@curtin.edu.au, j.swaine@curtin.edu.au, Jo.McKenzie@uts.edu.au

Please cite as: Pelliccione, L., Dixon, K., Siragusa, L., Howitt, C., Atweh, B., Dender, A., Swaine, J. & McKenzie, J. (2009). Academic peer review: Enhancing learning environments for global graduates. In Teaching and learning for global graduates. Proceedings of the 18th Annual Teaching Learning Forum, 29-30 January 2009. Perth: Curtin University of Technology. http://otl.curtin.edu.au/tlf/tlf2009/refereed/pelliccione.html

Copyright 2009 Lina Pelliccione, Kathryn Dixon, Lou Siragusa, Christine Howitt, Bill Atweh, Alma Dender, Jillian Swaine and Jo McKenzie. The authors assign to the TL Forum and not for profit educational institutions a non-exclusive licence to reproduce this article for personal use or for institutional teaching and learning purposes, in any format (including website mirrors), provided that the article is used and cited in accordance with the usual academic conventions.


[ Refereed papers ] [ Contents - All Presentations ] [ Home Page ]
This URL: http://otl.curtin.edu.au/tlf/tlf2009/refereed/pelliccione.html
Created 20 Jan 2009. Last revision: 20 Jan 2009.