Teaching and Learning Forum 2010 Home Page

Category: Professional practice

Teaching and Learning Forum 2010 [ Refereed papers ]
Evidence of quality: The teaching criteria framework at UWA

Jacqueline Flowers
The University of Western Australia

Sustainability in higher education can be defined in the approach we take to teaching our students - by using sustainable practices in our definitions of good teaching, and incorporating them in to the expectations we have for teaching staff, we can create a sustainable teaching and learning environment. In 2008, as part of UWA's involvement in the national ALTC Teaching Quality Indicators (TQI) project a new framework of teaching criteria was developed to better support good quality teaching and learning within the institution, and enhance the value placed on teaching by enabling more robust developmental and evaluative systems for promoting, recognising and rewarding good teaching.

The framework is premised on a shared understanding of good teaching which has been developed through extensive consultation across the University throughout the project, and which is based in the research evidence about the types of teaching practice which improve student learning outcomes. The project has developed this shared definition of good teaching, along with providing a structure around which to talk about teaching, and the associated development of examples and guidelines of sources and types of evidence staff can collect which can demonstrate the achievement of good teaching. The new teaching criteria framework underpins the teaching component of the academic portfolio, and will therefore inform all of the University's reward and recognition processes from developmental annual review discussions, through to promotion applications. By robustly defining what is meant by good teaching, and assisting staff to understand how to demonstrate it, the framework may be able to improve the value placed on teaching as an academic endeavour and go some way to supporting sustainable education.

This paper describes the process UWA went through in developing the new framework, the way that good teaching has been defined in the UWA context, and how the resulting framework is being used to enhance and support good teaching practice.


Introduction

Sustainability in higher education can be defined in the approach we take to teaching our students - by using sustainable practices in our definitions of good teaching, and incorporating them in to the expectations we have for teaching staff, and by ensuring that the way we evaluate, reward and recognise high quality teaching is based in the evidence of the types of teaching that assist learning, we can create a sustainable teaching and learning environment.

In 2008 the University of Western Australia (UWA), developed a new framework of teaching criteria designed to underpin the teaching component of the academic portfolio and provide a robust, clear, and consistent definition of good teaching which was appropriate to the UWA context and grounded in the research. The resulting teaching criteria framework aims to provide guidance to both academic teaching staff and their supervisors in evaluating teaching quality - providing a consistent language with which to talk about the teaching that we do, clarifying the types and sources of required evidence and a framework within which both developmental and summative discussions can be held.

This paper commences with a brief background to the project, before discussing the development of a University-wide definition of good teaching, followed by a description of the process of developing the teaching criteria framework to support that definition, the mechanisms which can be used to demonstrate good teaching, and the way in which the framework will be used at UWA to inform the reward and recognition of teaching and teachers.

Background - the Teaching Quality Indicators Project

In 2007 - 2008 the University of Western Australia was a pilot institution in a national Australian Learning and Teaching Council (ALTC) project "Teaching Quality Indicators" (TQI). Subtitled "Rewarding and recognising quality teaching in higher education through systematic implementation of indicators and metrics on teaching and teacher effectiveness" the TQI project set out to develop and implement a framework that identifies indicators and outcomes of teaching quality at the institutional and individual levels. The framework that resulted from that project is primarily methodological in nature, identifying dimensions of teaching likely to affect quality, the levels at which evaluation can drive change, and the types of indicators that in balance can assist institutions to more robustly evaluate their practice (Chalmers, 2007).

The TQI framework consists of four dimensions which have been identified as affecting teaching quality:

The framework suggests appropriate indicators (input, process, output and outcome) at various levels of a University (institution, faculty, department and individual) for each of the dimensions.

As a pilot institution, UWA chose to focus its efforts for implementing the framework in the dimension of 'Institutional Climate and Systems', and in particular on the promotion, recognition and reward of teaching. The research indicates that exposure of students to good practices in undergraduate education is positively related to gains in student learning (Kuh, Pace & Vesper, 1997); that by encouraging good teaching, good learning results. The evidence also shows that although traditionally teaching quality has been left up to the individual academic teacher, and good teaching considered to be a personal trait that cannot be taught (Keig & Waggoner, 1994); in fact the institutional culture of a university, and the way that teaching and learning is valued by that university and systematically promoted, recognised and rewarded does have an impact on the quality of the teaching which results (Chalmers, 2007). Good teaching is not mysterious, there is a large body of research about good teaching practices in higher education, and it is possible to measure the quality of teaching that our staff are achieving - for example, evidence of a student-centred learning approach is a strong indicator of teaching quality (Chalmers, 2007).

Recognising a lack of clarity in existing evaluative systems for teaching and learning at the individual level, and alongside other projects aimed at developing institutional level indicators, the University set out to develop a new set of teaching criteria to enable evaluation of teaching at the individual level, for developmental and summative purposes, which could apply to all staff regardless of discipline and be used consistently across the University.

Introducing a definition of good teaching quality

Key to any focus on 'promotion, recognition and reward' of teaching, was a shared understanding of what good teaching is - what is it that we want to promote, recognise and reward? Once we had defined good teaching, equally important for the project was to ask how we could measure it. A review of the University's current guidelines for the development of portfolios, professional development review processes, and promotion criteria revealed a lack of consistency in the messages to staff relating to expectations for their teaching, and a lack of clarity regarding the way that criteria would be assessed, in particular relating to the lack of an explicit sliding-scale of expectations by career level. In addition, it was felt that teaching evaluations, both for developmental and summative purposes were over-reliant on self-reflection, with little emphasis given to alternative means of evaluating teaching such as peer review.

A review of the literature reveals broad consensus over a significant period of time about the types of teaching practices which are more likely to lead to learning, and the ways in which they can be evaluated. From the Seven Principles of Good Teaching (Chickering & Gamson, 1987) in the late eighties, through Boyer's introduction of the concept of 'scholarship' in teaching (Boyer, 1990) and expressed in Australia through the ALTC Teaching Award criteria, the same principles arise again and again. Although the "list" of good teaching practices is long, and necessarily diverse, it can be summarised as follows, noting that the references provided here are examples only of a large body of research. Good (sustainable) teaching is student centred and culturally inclusive (Chalmers, 2007), it engages students with the subject matter through active learning (Braxton, 2006); provides prompt and meaningful feedback (Chickering & Gamson, 1987) provides students with learning outcomes that are aligned with assessment (Biggs, 2003) and is enhanced when the teacher is engaged with the scholarship of teaching and learning (Boyer, 1990).

The research undertaken for this project identified four main instruments available for evaluating the quality of teaching - student evaluations (Barrie et al 2007); student learning (Chalmers, 2007); peer review; and self-reflection (Atwood, 2000). All four are valid and important ways of measuring the success of an individual's teaching and bringing about improvement; each may be suited to the evaluation of different aspects of teaching practice, and has its own strengths and weaknesses. It is therefore important not to rely on a single evaluation tool, but to use a number of instruments to measure quality (Barrie et al, 2007) and ensure that a wide range of practices are evaluated, from classroom practice to teaching and learning materials, and student learning outcomes. This recognition that a robust evaluation requires evidence not only from a number of different sources, but also different types of sources, is one of the main principles arising out of the TQI project, which attempts to strike a balance between qualitative and quantitative measures of quality (Chalmers, 2007).

Developing a UWA framework

The TQI pilot project at UWA used a predominantly research-based iterative methodology, consisting of a cycle of evidence gathering, mapping against the TQI framework, drafting, consultation and review, for each of its sub-projects. This consultative, investigative methodology was used to try and maximise engagement and commitment from university stakeholders, and whilst labour intensive, was successful in obtaining at least some comment on the proposals from a large number of teaching and learning leaders across the University. The proposal went through a number of consultative phases throughout the project, involving faculty forums, one on one feedback, workshops for Heads and Deans, and the involvement of the Teaching and Learning Committees. The development of the teaching criteria framework took place in the context of a wider review of the University's Professional Development Review (PDR) process, and from this a discussion across the University relating to standards for each element of the academic role and the University's purpose - research, teaching and service. The TQI projects' work on defining and evaluating good teaching was incorporated in to these larger discussions about academic standards and evaluating academic performance, and the resulting teaching criteria framework is now being used across the University to inform such evaluation.

Once good teaching had been defined through the review of the literature, an initial consultation period was undertaken with the University community to clarify the scope of the project, and set out a series of principles upon which any evaluative system for teaching and learning should be based. This enabled the project to be placed specifically in the UWA context, and gave stakeholders the opportunity to contribute not only to a robust definition of quality, but also to the approach the University should take in its efforts to better promote, recognise and reward good teaching. A number of principles were set out for any evaluative system, which emphasised that the quality of teaching was an institutional responsibility, not just an individual one; that evaluation of quality should be systematic, robust and comparative; criteria used for teaching needed to be consistent across all mechanisms, and based in the current research; that criteria need to be contextualised by discipline, detailed but not prescriptive, differentiated by level, and enable developmental approaches to enhancement. At this stage, a number of desired outcomes for the project were also articulated - that robust teaching criteria and evaluative systems would raise the status of teaching in the institution; improve teaching quality; and enable the identification of strengths and weaknesses, and targeted professional development.

Following this consultation period further research was undertaken to identify a number of alternative models for an evaluative system at UWA. A desk-top audit of practices at other Australian institutions was undertaken, along with the identification of innovations internationally. A number of approaches in particular to criteria used for promotion processes were identified within the Australian context, and from this it was proposed that a model was required which combined an aligned structure across the University (consistent definitions of quality for all purposes) with the use of standards to differentiate between career levels. Different measures (qualitative and quantitative) could be used as evidence to demonstrate how each of the qualitative criteria were being met, to the level set out in the relevant standard. This schema is supported by the research which suggests that evaluative systems which use a qualitative approach incorporating a developmental cycle are most effective in driving improvement (Garlick & Pryor, 2004).

In identifying the need to implement an evaluative system which provided a consistent set of expectations for all purposes it was proposed to develop a new framework for staff preparing the teaching component of the academic portfolio. By concentrating on reform at this level, it would be possible to devise a set of University wide expectations and definitions of quality which would permeate all of the reward and recognition processes associated with teaching - from developmental discussions with supervisors and identification of professional development opportunities through to formal annual review processes, probation and tenure requirements and promotion applications, the same underlying principles, expectations and definitions of quality could be used consistently across the University and would translate externally.

The research undertaken for this stage of the project had identified the UK Higher Education Academy's (HEA) Professional Standards Framework as one whose underlying concepts, and also its practical definitions of good teaching quality, aligned well with the University's perspective and focus. Rather than developing a whole new system, it was therefore proposed to use the HEA framework as the basis for developing a new set of criteria for UWA.

The UK HEA Professional Standards Framework was developed as a way of accrediting professional development programmes for teaching in the UK and also as part of the professional recognition scheme which confers fellow status on individual higher education teachers. There are three standard descriptors each of which is applicable to a number of staff roles and to different career stages of those engaged in teaching and supporting learning. The standard descriptors are underpinned by areas of professional activity, core knowledge and professional values. The framework provides a reference point for institutions and individuals as well as supporting ongoing development within any one standard descriptor. (HEA, UK website, 2009)

A further extensive consultation period was undertaken involving academic leaders at the University (Deans, Heads of School, Associate Deans (Teaching and Learning) through which the Professional Standards Framework was re-worked for the UWA and Australian context, and a version of the framework developed to form the basis for the teaching component of the academic portfolio to underpin all evaluative processes relating to teaching at UWA. This consultation period was invaluable in determining how to structure the framework, how to guide its use and what emphasis was appropriate in the UWA context in relation to teaching expectations for academic staff. This consultative period revealed the extent of discipline variation in expectations for teaching staff, and led to the very flexible nature of the resulting framework.

The UWA Teaching Criteria Framework

The resulting UWA Teaching Criteria Framework consists of six areas of activity, within which staff demonstrate their knowledge and understanding of six types of core knowledge, and six professional values through the evidence that they present. The activities focus on design and planning, teaching and supporting learning, assessment and feedback, student support, integrating scholarship, research and professional activities, and continuous evaluation and development. Within each of these activities, staff demonstrate an understanding of their subject material, appropriate methods, student learning processes, appropriate use of technologies, and evaluative methods. And demonstrate commitment to the professional values of respect for learners, development of communities of learning, encouraging participation, acknowledging and extending understanding of cultural diversity (including indigenous knowledges), and to continuing professional development (UWA Teaching Criteria Framework website, 2009).

More substantial evidence is expected of staff as they move through the career levels, and for each area of activity examples of the types and sources of evidence which may be appropriate are provided to assist staff in developing their portfolios.

To be useful in a practical sense, the framework would need to be able to be contextualised by disciplines across the University. The types of activities staff are engaged in at different levels of their career varies significantly across different disciplines, which was revealed when consultation took place on the examples of sources / types of evidence for each area of activity. Faculties and Schools of the University have been invited to contextualise the framework to their own needs - to emphasise some activities over others, or to define which types of evidence would be expected at different levels of an academics' career. This flexibility within the framework means that it can be used to develop standards of teaching that are meaningful to the individual academic teaching bodies, whilst retaining a consistent definition of what the University expects of its teachers. All schools and faculties of the University are engaged in a process of defining standards in their discipline contexts, not just for teaching and learning but for all aspects of the University's role (teaching, research, service) and the teaching criteria framework is now informing that process for teaching by providing schools and faculties with a consistent structure within which to define discipline specific standards, a common language to talk about expectations, and examples of the kinds of measures which can be used not just at the individual academic level, but also for departmental level evaluations.

In addition to the framework itself, standard descriptors for each academic career level set out general expectations for the teaching component of an academic staff member's role. These descriptors support the Minimum Standards for Academic Levels (MSAL) by providing more specific quality expectations (rather than defining types of activities), and also support the UWA Promotion criteria by providing guidance on the way that the promotion criteria relating to 'high quality teaching' can be demonstrated at an appropriate level. The standard descriptors retain the UK Professional Standards Frameworks' emphasis on the student learning experience, and the teaching-research nexus, as these relate directly to the UWA mission and objectives. Each teaching standard describes expectations in terms of teaching quality, commitment to continuous development, and leadership (UWA TCF website).

Types of evidence

The framework identifies four main mechanisms for providing evidence about teaching quality - student learning; student satisfaction; peer review (broadly defined); and self-evaluation. Student evaluations have long been recognised in Australian institutions as a key indicator for measuring teaching quality, and are relied on more heavily in Australia than is the case internationally (Barrie et al, 2007). Recent work for the TQI project by Barrie et al suggests that such evaluations are (as is the case with any evaluation system) only useful when used in combination with other measures, robustly analysed and their limitations accepted (Barrie et al, 2007).

There are some aspects of teaching that cannot be evaluated effectively by students, for example student evaluations cannot provide an assessment of elements such as the teaching content, pedagogical content and ethical standards of practice - these dimensions of quality teaching are best assessed by colleagues (Keig & Waggoner, 1994) and this is where the use of peer review in evaluating teaching quality is most appropriate. In this context peer review is interpreted broadly and includes the review of learning materials; observations of teaching; mentoring arrangements and teaching communities (Atwood et al, 2000). In addition, the TQI framework identifies evidence like testimonials from colleagues on the same curriculum team or committee as further legitimate peer review. The recognition that peer review is an important aspect of teacher evaluations has been gradual, however following the introduction of the new Teaching Criteria Framework, the University has now also formalised a set of Guidelines for Peer Review in the form of a good practice guide after a number of years of discussion and development. Whilst the University does not require staff to use peer review as a teaching evaluation method, it does strongly encourage its use.

Self-evaluation, incorporating the principle of self-reflective practice, remains an important aspect of evaluation, and is crucial in the process of interpreting the other evidence gathered in to a coherent analysis of a staff member's teaching practice within the framework structure.

Evidence of student learning is perhaps the least-utilised measure of teaching quality at present. Such evidence includes external moderation of assessment tasks, external marking / moderation of student work (including of the quality of student attainment of understanding of key discipline concepts), accepted tests of student learning (e.g. pre- and post-testing), or official university statistics (e.g. for research student completions). By giving specific examples of the types of evidence that may be appropriate when demonstrating particular aspects of each teaching activity, it is hoped that the framework will assist staff in using a wider range of evidence to evaluate their teaching and in particular encourage the use of direct evidence of learning.

What to collect?

The guidelines for the development of an academic portfolio at UWA recognise that the purpose for which the folio is being presented is important in determining the evidence that will be presented as part of that portfolio. For developmental annual reviews a staff member may need to present a summary of all of their activities in the review period; whereas for a promotion application the staff member would present a selection of evidence which supported a particular claim to high achievement. Different again is the portfolio used to support a nomination for an award, where the evidence presented will make a specific case for excellence against external criteria. The teaching criteria framework and its suggested sources of evidence encourage staff to collect information about their teaching over a whole career, to gradually build a portfolio of evidence about the quality of the teaching that they do which will then allow them to build a case for any purpose which may be required and also facilitate developmental discussions about their teaching by showing sequential change. The examples of sources and types of evidence provided in the framework appear dauntingly large - there are so many things a staff member might do which demonstrates good teaching, however if considered over a whole career, with different types of evidence suiting different purposes, the framework provides structure and guidance to staff about what to collect and why, actually simplifying the process.

Conclusion: Implementation and feedback

Whilst implementation of the framework is in the early stages, there are signs that its aims to make the process of evaluating teaching quality simpler and more robust for staff and reviewers will be achieved. Extensive feedback received throughout the development and consultation process associated with finalising the framework indicates that it can assist staff in understanding expectations around teaching, and providing clarity around the process of reflecting on, and talking about, teaching.

Because the framework was developed through an extensive consultation process, feedback from staff has already been incorporated in to the final framework, and most of the issues raised have been addressed. For example, the framework incorporates innovation as a core principle for quality teaching, following concern that using a categorising framework as an evaluative tool had the potential to limit innovative teaching practices. Concerns around workload issues remain, however through comparison with current requirements for teaching portfolios and other evaluative tasks such concerns can be minimised. The framework has its limitations, it can look daunting and complex, but may in fact over simplify expectations around teaching, and this a reflection of an attempt to provide a University-wide framework whilst also providing guidelines and examples that staff can use. The framework will be monitored over time to evaluate its impact on enhancing the sustainability of the student learning experience offered at UWA.

Implementation of the new criteria is staggered, with staff only interacting with the new framework when they come to revise or create their academic portfolio, and staff will not be expected to have a portfolio which aligns to the new framework for a number of years. Over time more staff will go through the process of engaging with the framework, collecting different types of evidence to support their claims for good quality teaching, engaging in professional development around developing a portfolio, and discussing the standard descriptors with their supervisors. Similarly, schools and faculties will go through a process of contextualising the framework for their discipline, and using it to assist in framing departmental level evaluations of teaching quality. As these activities occur across the University over the next couple of years it is hoped that the way in which teaching is discussed and considered within the University will evolve, and assist in the process of enhancing the value placed on teaching within the institution.

Whilst the project's longer term objectives of improving teaching quality remain to be seen, the process of defining good teaching in the UWA context and identifying robust tools for its evaluation is seen as worthwhile in and of itself. The ultimate goal is to develop a culture where good teaching is seen as an activity that is valued, respected, rewarded and recognised across the institution, and only by achieving this will we achieve the goal of sustainable education.

References

Atwood et al, (2000). "Why are Chemists and other scientists afraid of the peer review of teaching?" Journal of Chemical Education, 77:2, February 2000.

Barrie, Ginns & Symons (2008). "Student Surveys on Teaching and Learning: Final Report", Institute for Teaching and Learning, University of Sydney, for the Carrick Institute of Learning and Teaching in Higher Education, Retrieved 14 October 2009, from http://www.catl.uwa.edu.au/page/146823

Braxton, J.M. (2006). "Faculty Professional Choices in Teaching that Foster Student Success" National Postsecondary Education Cooperative, Retrieved 14 October 2009, from http://nces.ed.gov/npec/pdf/Braxton_report.pdf

Chalmers, D. (2007), "A review of Australian and international quality systems and indicators of learning and teaching" Carrick Institute for Learning and Teaching in Higher Education, Retrieved 13 October 2009, from http://www.catl.uwa.edu.au/__data/page/146827/T&L_Quality_Systems_and_Indicators.pdf

Garlick S. & Pryor G. (2004). Benchmarking the university: Learning about Improvement A report for the Department of Education, Science and Training, Retrieved 15 October 2009 from DEEWR Publication and Resources website: http://www.dest.gov.au/sectors/higher_education/publications_resources/profiles/benchmarking_the_university.htm

Higher Education Academy, UK (undated). Professional Standards Framework, accessed on 22 October 2009 http://www.heacademy.ac.uk/ourwork/policy/framework

Keig, L. & Waggoner, M.D. (1994). Collaborative peer review: The role of faculty in improving college teaching, ASHE-ERIC Higher Education Report No. 2. http://www.eric.ed.gov:80/ERICWebPortal/contentdelivery/servlet/ERICServlet?accno=ED378925

Kuh, G. D., Pace, R. C., & Vesper, N. (1997). The development of process indicators to estimate student gains associated with good practices in undergraduate education. Research in Higher Education, 38(4), 435-454.

The University of Western Australia, Teaching Criteria Framework website, accessed on 22 October 2009 http://www.catl.uwa.edu.au/tcf

Author: Jacqueline Flowers, Centre for Advancement of Teaching and Learning, University of Western Australia. Email: Jacqueline.flowers@uwa.edu.au

Please cite as: Flowers, J. (2010). Evidence of quality: The teaching criteria framework at UWA. In Educating for sustainability. Proceedings of the 19th Annual Teaching Learning Forum, 28-29 January 2010. Perth: Edith Cowan University. http://otl.curtin.edu.au/tlf/tlf2010/refereed/flowers.html

Copyright 2010 Jacqueline Flowers. The author assigns to the TL Forum and not for profit educational institutions a non-exclusive licence to reproduce this article for personal use or for institutional teaching and learning purposes, in any format, provided that the article is used and cited in accordance with the usual academic conventions.


[ Refereed papers ] [ Contents - All Presentations ] [ Home Page ]
This URL: http://otl.curtin.edu.au/tlf/tlf2010/refereed/flowers.html
Created 22 Jan 2010. Last revision: 22 Jan 2010.