Category: Professional practice
|Teaching and Learning Forum 2009 [ Refereed papers ]|
Centre for the Advancement of Teaching and Learning
Faculty of Arts, Humanities and Social Sciences
Diana Jonas-Dwyer and Sandra Carr
Faculty of Medicine, Dentistry and Health Sciences
Sue Miller and Coral Pepper
Faculty of Natural and Agricultural Sciences
Faculty of Law
Faculty of Engineering, Computing and Mathematics
The University of Western Australia
This paper describes the efforts of a group of academic developers at The University of Western Australia (UWA) working collegially to address an issue identified as a teaching and learning priority, namely the provision of improved formative feedback to students to enhance their learning outcomes. It takes a case study approach, emphasising both generic and context specific issues that arose. By taking similar but contextually different approaches to the problem the study is able to identify some challenges facing academic developers as they assist teachers in their 21st Century role.
The implications of this for university teachers are great, particularly since the context within which they operate has changed and continues to change dramatically. The demands of an apparently ever-increasing range of stakeholders, including the students, the university and potential employers, must be addressed in conditions of high workload (Maslen, 2000) with increasingly larger groups of students of diverse ability (Marginson & Considine, 2000) who are demonstrably less engaged with their learning than the students of a generation ago (Krause, 2005). All this is occurring in an environment where modes of delivery are changing in response to societal shifts and technological advances. The teachers need help!
Enter the academic developer. The following description captures well the function of academic developers. They have:
an enabling role in improving the teaching and learning environment of the University through the provision of academic development activities and resources in the areas of teaching, learning, research education and assessment. Academic development staff work with academic staff to achieve the strategic teaching and learning directions of the University and focus specifically on the learning outcomes of students. [emphases added] (University of South Australia, 2008)This paper focuses on a case study of the 'enabling role' of members of the CATLyst Network (Ingram & Thompson, 2001) at UWA. The CATLyst Network consists of faculty representatives (known as CATLysts) who have responsibility for promoting teaching and learning within their faculty and working collaboratively with other members of the Network to promote teaching and learning within the broader UWA community. As academic developers, they combined to investigate and provide 'resources in assessment', namely, ways to improve the delivery of formative feedback to students.
There are few issues in teaching and learning more agreed upon than the crucial part played by feedback and its contribution to student learning (Biggs & Tang, 2007). Despite consensus on the value of feedback it remains problematic from the perspective of students who often complain about its relevance (Rowe, Wood & Petocz, 2008) and from the perspective of teachers who complain about the time consuming nature of providing meaningful critique of students' work (Flinders University, 2008).
The project reported here builds on previously presented work (Carr, et al., 2008). Having surveyed academic staff from across the university (see Appendix A for online survey) and interviewed students in a series of focus groups, it became apparent that while there were generic requests from students regarding the feedback they received, there were also faculty specific aspects to their requirements for feedback. For example, students in the Arts based disciplines noted the difficulty they experienced in adapting to the discipline specific discourse that was used by tutors in providing feedback. In more numerically based disciplines, such as Engineering and Mathematics, and in Law, a particularly competitive environment in terms of future employment prospects, students were keen that their feedback provided an indication of their position relative to other students in the class.
In addressing the requirements of students for relevant, formative feedback, CATLysts from six faculties undertook to work with teaching staff, trialling different processes to facilitate the provision of improved feedback in one or more units within their faculties. Their experiences follow in a series of brief case studies.
The intervention strategy, applied with the cooperation of the new unit coordinator, was to give a statement of expectations for the project, along with details of the criteria which would be used for marking to the students prior to the commencement of the assessment. Following submission of the projects, individual descriptive feedback, which addressed each of the marking criteria, was provided to students in relation to their submitted work. To evaluate the effectiveness of this approach, questions relating to the feedback given in this unit were included in the Student Perceptions of Teaching (SPOT) survey at the end of semester.
The students responded positively to the feedback provided in both the SPOT and Student Unit Reflective Feedback (SURF) surveys, The SURF scores for this unit, in particular, registered higher than the University averages. This represented a significant improvement in the level of satisfaction with the unit, which previously had scored as one of the worst within the whole Faculty. It is reasonable to assume that the improvement in the level of satisfaction was at least in part due to the improved provision of feedback on the digital media assessment projects. This model of feedback provision on digital media projects is now being extended to other units in the Faculty running similar projects.
By comparison, some Business School staff have expressed the view that students are often only interested in the mark and they wonder whether students really consider the written comments provided on their work. When surveyed, most lecturers in the School indicated that overall, they had a comprehensive and accurate understanding of the term 'student feedback'. Furthermore, those surveyed generally perceived the aim of feedback was to assist students in their learning process and to help them to improve. Most respondents were satisfied with the feedback they provided, given the resources available and the large numbers of students in some units. Ideally they preferred smaller class sizes and more tutors to assist in the marking process. They indicated that on written assignments (eg, essays) students usually receive a grade and written comments.
Taking into account the outcomes of the staff survey and student focus interviews, a small project was conducted in an undergraduate unit where students had an essay to write mid semester. The aim of the project was to provide students with specific and constructive written feedback on their essay, to indicate how the mark related to the marking criteria, and to ensure the written feedback was legible.
The project involved a second year management class of approximately 50 students, the lecturer, two tutors and the faculty CATLyst. The requirements for the essay were explained in class and there was also detailed written information in the unit outline on the general process of researching and writing essays (eg, essay structure, developing an argument, referencing etc). The lecturer and tutors were all very experienced and had taught this unit together on several occasions. The teaching staff and CATLyst met to discuss the aims of the project and to plan the approach to marking. An electronic copy of the detailed marking sheet was prepared and used by the tutors to word process their feedback. The feedback and marking sheet included all the aspects of a good essay highlighted in the unit outline. In addition, the tutors continued their usual practice of hand writing comments dispersed throughout the essay. Following the marking and return of the essays, the students were asked to complete a brief questionnaire regarding the feedback on their essay.
Forty students completed the survey. The vast majority (92%) of students agreed the feedback provided identified their strengths and provided suggestions on how to do better on their next essay. Ninety five percent of the students reported that they read the hand written comments dispersed throughout their essay and indicated that they found these comments useful. The majority of students (70%) were generally ambivalent as to whether the comments were hand written or word processed, provided the writing was legible.
There were a number of factors conducive to the students having a high quality learning experience in this unit. These included the inclusion of comprehensive information in the unit outline about how to go about writing the essay and the in-class discussions about the essay topic prior to students attempting the assignment. In addition, the lecturer and tutors had worked as a team previously and met on a weekly basis to discuss their teaching. The CATLyst brought a slightly new dimension to their approach, in this project suggesting modifications to the marking guide that were aimed at providing consistency and improved feedback to students.
Two first year core units were identified as the focus of the feedback project: Legal Process and Criminal Law. The feedback project implemented by the Unit Coordinators of these units with input from the Faculty CATLyst and the University's Learning, Language and Research Skills team (LL&RS) was integrated into a Law School project for developing and embedding a writing skills programme into the first year LLB curriculum. The assessment included two take home case analysis exercises. By way of feedback on the first case analysis, students received a detailed marking rubric and feedback sheet. Apart from the student's mark for each aspect of the case analysis, the feedback sheet provided details of areas of weakness and what further work the student could do to improve his or her writing skills including referral to specified LL&RS workshops and invitation to an individual discussion session with the student's seminar teacher. The feedback sheet indicated whether a student was required to reflect and report on follow up work undertaken to improve performance in an assessable reflective journal to be submitted at the end of the semester. In addition, general feedback was given in small groups detailing common errors. Results for the assessment (by student number) were posted on the unit webpage so that students could assess their relative performance.
Following on from the Legal Process assessment, in Criminal Law students were required to complete an assignment which included analysis of three recent cases. In this assignment, students were expected to draw on the feedback they received on their analyses for Legal Process.
Although, the primary feedback strategy used in this project was a marking rubric and feedback sheet, student concerns with and apparent dislike of marking rubrics were addressed by developing a document that not only recorded a student's performance but provided details on the areas of weakness and possible strategies the student might implement to facilitate improvement. A student could gauge and reflect on the effectiveness of the strategy or strategies which he or she selected in the follow up assessment and reflective journal. Requiring students to submit a reflective journal on their learning journey during the semester negated the risk inherent in marking rubrics and feedback sheets, indeed any form of written feedback, that students simply do not read them.
The feedback strategies developed for this project addressed a number of general concerns Law student had raised with the CATLyst network regarding the formative nature of the feedback they were receiving: 1) students could assess not only where they needed to improve, but how they might go about doing so; 2) the marking rubric was made more meaningful by requiring markers to identify and articulate areas of weakness and strategies for improvement; 3) students could assess whether they had in fact improved over the semester; 4) each student was aware of how they performed relative to the rest of the year group; and 5) individual feedback was made available to those students with the greatest need for one-on-one guidance. In facilitating this project, the CATLyst was able to assist academic staff in providing feedback that addressed the unique concerns of law students.
Laboratory sessions are an integral part of any science course at UWA. In Semester 1, 2008, each student in a Level 1 unit was given a marking guide for a laboratory report prior to the work being submitted. More than 250 students completed a survey after the report had been marked and returned. The vast majority of the students (95%) referred to the marking guide while preparing their laboratory report. Of these, half found the marking guide extremely useful, a quarter found it very useful and the remainder found it useful. Comments that were expressed included "Having not written a formal laboratory report before I found all of the sections of the marking guide useful", "With marks there you can tell how much to write/what's expected" and "All parts of the marking guide were useful as it helped me to check if I was meeting the requirements the marker would be looking out for in the report". Those that did not use the guide did so because they "used other guidelines from a textbook" or they "didn't notice it". All students said that they would use a marking guide in future, if it was supplied.
The students were also asked whether they were able to see how their work could be improved in future. The majority of students (84%) responded positively. Comments regarding how the students would improve their laboratory report the next time included "Make it clear where my hypothesis and conclusion are", "I will do more research and write more concise statements" and "More preparation into planning what I will write". Having a marking guide provides the students with clear direction for completing an assessment. When returned, they can use the formative feedback on specific sections of the work to improve future laboratory reports.
Some comments suggested that there were variations of marking and feedback provided in the different laboratory groups. In a previous survey of staff it was found that not all unit coordinators conduct moderation exercises. This highlights the importance of moderation across markers, including tutors.
Overall the use of a detailed marking guide assisted the students with knowledge of what was expected of them in each section of the report and provided feedback for writing future laboratory reports. It also provides staff with clear direction for allocating marks in each section of the report. Using a marking guide appears to be of benefit to both staff and students. Staff members in the Faculty are keen to improve feedback to students and are willing to use tools such as the marking guide in their units. This provides an excellent opportunity for the academic developer, in this case the CATLyst, to offer an effective tool for staff members to adopt.
A third year unit was chosen for the project where students complete an essay on ethics. A guide on how to complete the assignment was provided to the students, including a comprehensive marking guide. The unit coordinator and two tutors were recruited for the project. The tutors were experienced in providing written feedback to students. The faculty CATLyst provided paper based resources and one-on-one training to staff on providing verbal feedback to students using an MP3 player, recording, editing and uploading the files through the assignments tool in WebCT. The students were asked to take part in focus groups and surveys about their experience, while the tutors kept a record of the time it took them to create the feedback and which sections of the essay that they provided the most feedback on.
The majority of students (86%) read or listened to the feedback at least once. An equal number of students preferred MP3 to written feedback and a small number indicated no preference. Some of the students liked to receive feedback this way "didn't get written feedback but I think MP3 feedback is a more honest and unrestricted style, that's easier for tutors to provide feedback" while others did not "It was too personal and I was nervous about hearing it. It seemed as it I was going into the office and listening" and "Written feedback seems to be more useful. I prefer having things in front of me to see rather than listening to feedback."
The tutors commented on their experiences with creating the MP3 feedback and although it was a steep learning curve it made them examine their own feedback practices while allowing them to provide more detailed feedback and to convey emotion.
The marks awarded to essays written by final year engineering students in a unit with a strong sustainability focus were a source of contention. The essays were designed to be open-ended with no one correct 'solution'. Academic writing, analytical thinking and the ability to present a well-researched, cogent argument were expected. The students expressed disappointment with the awarded mark.
The use of a rubric was suggested by the faculty CATLyst as a way of addressing these difficulties. The lecturer was introduced to the evidence-based acceptance of rubrics as a valuable tool for teaching, learning and assessment (Stevens & Levi, 2005). The rubric would clarify both student and marker expectations in the essay. Despite this, the lecturer expressed apprehension about the use of a rubric - was it just telling the students what to do? In an effort to allay these concerns, the CATLyst invited the lecturer to a hands-on workshop on rubrics given by the School of Education. The lecturer's apprehension was addressed in the discussions - the rubric, while it may indicate 'what to do', does not guarantee performance.
The lecturer developed a rubric for the essay, but the lecturer's colleagues responded by raising warnings and expressing doubts about rubric use. A compromise was achieved - the rubric was withheld from the students, but given to the tutors. This was seen as positive step, since now at the least the tutors had a consistent set of expectations.
The opinions of one's colleagues is a powerful influence. Sometimes it is easier to defer the battle until later, and take small steps in the meantime. The soundness of assessment tools such as rubrics, oral presentations and one-on-one questioning as valid and consistent measures of achievement takes time to be appreciated. The exam or test, with its 'number', is one of many assessment tools. It should not be seen as the only valid tool. Part of the role of the academic developer is to continue to provide research based-evidence of the educational robustness of these tools of assessment.
Though sometimes only subtly different in their approach, taken collectively, the outcomes of these projects provide a wealth of information for the academic developer to consider when encouraging changes in established patterns of practice. The lessons learnt from this combined endeavour can be summarised in a few key elements. Though originating from this project, we would suggest that these elements be recognised as key considerations for any academic development undertaking.
the process of selecting or tailoring a particular approach from a toolkit of practice in Academic Development. These tailored approaches respond to context, draw on individual academic developer's experiences, ideas, strengths, values and stances and meld multiple theoretical bases (Carew et al., 2008. p.53).As Carew et al. (2008) go on to note, practitioners must 'ply their trade' in a complex context which includes the changing university environment, impacted on by the educational, societal and technological aspects referred to earlier in this paper. Over the last decade and a half, Academic Development centres have increasingly become part of the landscape at most universities in Australia and around the world. More recently, the decentralised operation of these units to address faculty and disciplinary-specific needs has gained momentum. As the academy recognises the imperative to proactively support and encourage quality teaching, so acceptance of the role of the academic developer is becoming better known. Regardless of this level of acknowledgement, the role remains challenging. It is in this context, that academic developers as change agents, find themselves 'leading from behind'.
Carew, A.L., Lefoe, G., Bell, M. & Armour, L. (2008). Elastic practice in academic developers. International Journal for Academic Development, 13(1), 51-66.
Carr, S., Bovell, P., Delves, L., Miller, S., Longnecker, N., Skead, N., et al. (2008, January). What students want: A ten minute guide to more relevant feedback. Paper presented at 2008 Teaching and Learning Forum: Preparing the Graduate of 2015, Curtin University, Perth, Western Australia. http://otl.curtin.edu.au/tlf/tlf2008/abstracts.html#carr
Flinders University (2008). Teaching strategies. Giving feedback. [viewed 23 Oct 2008] http://www.flinders.edu.au/teach/t4l/assess/feedback.php
How online instructors improved communication with students (2008). Diverse Issues in Higher Education,24 Jan. [viewed 22 Oct 2008, verified 24 Jan 2009] http://findarticles.com/p/articles/mi_m0WMX/is_25_24/ai_n24264153
Ingram, D. M. & Thompson, E. (2001, July). CATLysts for change: A university network for advancing teaching and learning. Proceedings of the Annual Higher Education Research and Development Society of Australasia (HERDSA) Conference Newcastle, New South Wales, Australia. http://www.herdsa.org.au/wp-content/uploads/conference/2001/Papers/Ingram_Thompson.pdf
Krause, K. (2005, September). Engaged, inert or otherwise occupied? Deconstructing the 21st century undergraduate student. Keynote address at the James Cook University Symposium 2005, Sharing Scholarship in Learning and Teaching: Engaging Students, James Cook University, Townsville/Cairns, Queensland, Australia. [verified 25 Jan 2008] http://www.griffith.edu.au/__data/assets/pdf_file/0005/53465/Engaged,inert2005.pdf
Marginson, S. & Considine, M. (2000). The enterprise university: Power, governance and reinvention in Australia. Cambridge: Cambridge University Press.
Maslen, G. (2000). Aussies crushed by overwork. The Times Higher Education Supplement, 14 July, p10. [verified 25 Jan 2009] http://www.timeshighereducation.co.uk/story.asp?storyCode=152591§ioncode=26
Merry, S. & Osmond, P. (2007, Autumn). Feedback via MP3 audio files. Centre for Bioscience Bulletin. [viewed 23 Oct 2008] http://ctlt.jhsph.edu/resources/views/content/files/82/bulletin22p5.pdf
Rowe, A.D., Wood, L.N. & Petocz, P. (2008). Engaging students: Student preferences for feedback. In Engaging communities. Proceedings HERDSA 2008, Rotorua, New Zealand. http://www.herdsa.org.au/wp-content/uploads/conference/2008/media/Rowe.pdf
Stevens, D.D. & Levi, A.J. (2005). Introduction to rubrics: An assessment tool to save grading time, convey effective feedback and promote student learning, Stirling , Virginia: Stylus Publishing.
University of South Australia (2008). Learning Connection. [viewed 23 Oct 2008] http://www.unisanet.unisa.edu.au/learningconnection/about/teams/professional.asp
|Authors: Lee Partridge, Centre for the Advancement of Teaching and Learning, The University of Western Australia|
Leitha Delves, Faculty of Arts, Humanities and Social Sciences, The University of Western Australia
Diana Jonas-Dwyer and Sandra Carr, Faculty of Medicine, Dentistry and Health Sciences, The University of Western Australia
Sue Miller and Coral Pepper, Faculty of Natural and Agricultural Sciences, The University of Western Australia
Natalie Skead, Law School, The University of Western Australia
Jolanta Szymakowski, Faculty of Engineering, Computing and Mathematics, The University of Western Australia
Eileen Thompson, Business School, The University of Western Australia
Email: email@example.com, firstname.lastname@example.org, email@example.com, firstname.lastname@example.org, email@example.com, firstname.lastname@example.org, natalie. email@example.com, firstname.lastname@example.org, email@example.com
Please cite as: Partridge, L., Delves, L., Jonas-Dwyer, D., Carr, S., Miller, S., Pepper, C., Skead, N., Szymakowski, J. & Thompson, E. (2009). Leading from behind: The role of academic developers in preparing graduates of the 21st Century. In Teaching and learning for global graduates. Proceedings of the 18th Annual Teaching Learning Forum, 29-30 January 2009. Perth: Curtin University of Technology. http://otl.curtin.edu.au/tlf/tlf2009/refereed/partridge.html
Copyright 2009 Lee Partridge, Leitha Delves, Diana Jonas-Dwyer, Sandra Carr, Sue Miller, Coral Pepper, Natalie Skead, Jolanta Szymakowski and Eileen Thompson. The authors assign to the TL Forum and not for profit educational institutions a non-exclusive licence to reproduce this article for personal use or for institutional teaching and learning purposes, in any format (including website mirrors), provided that the article is used and cited in accordance with the usual academic conventions.