|Teaching and Learning Forum 2011 [ Refereed papers ]|
Julia Wren, Alistair Campbell, John Heyworth and Rachel Bartlett
Edith Cowan University
firstname.lastname@example.org, email@example.com, firstname.lastname@example.org, email@example.com
Assessment in the arts can be challenging. The problems associated with assessing student performances are that judgements are complex and even more so when they involve multiple markers (Campbell, 2005). It can be difficult to ensure assessment is fair, valid and reliable and that students are provided with clear feedback that is meaningful to them. This paper describes the development and trialling of an innovative, technology supported tool, designed to improve the confidence, efficiency and effectiveness of student performance-based assessment in arts education.
This project is a work in progress and early trials of the Internet based digital computer technology tool suggest that it reduces tutor anxiety, increases markers' confidence, improves the management of assessment and record keeping, facilitates easy and quick moderation, and provides explicit feedback to students to enhance ongoing learning. The perception by stakeholders is that this assessment is far more accessible and easy to use than previous methods. The project was implemented in a Bachelor of Education course, with 170 third year pre-service teacher students. Group performances were assessed by three tutors simultaneously during the live performance and for one week afterwards by viewing online video recordings of each performance. The tutors were able to discuss the assessment online via a confidential 'tutor box' attached to the marking key and only visible to tutors. The assessment was presented to the students as a one page electronic marking key with the video of their performance embedded into this page. This was saved as a PDF document and emailed to students. The students' responses have been overwhelmingly positive. They reported that ease of access to this electronic assessment meant that they engaged with their feedback multiple times. The tutors reported that the process was far more streamlined and fair.
Use of online e-marking has been extensively researched by Campbell (2005) who found that the tool enables a paperless assessment that increased marker productivity, improved moderation practices as well as streamlined evaluating and analysis of overall results. The tool used in this study was modelled on earlier e-marking tools developed by Campbell.
Marking and moderation practices among multiple tutors which result in fair assessment for students are difficult to implement in the arts (Campbell, 2005) largely because of the ephemeral nature of live performances. Accuracy in assessment of performances where multiple markers are involved with large numbers of students and relatively short time frames is an issue.
When using the Internet based digital computer technology tool to assessing live performances, tutors in this study found that what is usually a difficult and time consuming assessment process was streamlined, and improved the 'on the day' assessment processes and moderation practices among tutors, which resulted in assessment that was more valid. It took away the need to make instant decisions and enabled the tutors to mark away from classroom distractions (Brunvand, 2010). In addition, the use of technology enabled the quality of feedback given to students to be far more comprehensive and educative. McConnell, Lundeberg, Koeler, Urban-Lurain & Zhang (2008) show that when teachers review videos as a part of their assessment, they are more likely to base their judgements on evidence rather than memory. When an immediate response is not necessary, the teacher then has the luxury of being able to engage with a deeper reflection and analysis (Romano & Schwartz, 2005).
In addition, the use of electronic marking tools with video clips of students' performances attached to them will provide explicit, educative and comprehensive feedback to the students on the quality of their learning. Providing quality feedback to students about their learning is a significant factor in enhancing learning outcomes (Dinham, 2010). Feedback given to students in this project is based on authentic assessment, which relates to their future work as teachers.
The assessment contributes to student learning through the quality and educative value of assessment (assessment as and for learning) as it makes explicit the way in which marks are awarded and comments made. The assessors' marks and comments become educative annotations as the students view videos of their own performance and read the assessor's marks and comments. In addition, video clips of performances will be selected for future use as exemplars to demonstrate particular concepts, skills and understandings.
The video recordings offer students a view of their performances from the audience perspective. Comparing their own performance from this perspective against the criteria and marks/comments given provides quality feedback to individuals within groups.
Assessing the drama performances using the electronic rubric simplified the marking process. I was able to view the live performances in their entirety... It made the moderation process instantaneous, as the team were able to view each grade and comments as it was submitted on the rubric (Drama Tutor).In addition, moderation of marks after the live performances need not take place in face-to- face meetings, but rather over a period of time where tutors are able to access the online marking tool and video clips of each performance at a time suitable to them. The online marking tool has provision for assessors to chat via a confidential box on the marking tool, only available to the markers. The comments in this box function to enhance moderation and record discussion between tutors for future reference.
While I was still able to respond to the immediate impact of the performance, I was also able to go back and view the videos, reflect on my assessments and, if necessary, adjust some of my judgements. At the performance I could view my assessment criteria, the students' names and have immediate interaction with the other tutors who were assessing different elements of the arts. I felt that I was able to make informed and valid decisions.rather than the 'quick draw' reactions of previous assessment experiences. (Music Tutor) The online marking tool eliminates the need for paper-based documentation of assessments and enables far more efficient management of records for both the users and recipients. It has potential to be used across a number of educational settings where students undertake any kind of performance (including presentations and practical assignments) where one or more markers assess their learning.
In truth I was quite nervous before hand; the performance and the feedback were over so quickly that I had little time to reflect upon them. Allowing me the opportunity to view the performance and hear the peer feedback, coupled with yours, J's and R's professional feedback through the rubric, has presented me with a rare and much valued opportunity to review what I had learnt and what I still needed to learn (Student).Miels (1999) emphasises the positive effects and the value that is added to learning when students are given multiple opportunities to view their videos. In addition, viewing their own videos with the tutor feedback enables the students to see actual events and why they were awarded the marks they were as opposed to what they thought might have happened (Romano & Schwartz, 2005).
Previous studies of assessment in tertiary education have indicated that students are largely concerned with the fairness of assessment marks (Wren, Sparrow, Northcote, & Sharp, 2008). Students in this study report that they are satisfied with the electronic marking as a means to further moderation among markers.
Students report that they like their assessment sent to them electronically. They feel that "it is an incredible use of technology. This method of feedback is truly unique. The opportunity to see the group performance and the rubric was fantastic" (Student). Receiving their marks and video clip via email complements their preferred mode of learning. They are able to access this from anywhere and as many times as they need.
The ease of accessing an electronic document and the preference students have for electronic documents, gave students incentive to review the feedback. Many students reported that they accessed and reviewed the feedback rubric multiple times, each time focusing on a different aspect that was assessed.
This method of marking allowed for multiple moderation sessions, which satisfied the markers and students that the results given were fair and valid. The markers felt that the strength of this method of assessment was that they could review their initial marks and comments at a later time that was convenient to them and use the tutor chat box for discussion between tutors.
For a tutor to assess these [performances] during a one off short performance seems to be a task that is in danger of being a 'hit and miss' affair. In this trial, all that changed. (Tutor)
The opportunities for learning and teaching using Internet based assessment tools are many. The availability of electronic feedback combined seamlessly with the actual performance makes access to it convenient for students both on or off campus, and at those times that are most productive educationally for each individual. In addition, the electronic assessment provides opportunities for students to review their feedback and performance multiple times.
ICT is changing the traditional learning and assessment processes (Campbell, 2005). Wheeler and Wheeler (2009) believe that there is a growing challenge for many academics to discover how universities can harness the power of online tools within the formalised structures of the institution. This project demonstrates an example of this. The possibilities are that this type of online assessment tool can provide educative feedback to students, and could be adapted to empower learning through self and peer-assessment. These new forms of digital assessment tools provide the opportunities to engage in innovative assessment strategies previously beyond the financial resources of most institutions. Through self and peer-assessment we become lifelong learners and digital forms of assessment, like the one described, is making it possible within the resources available to educational institutions.
The ease with which this Internet based digital computer technology tool can be used in classrooms means that it is applicable in a wide range of educational settings and can easily be transferred to many educational contexts.
Campbell, A. (2005). Application of ICT and rubrics to the assessment process where professional judgement is involved: the features of an e-marking tool. Assessment & Evaluation in Higher Education, 30(5), 529-537.
Brunvand, S. (2010). Best practices for producing video content for teacher education. Contemporary Issues in Technology and Teacher Education, 10(2), 247-256. http://www.citejournal.org/articles/v10i2currentpractice1.pdf
Denton, P. (2001). Generating coursework feedback for large groups of students using MS Excel and MS Word. University Chemistry Education, 5(1), 1-8. http://www.rsc.org/pdf/uchemed/papers/2001/p1_denton.pdf
Dinham, J, (2010). Delivering authentic arts education. Cengage Learning, South Melbourne, Vic.
Ladson-Billings, G. (1998). Teaching in dangerous times: Culturally relevant approaches to teacher assessment. The Journal of Negro Education, Summer 1-2.
McConnell, T. J., Lundeberg, M. A., Koehler, M. J., Urban-Lurain, M. & Shang, T. (2008). Video Based Teacher Reflections: What is the real effect on reflections of in-service teachers? Proceedings of the 2008 International Conference of the Association of Science Teacher Educators, Saint Louis, MO (1-14).
Miels, J. (1999). Videotaped teaching segments and the pre service teacher: Developing reflective practice. The Teacher Educator, 34(3), 181-188.
O'Toole, J. (2009). Research in drama education. The Journal of Applied Theatre and Performance, 14(4), 479-501.
Rich, P. & Hannafin, M. (2009). Scaffolded video self-analysis: Discrepancies between pre service teachers' perceived and actual instructional decisions. Journal of Computing in Higher Education, 21(2), 128-145.
Romano, M. & Schwartz, J. (2005). Exploring technology as a tool for eliciting and encouraging beginning teacher reflection. Contemporary Issues in Technology and Teacher Education, 5(2), 149-168. http://www.editlib.org/d/6026/article_6026.pdf
Song, K. & Catapano, S. (2008). Reflective professional development for urban teachers through videotaping and guided assessment. Professional Development in Education, 34(1), 75-95.
Wheeler A. & Wheeler D. (2009).Using wikis to promote quality learning outcomes in teacher training. Learning, Media and Technology, 34(1), 1-10.
Wren, J., Sparrow, H., Northcote, M., & Sharp, S. (2008). Higher students' perceptions of effective assessment. The International Journal of Learning, 15(1), 11-24.
|Please cite as: Wren, J., Campbell, A., Heyworth, J. & Bartlett, R. (2011). Improving marking of live performances involving multiple markers assessing different aspects. In Developing student skills for the next decade. Proceedings of the 20th Annual Teaching Learning Forum, 1-2 February 2011. Perth: Edith Cowan University. http://otl.curtin.edu.au/tlf/tlf2011/refereed/wren.html|
Copyright 2011 Julia Wren, Alistair Campbell, John Heyworth and Rachel Bartlett. The authors assign to the TL Forum and not for profit educational institutions a non-exclusive licence to reproduce this article for personal use or for institutional teaching and learning purposes, in any format, provided that the article is used and cited in accordance with the usual academic conventions.