Teaching and Learning Forum 2012 Home Page

Embedding innovative assessment practices: Developing students' critical appraisal skills

Susan Teather and Catherine Moore
Edith Cowan University
s.teather@ecu.edu.au, c.moore@ecu.edu.au

Category: Refereed
Professional practice

[ Refereed papers ]

There is a growing imperative in tertiary education (nationally and internationally) to produce graduates who can work collaboratively to assess and improve their own work and that of others (Bologna Process, 2010: Boud & Falchikov, 2006; Oliver, 2011). At ECU our new undergraduate curriculum framework titled Curriculum 2012: Enabling the learning journey promotes assessment for learning. This develops both the capacity to learn and the ability to direct learning. In order to successfully direct their own learning beyond university, students need to be able to identify the standard of performance to which they should aspire as a result of that learning, accurately locate where they are in relation to the standard, and then develop pathways to bridge the gap. In other words they need to engage in formative assessment.

This paper reveals how one lecturer introduced innovative practices in teaching and assessment in order to enhance her students' ability to direct their own learning, to increase the value students place on their feedback, to ensure their active engagement with feedback, and ultimately to develop students' ability to calibrate their own judgement about their learning. The innovation was designed to increase student understanding of what constitutes academic rigour within the discipline as well as the standards required for success in real world endeavours.
Keywords: critical appraisal, assessment, feedback, formative, academic rigour, lifelong learning


Context

Designing Family Support Programmes (CHN3206/4115) is a final year unit within Edith Cowan University's School of Psychology and Social Science. Whilst the majority of students enrolled are within this School, others do choose this unit as an elective and therefore there is a diversity of students involved. The unit is offered in both online and face-to-face modes.

This unit seeks to develop the graduate attributes of effectively communicating with others, working in teams, generating ideas, considering cross-cultural and international perspectives; and critical reflection and appraisal skills. The development of these attributes is essential to these final year students who will in the near future graduate and embark upon their new career.

The planned learning activities in this unit are based around authentic assessment tasks in a situated learning context, thus encouraging active and deep learning (Wiggins, 1990; Lave & Wenger, 1991; Bonwell & Eison, 1991). The assessment tasks require that each student identifies a population for which they will design a programme, giving students agency in selecting learning activities relevant to them and their area of interest, and enhancing independent learning (Candy, 1991). After this initial identification of a target population, the students embark upon designing a family support programme for this group. This task is complex, involving many aspects, but each is introduced through the content of weekly lectures, tutorials and comprehensive modules to ensure that students develop a full understanding of all the requirements of designing a family support programme.

Students commence this unit by revisiting social science theories and reviewing literature focused upon the needs of families and the variables that will support and address these needs. They develop their knowledge and understanding of different populations and the importance of creating programmes designed to support the individual needs of families within various populations. Students learn to combine practical knowledge with the theories underpinning family support that are essential to embrace when working effectively and respectfully with families.

The influence of Bronfenbrenner's ecological theory

Social Science students are introduced to Bronfenbrenner's ecological theory early in their studies. The development of understanding of this theory emerges throughout the degree programme and by the time students are in their final year they are expected to be able to apply the theory as they design an appropriate and effective family support programme.

Bronfenbrenner's ecological theory (Bronfenbrenner, 1979) suggests that to fully understand a developing child, it is essential to consider all the factors that impact upon that child, such as their family, friends, neighbours, school, community environments, services whilst also extending to include societal attitudes and government policies. By extension then those factors, such as cultural background, socioeconomic status, religious beliefs, education and work status which affect the family, also need to be considered as these influence the family's approach to raising children. Understanding the links between these factors, and the strength of relationships or connections between them is also critical to the design of effective family support programmes. There is no such thing as a decontexualised child and therefore, the child must be considered in the context of all the environments they are involved in (Bronfenbrenner, 1979). The people within these environments that directly link to the child are influenced by other environments and other people, whilst also influencing these environments and people. Therefore, it is essential that students demonstrate their understanding of all these factors as they develop a family support programme.

Bronfenbrenner's ecological theory also influenced the introduction of the innovative assessment practice in this unit. Students bring to their learning environment their individual characteristics such as readiness to learn, culture and self-efficacy, which impact upon their development as a person (Yih Chyn Kek, Darmawan & Chen, 2007). In addition, students' individual characteristics impact upon the connections they have between various systems they are involved in. The application of the theory continues as students explore the impact of culture, societal attitudes and government policy upon the people they are designing the family support programme for. Awareness of this enhances students' understanding of the application of the ecological theory and further develops the skills that will assist them as graduates. Overall, students are immersed in a learning environment that is authentic and clearly links theory to practice.

Consistent with Lave and Wenger's (1991) description of situated learning as a model of learning within a community of practice that is based on authentic problem-solving requiring interaction with others and with environments, the assessment for this unit asks students to design a family support programme for a real population of their choice. This opportunity to choose means, for example, that:

By applying the ecological theory to the students' learning, encouraging students to introduce their context to the assessment, and supporting an international and cross-cultural approach, students are engaged in deep learning in an authentic context.

Drivers for change in assessment practice

Before the introduction of the innovation, the assessment scheme contained two linked assignments. The overall outcome for both assessments was the design of an entire family support programme, but to scaffold the learning of the students, the designing of the programme was divided into two. The first assignment involved students undertaking research into the group of their choice and determining how best to support these families and their needs. The second assignment enabled students to complete the proposed programme as they provided detailed descriptions of the strategies, with specified outcomes and methods of evaluating the programme. In addition to the new information required for assignment two, students needed to respond to comprehensive feedback provided by the lecturer. Students were expected to improve previous information and to incorporate lecturer suggestions and comments into the second assignment. Both were authentic tasks requiring a combination of academic rigour and real world application.

With high numbers of students the workload for the lecturer of providing extensive feedback for each student was extremely high and not sustainable. Responsibility for directing the learning required to close the gap between current and desired performance also fell to the lecturer rather than the students. Although marking guides were supplied with tasks, the interpretation of these guides was done by the lecturer while students passively awaited feedback. Therefore, students never really developed an understanding of what the various levels of performance actually looked like.

The lecturer also wished to assist her final year students to realise their potential as future leaders. This motivated the lecturer to create a learning experience that would develop their ability to lead others in their practice. To accomplish this, the skills of critical reflection and appraisal are essential and this unit provided an ideal opportunity for students to attend to these skills.

With a desire to address the challenges of high teacher workload in giving high quality feedback, lack of student engagement with feedback, and a desire to promote lifelong learning, collaborative and leadership skills in graduates, it became clear that changes needed to be made to the assessment strategy. To begin this change process the lecturer contacted an academic advisor in the Centre for Learning and Development at Edith Cowan University and a productive collaborative relationship began. An initial analysis of the assessments for this unit was followed up with further meetings. Further reflection by both partners, and a willingness to achieve a desired outcome, led to more substantive changes being embarked upon which centered on the implementation of peer review processes.

The innovation

While the core assessment tasks were essentially retained (design of a family support programme) the assessment practice around them underwent significant changes. A key aspect of these changes is how assessment is conceptualised as not only measuring learning, but also impacting on learning (Juwah et al., 2004). The changes intended to make assessment of student work (both their own and that of their peers) part of students' learning. Rust, O'Donovan and Price (2006) showed that engagement in the process of formative assessment improves academic outcomes. In addition, Van Den Berg, Admiraal and Pilot (2006) observed that active engagement in peer assessment produced better structured interaction between students, as well as more organised written work. In particular, giving feedback has been shown to have considerable benefits for student learning (Li, Liu & Steckelberg, 2009). Nicol and MacFarlane-Dick (2006) noted that, by commenting on the work of peers, students develop an understanding of standards which they then transfer to their own work.

The retention of the assessments ensured that students continued to draw upon Bronfenbrenner's ecological theory as they applied the theory through the designing of a family support programme and as they engaged in contextualised learning. As proposed by this theory, the interactions and relationships that an individual has with their environments are influential to outcomes (Bronfenbrenner, 1979). Therefore, it was important to ensure that students continued to apply this theory in their assignments whilst being engaging in learning that embraced interactions with peers. This approach reinforced the underpinnings of this theory that were essential to the assessment whilst also being essential to the students learning environment and therefore their outcomes.

In order to ensure deep learning leading to higher levels of cognitive performance, teaching strategies that promote active learning were embedded even more extensively throughout the unit. These included common elements cited by Bonwell and Eison (1991) such as student involvement in discussing and writing rather than just listening, emphasis on skill development rather than content transmission, and frequent use of higher order thinking skills such as synthesis and evaluation rather than those of explanation and application. To do this students draw upon a range of skills that are essential employability skills (DEST, 2002). Students must work with their peers both in class and on-line. This creates authenticity as within any workplace communication happens within groups, some are face-to-face and others involved virtual communication. Students discuss their ideas and concepts with peers, questioning and challenging each other, therefore enhancing their critical reflection and appraisal skills. All of which are essential attributes for students to possess and to utilise in their career and future workplaces.

Challenges and complexities

The complexity of the innovation and implementing changes produced a number of challenges for the lecturer. Within any University, students often have reservations about being involved in group work, frequently grounded in competitiveness and a perception of lack of equity. Perceived demands that students compete with each other for marks can also create resistance to sharing their work with their peers. Showing their work to their peers creates fear that their work is not good enough and that the student will be criticised. Yet for graduates of this course, working collaboratively with peers in the workplace is essential. Therefore, the lecturer needed to aware of this and then had to ensure that she introduced strategies to address the hesitation and resistance of the students. Clear and open communication of the process and the reasons for, and benefits of the process for students, and ongoing support and encouragement by the lecturer were utilised to ensure students understood the importance of being involved in the peer reviewing process.

A further challenge was ensuring that students were actively participating in their learning. In some incidences students become dependent upon their lecturer for feedback, not seeking the consult with their peers and to actively engage in constructive and reflective processes. Students in this unit were being asked to take on a leadership role and to provide constructive and respectful comments to their peer. They were being asked to actively participate in this process that in the past had been reserved by the lecturer. This active participation is another important factor for students to embrace, as within workplaces graduates will be expected to take on leadership's roles and to actively engage and work with their peers. Students could have considered that providing feedback to their peers was not part of their role as a students, but clear communication of the process and clear explanation of the concept of peer reviewing ensured that students were well informed and therefore actively engaged (Cathy do you have a reference here about doing this?). This involvement from the beginning of semester ensured that this potential challenge was addressed and did not emerge as an issue later on.

Engaging students in reviewing their peers work required considerable contemplation, particularly around the actual feedback. The lecturer provided instructions detailing the process and how to provide the feedback. On-campus students were given both verbal and written instructions of the process and time was spent, from the beginning of semester, explaining the process and the expectations of the students. Off-campus students were encouraged through Discussion Board and email contact to gain a clear understanding of the processes. All students were provided with guidelines of what was expected of them in the peer review process. These guidelines indicated that students must "provide considered, professional and respectful comments to your peer". Enforcing this was considered a potential challenge, yet again this did not emerge. The lecturer considers that the level of communication and the clear student expectations were the key contributing factor.

Anticipation of potential issues was important to the overall process. Through this, strategies could be set in place at the very beginning of semester. This was effective as issues did not emerge.

Factors critical to success

The changes to assessment practice required considerable upskilling of students so that they would be able to employ critical appraisal skills in reviewing each others' work. Care was taken to ensure students knew how to provide effective feedback and understood the marking criteria and levels of performance required.

The innovation incorporated factors such as willingness, openness, preparedness, timing and open and clear communication. These were considered essential factors to the success of the proposed innovation and therefore the possession of these by the lecturer was critical. The first factor was the lecturer's degree of willingness to facilitate the learning process required for peer feedback. The lectures openness to change was a vital precondition to embedding the innovation successfully and this was demonstrated by her willingness to work collaboratively and consider 'possibilities' suggested by her colleague, the academic advisor.

In addition to willingness, preparedness and the allocation of sufficient time to reflect upon the innovation, was another essential factor to successful implementation. The lecturer took time to contemplate the implementation, prepare herself and then prepare the students. A timeline was established with sufficient time allocated to reflect and make necessary adjustments. Students had sufficient time to prepare themselves for their involvement in the innovation. Clear communication of the processes and the requirements were provided at the beginning of semester, and communication continued until after the peer reviewing session in week seven.

Additional to these factors, the ability of the lecturer to implement the peer reviewing process was critical to success. The collaborative working relationship again contributed to achieving this as the lecturer engaged in conversations with her academic advisor. She was introduced to literature to help her increase her understanding of peer reviewing and engaged in constructive conversations. Success was also impacted upon the lecturer's ability to communicate the process and to engage students in this aspect of their learning. Experience of working with students and having developed relationships with students contributed to the lecturers' ability to facilitate this process successfully. To enhance the innovation and to ensure rigour, Lave and Wenger's (1991) idea of situated learning was encapsulated into the two CHN3206/4115 assessments. There was a natural fit between situated learning and these assessments as the assignments emphasised authentic problem-solving with others and within environments. The alignment between situated learning approaches and practices within this Social Science unit is illustrated in Table 1 (below).

Table 1: Alignment of situated learning and CHN3206/4115

Situated learningCHN3206/4115
Authentic contexts reflect the way knowledge will be used in real lifeStudents design a family support programme to meet a real need in a real situation
Authentic activities reflect future professional activitiesStudents obtain feedback from others in designing the programme
Access to expertsResearch, lecturer input, community input
Modelling of authentic processesScaffolded assignments mimic development stages of real programmes
Multiple roles and perspectives in the collaborative construction of knowledgeStudents provide feedback to peers and gain the perspective of peers on their work
Promote articulation to enable tacit knowledge to be made explicitStudents discuss required standards of performance and what they might look like
Reflection to enable abstractions to be formedWritten feedback and reflection on feedback in order to respond

Time and timing were critical to success. Therefore, a timeline (see Table 2 below) was set for the semester to ensure that students had sufficient time to learn about and to reflect on the peer reviewing process. The lecturer explored and read about peer reviewing to increase her knowledge of the process and the benefits of engaging in this within the learning environment. Furthermore, the lecturer continued to consult with the academic advisor to ensure the quality of the peer reviewing process being introduced.

Table 2: Timeline

Week 1Details of peer reviewing provided to students.
Week 2Students engaged to look at the guidelines to be used in the peer reviewing process.
Week 3Lecturer considers feedback from previous week and adjustments are made to peer reviewing document.
Week 4Students provided with sample assignments and Peer Reviewing Key. On-campus students undertake a review of these assignments in class. Off-campus students engage in the same process through using Blackboard.
Week 7Students engage in the peer reviewing process in class and online
Week 9
(approximately)
Students are provided with a copy of the review of their work when marked assignments returned.
Week 13Students submit their final assessment in this week. Students are required to consider the feedback provided by their peer. Whilst they do not have to use this feedback the student must consider it and reflect on the feedback, explaining why used or why they didn't use the feedback.

The experience

The learning process for the students began with a collaborative approach between the lecturer and the academic advisor to adjust the marking key for the first assessment and to ensure that the criteria for the assessment was clear and understandable. With these changes made, an additional document was created that would be used in the peer reviewing process. Each criteria was broken down and defined to ensure that students clearly understood the expectations for each criteria they were being assessed upon.

To further assist students' understanding and the expectations, guidelines of the peer reviewing process were developed. Time was allowed to develop and then reflect upon these guidelines but to ensure their accuracy and the effectiveness the lecturer recruited five students (both on and off-campus as instructions differed slightly depending upon students mode of enrolment) to review them. Feedback from the students identified that some points requiring further clarification. For example, the time frame for the reviewing was not considered clear and the guidelines for the off-campus students required further clarification around the actual process of reviewing. These comments resulted in a revision of the guidelines. These were again shown to two on-campus students to clarify that the changes made were sufficient.

Following this initial stage, in week four the students were introduced to peer reviewing in class and on-line. On-campus students were provided with sample assignments from previous years and the peer reviewing key. To facilitate this process the lecturer explained the process and encouraged the students to first consider the assignments on their own and then to discuss in their groups. Similarly, off-campus students were provided with sample assignments and peer review key and encouraged to go through the same process but to use Discussion Board as the means of discussion with peers. Group discussions enabled students to consider differences and similarities between comments and in relation to their assignment, consider and gaps and what they can do to rectify these prior to the submission of the assessment.

A further aim of the week four reviewing was to gather feedback on the peer review key as it was important that the document was useful and provided a high degree of clarity. Both off and on-campus students were asked to provide comments and they indicated that the document did not require any further changes.

With this task completed the next task was to undertake the actual peer review session. This session will be conducted in week seven and differed slightly to suit both on and off-campus students. On-campus students brought a copy of their assignment to class for reviewing by their peer. Students were allocated another student's assignment to review. They were given a peer reviewing key and were allocated 40 minutes in which to review the document. Once completed, students paired with the students whose assignment they had reviewed and provided verbal feedback in a respectful, reflective and professional manner. To ensure that all students had the opportunity to give and to receive feedback this was undertaken twice.

Off-campus students undertook this task similar to on-campus students. The lecturer allocated students with a peer's assignment to review. The students conducted this review and wrote comments to their peer in the peer reviewing key document. The reviewed assignment was sent back to the lecturer who then forwarded the document to the appropriate student. As students could not engage in face-to-face conversations they provided considerable comments within each criteria and a summary of their comments at the end of the peer reviewing key document.

To finalise this aspect of the assessment, students were required to reflect on the feedback receive from their peer and to provide this reflection in their second assignment. This second assignment builds on the first one, therefore having this opportunity to receive additional feedback and respond to it, enabled students to improve their performance. Students did not have to agree with their peers comments or to even implement their suggestions, but it was essential that they show that they had considered the comments and reflected upon these. This again provides an authentic learning environment for the students as within workplaces graduates are required to consider the comments of their work colleagues when working collaboratively. At times comments are implemented and on other occasions comments are discarded. The key to this learning was around 'hearing' considered comments and determining the appropriateness of this information.

Outcome and benefits

At the completion of this unit in 2011 there has been time for the lecturer to reflect on the process and the outcomes for the students. The lecturer gathered feedback from the students about the peer reviewing process. Students were generally positive about the change to assessment practice. A typical student comment is:
The critical reflection [peer review] I completed on a peers assignment was a great exercise for it helped me expand my abilities to constructively critic, see just how very different assignments can be completed, understand just how difficult marking can be and develop self reflection skills.
To this comment, this student informed the lecturer that she and the person she reviewed had continued in the peer reviewing process as they had looked at each other's second assignment for this unit and then had expanded this as they had looked at an assignment for another unit. The process of engaging students as active participants in their learning and working collaboratively with their peers was clearly evident in these students academic life.

Other comments from students similarly reflected the success of this innovation. One student commented that "Feedback on assignments was great, the best from all classes. It really helped for the second assignment." Another student stated that "The feedback provided helped me to improve on my assignment." This process of providing feedback had also provided the opportunity for the reviewing student to develop a clearer understanding of the requirements of this assessment. Students had become more engaged, developed clearer understanding of the requirements and had developed a range of skills and attributes by being involved in this innovation.

References

Bologna Process. (2010). Employability. Downloaded 1 September, 2011 from http://www.ond.vlaanderen.be/hogeronderwijs/bologna/actionlines/employability.htm

Bonwell, C.C. & Eison, J.A. (1991). Active learning; Creating excitement in the classroom. ERIC Digest.

Boud, D.J. & Falchikov, N. (2006). Aligning assessment with long-term learning. Assessment and Evaluation in Higher Education, 31(4), 399-413.

Bronfenbrenner, U. (1979). The ecology of human development: Experiments by nature and design. Cambridge, Mass: Harvard University Press.

Candy, P.C. (1991). Self-direction for lifelong learning. Jossey-Bass Higher and Adult Education Series San Francisco, California.

DEST (2002). Employability skills for the future. A report by the Australian Chamber of Commerce and Industry and the Business Council of Australia for the Department of Education, Science and Training, Canberra.

Juwah, C., Macfarlane-Dick, D., Matthew, B., Nicol, D., Ross, D., & Smith, B. (2004). Enhancing student learning through effective formative feedback. Higher Education Academy: London, UK.

Lave, J., & Wenger, E. (1991). Situated learning: Legitimate periperal participation. Cambridge, UK: Cambridge University Press.

Li, L., Liu, X. & Steckelberg, A.L. (2009). Assessor or assessee: How student learning improves by giving and receiving peer feedback. British Journal of Educational Technology, 41(3), 525-536.

Nicol, D.J., & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199-218.

Oliver, B. (2011). Assuring graduate outcomes. Support for the original work was provided by the Australian Learning and Teaching Council Ltd, an initiative of the Australian Government.

Rust, C., O'Donovan, B. & Price, M. (2006). A social constructivist assessment process model: How the research literature shows us this could be best practice. Assessment and Evaluation in Higher Education, 30(3), 233-241.

Van Den Berg, B.A.M., Admiraal, W.F. & Pilot, A. (2006). Design principles and outcomes of peer assessment in higher education. Studies in Higher Education, 31(3), 341-356.

Wiggins, G. (1990). The case for authentic assessment. Practical Assessment, Research & Evaluation, 2(2). http://pareonline.net/getvn.asp?v=2&n=2

Yih Chyn Kek, M., Darmawan, I.G. & Chen, Y.S. (2007). Family, learning environments, learning approaches, and student outcomes in a Malaysian private university. International Education Journal, 8(2), 318-336. http://ehlt.flinders.edu.au/education/iej/articles/v8n2/Kek/paper.pdf

Yorke, M. (1998). Assessing capability. In J. Stephenson & M. Yorke (Eds), Capability and Quality in Higher Education. London: Kogan.

Please cite as: Teather, S. & Moore, C. (2012). Embedding innovative assessment practices: Developing students' critical appraisal skills. In Creating an inclusive learning environment: Engagement, equity, and retention. Proceedings of the 21st Annual Teaching Learning Forum, 2-3 February 2012. Perth: Murdoch University. http://otl.curtin.edu.au/tlf/tlf2012/refereed/teather.html

Copyright 2011 Susan Teather and Catherine Moore. The authors assign to the TL Forum and not for profit educational institutions a non-exclusive licence to reproduce this article for personal use or for institutional teaching and learning purposes, in any format, provided that the article is used and cited in accordance with the usual academic conventions.


[PDF version] [Refereed papers] [Contents - All Presentations] [Home Page]
This URL: http://otl.curtin.edu.au/tlf/tlf2012/refereed/teather.html
Created 22 Jan 2012. Last revision: 22 Jan 2012.