Teaching and Learning Forum 2004 Home Page
Teaching and Learning Forum 2004 [ Proceedings Contents ]
Assessment for learning: Some insights from collaboratively constructing a rubric with post graduate education students

Lesley Newhouse-Maiden and Terry de Jong
Edith Cowan University

Learning to work collaboratively is an important component of successful middle schooling practice (Jackson & Davis, 2000). In their preparation to become middle years teachers, students completing the Postgraduate Diploma in Education (Middle Years) at Edith Cowan University (ECU) are required to engage in a range of collaborative group processes. An example of this is a small group-based project in the From Alienation to Engagement unit which necessitated students investigating and presenting their findings on the services offered to adolescents by youth-focussed agencies. As part of this project, students had to reflect upon their individual contribution to the group process. To help facilitate this, students were required to construct an assessment rubric and use this instrument to inform their reflections. This paper describes the rationale for and process used in constructing the rubric. It discusses four key themes of student learning identified from the students' reflection papers.


Introduction

This paper emanates from our desire to "practice what we preach" in educating post-graduate students to be and become effective middle years teachers (Newhouse-Maiden, 2002). We were concerned about aligning our unit of study From Alienation to Engagement which is part of ECU's Postgraduate Diploma of Education (Middle Years) to two of the Graduate Attributes at ECU (2002), namely. "problem-solving/decision making" and "teamwork".

The task we set our students was to work in small teams to investigate a community agency that supported the needs of, and challenges faced by, young adolescents. Our students were required to "actively accomplish complex and significant tasks", while bringing to bear their "prior knowledge, recent learning, and relevant skills" (Stowell & McDaniel, 1997, p.142), and to present their findings to their peers. An additional challenge for members of the small groups was to evaluate the effectiveness of their individual contribution to the group.

Our challenge was to develop our students into effective middle school team members and to incorporate teamwork in their middle years pedagogy. In accord with good practice we facilitated a process which created an assessment rubric for each performance task (Jackson & Davis, 2000, p. 60), where a rubric:

... outlines a set of criteria, usually on a four-or six-point scale with performance descriptions that define a range of performance on an authentic assessment task (Stowell & McDaniel, 1997, p. 142).
Our reasoning was that students needed to develop their skills in rubric construction as future middle years teachers. We were guided by McTighe's (1996/1997) contention that "if we expect students to improve their performances on these new, more authentic measures, we need to engage in "performance-based" instruction on a regular basis" (p.7).

We also speculated that if we "crafted a rubric" with the help of students (Jackson & Davis, 2000), they might take more ownership for their performance and gain more "enduring understandings" of group processes as teachers if:

... at every point on the rubric's scale its creators must decide what specific evidence they should see of the understandings and skills the assessment is to incorporate" (p.60)
This rubric was subsequently used as a means of students giving their peers feedback about their contribution to the group. Each student had to collate the feedback (about five per student) and then reflect upon his/her contribution in an individual paper that was submitted as part of the group project assignment.

In sum, there were two key intended outcomes associated with the inclusion of the assessment rubric in the From Alienation to Engagement unit:

  1. Using a collaborative approach, students should be able to construct an authentic assessment rubric applicable to the middle years of schooling which enhances learning, and reflects the key principles of validity and fairness.

  2. Using the assessment rubric as a feedback source and self-reflection tool, students should have a better understanding of the complex dynamics of working in groups, and more insight into their current and potential contribution to group work.
Based on our 2003 course review and formative evaluations of our students completing their Assistant Teacher Practicum, there is clear evidence to suggest that these outcomes are being met. In endeavouring to understand the nature of this evidence better, we were curious to find out what our students appeared to have learnt about the potential of a collaboratively constructed rubric as an assessment "tool". Using Miles and Huberman's (1994) data analysis processes of data reduction, data display, and conclusion drawing we examined the students' reflection papers to respond to this enquiry. The purpose of our paper is to discuss our findings and reflect on what we have learnt from the process too. Before doing so, we briefly describe how the assessment rubric was constructed.

The process of constructing the assessment rubric

In facilitating the construction of the assessment rubric we were guided by four key principles (McTighe, 1996/1997), as follows.

Establish clear performance targets with understanding

It was our firm conviction that setting clear performance targets will encourage students to engage in a more meaningful learning experience. As (McTighe, 1996/1997, p.8) points out,
When students have opportunities to examine their work in the light of known criteria and performance standards they begin to shift their orientation from "What did I get? To "Now I know what I need to do to improve.
Summative outcomes are clearly important. The final grade carries weight. However, engaging students in an "action cycle" is a powerful way of encouraging a more equitable balance between "product" and "process". The rationale for collaboratively constructing the rubric was to ensure that students had a common understanding of the criteria and performance tasks so that the task of establishing group processes was made easier, and the teams' chances of meeting its goals (i.e. performance tasks) were maximised.

Teach strategies explicitly

During their orientation week students were introduced to the importance of building effective groups, and the implication of this for becoming a successful middle years teacher. We built on this experience and skill development by adopting Johnson and Johnson's (2003) The Distributed Actions Theory of Leadership as a theoretical basis on which to conceptualise 'effective groups'. According to Johnson and Johnson members of effective groups are capable of completing the task and maintaining collaborative relationships among members. Members must engage in both "task-leadership actions" and "maintenance-leadership actions" to maximise successful outcomes for the group. The characteristics of effective groups were highlighted. They included: goals being clarified; two-way communication; distribution of participation and leadership; power is equalised; matching decision-making with procedures; conflict and controversy seen to be positive; high problem-solving adequacy; evaluation of effectiveness of the group; and innovation encouraged (Johnson and Johnson, 2003). We then covered the process of designing a rubric for assessing the collaborative process of small group work (Webb, 1997; Goodrich, 1996/1997).

We encouraged the groups to decide which dimensions of their collaborative work they wished to assess and to list what counts as quality work for the dimensions chosen (Goodrich, 1996/1997). Each of the thirteen groups of students worked on a rubric to "include enough detail to guide students' efforts to succeed" (Jackson & Davis, 2000, p. 60) as an effective contributor to the group.

Use ongoing assessments for feedback and adjustment

Using the guidelines of Webb (1997) and Goodrich (1996/1997), we designed a rubric to assess and give feedback on first drafts from each group (see Appendix A). The following week, we returned their rubrics with a "mark" and feedback for refinement. Each group was encouraged to "rework" their rubric by: checking whether only one element was being assessed in each point; where necessary, re-ordering the list in terms of importance; checking each element against the purpose of the assessment; and defining three gradations of quality for each characteristic using language that could be understood by university students (see Appendix B). Having identified shared characteristics, the class decided to create a common rubric based on the thirteen reworked rubrics. This was considered to enhance validity and reliability. In the third week, the common rubric was presented to the students for final comment before implementation. After their group's presentation, each student used the rubric (see below for the final version) to give their peers feedback about their contribution to the group work.

Rubric: Assessment of individual's contribution and impact on group project

PROJECT TITLE:NAME:OVERALL MARK:    /10

The individual's ability to be a contributing member of the team is reflected in his/her: Always
demonstrated
9-10
Often
demonstrated
7-8
Sometimes
demonstrated
5-6
Not
demonstrated
0-4
Participation in the group
  • Participates actively in group discussion
  • Has an overwhelmingly positive attitude
  • Prepared to accept and use others' ideas
  • Exceptional contribution as a team member
  • Often participates actively in group discussion
  • Has a positive attitude
  • Listens to others' ideas
  • High level of contribution as a team member
  • Participates in group discussion
  • Has a generally helpful attitude
  • Uses others' ideas
  • Moderate level of contribution as a team member
  • Irregular or no contribution
  • Participates in a few activities
  • Rarely listens to others' ideas
  • Struggles to contribute to the team
Personal characteristics
  • Initiative
  • Innovation
  • Task focus
  • Group Nurturance
  • Outstanding level of creativity
  • Outstanding resourcefulness
  • Offers a wide range of approaches and initiatives
  • Contributes creative ideas
  • High level of resourcefulness
  • Offers a varied range of approaches and initiatives
  • Moderate level of creativity
  • Moderate level of resourcefulness
  • Offers a limited range of approaches
  • Little or no creativity
  • Little or no resourcefulness
  • Limited or no response to problem-solving
Conflict resolution and problem solving ability
  • Caters for others
  • Responds well to criticism
  • Willing to listen to all aspects of an issue
  • Displays understanding of complexity of team dynamics
  • Displays adaptability and flexibility in resolving issues
  • Caters for others often
  • Considers critical feedback and adapts
  • Listens to both sides of an argument
  • Displays awareness of group dynamics
  • Attempts to resolve issues
  • Caters for others occasionally
  • Aware of critical feedback
  • Supports the need to resolve conflict
  • Is aware of group processes
  • Prefers others to resolve conflict
  • Little thought for others
  • Nil or adverse reaction to criticism
  • Avoids confrontations
  • Displays a tendency to avoid any conflict
Task commitment and accomplishment
  • Completes all allocated tasks on time
  • Values attendance at group meetings
  • Willingly undertakes extra research and works for the final presentation
  • Completes most allocated tasks on time
  • Attends most meetings
  • Contributes towards the final presentation
  • Completes allocated tasks
  • Attends some meetings
  • Will take on allocated task for the final presentation
  • Will complete tasks in his/her own time-frame
  • Tends to put a low priority on group meetings
  • Tends to leave final presentation tasks to others

Document and celebrate progress

Students' own beliefs about their ability to be successful in new situations are a critical variable in maintaining feelings of self-efficacy. On their final practicum our students have been encouraged to keep trying and striving for greater competence in small group strategy work beginning with small steps and encouraging their own students to complete self-reflection sheets.

What did the students learn from the "rubric" process?

In addressing this question we examined the students' reflection papers (N=70) and focused our enquiry on distinguishing explicit and implicit references to what students appear to have learnt about the potential of a collaboratively constructed rubric as an assessment "tool". We used Miles and Huberman's (1994) qualitative data analysis processes to identify the most common themes pertaining to our enquiry. Four main themes emerged, each of which is briefly discussed below:

The rubric was a powerful means for self-reflection

At this risk of stating the obvious, it was evident from the reflection papers that the students not only appreciated the value of the rubric as a quality "tool" for self-reflection, but used the criteria purposefully to refect critically on their experience of working in groups. We were impressed with the general depth of insight that students demonstrated in relation to the complexity of working in groups and the challenges of applying group work in a middle years classroom. We identified the following main insights associated with the potency of the rubric as a self-reflective strategy:

The rubric acted as a compelling motivator

It was evident that many students recognised the powerful role that the rubric played as a source of motivation. As Angela put it: " Having a rubric assessment ensures there is motivation for the students". Although most students commented positively on the inclusion of the rubric, there were indications that some considered it as an incentive with more extrinsic than intrinsic value. Having a grade value of 20% of the assignment and 10% of the year mark clearly played a role in motivating the students. Charlotte's comment reflects this: "It was always on the back of our minds that our own team members were providing us a final grade so we must therefore contribute in a positive and productive manner". However, the intrinsic worthiness of the rubric as a self-reflective "tool" was acknowledged by many students, despite the somewhat circumspect acknowledgement that its mere inclusion raised effort levels. Consider Sally's reflection: "I have to praise the inclusion of a participation incentive and self reflection task for this project, since I believe that this was sadly enough an underlying reason for my additional effort in this group project".

The rubric contributed to higher levels of equity and fairness

Associated with students' recognising the motivational role the rubric played in contributing to the overall quality of their assignment was their appreciation of the cooperative learning principle of individual accountability (Johnson & Johnson, 2003), and how this practice enhanced equity of workload. An inherent weakness of group-based assignments is the risk of social loafing, which Johnson and Johnson (2003, p. 301) define as "a reduction of individual effort when working with others on an additive group task". This issue was raised at the beginning of the course with the students, in the context of the requirements of some of their group-based assignments and in relation to pedagogical challenges in a middle years classroom. The question we posed was "how do we minimise this dynamic?" The rubric was one strategy in response to this challenge. It was thus very encouraging to note that many students identified this in their reflections. For example, Angela pointed out that "also giving students tasks individually in the group is an effective strategy to ensure individuals contribute and allows accountability to take place". Simone stated: "I consider my contribution equitable as each member contributed similar workloads and reached a satisfactory result in accordance with group standards".

Constructing a valid and fair rubric is a complex undertaking

The construction of the rubric was guided by the key assessment principles of validity and fairness (WA Curriculum Framework, 1998, pp. 37-39). It was evident that a number of students recognised that meeting these principles is a complex undertaking. Considerable time was spent developing the rubric. It took three weeks of intensive consultation before collective agreement was reached on the final product. Like most new "tools", efficacy can only be determined through application. Despite the collective wisdom that directed its construction, the rubric was still open to varying interpretations. For example, some students found that their interpretation of creativity and the level of creativity was at variance with some of their assessors. Emma found it interesting that:

... the two members I gave marks to "always" outstanding creativity also marked me "always". Whereas the least creative members, by my estimation were more critical. I would have expected more criticism from the more creative members I offered creative suggestions and invited opinion from everyone on how they felt about the suggestion. Perhaps this could have been seen as indecisive and perhaps not creative. I think that non-visual creative input to group product and process is less tangible and members may view creativity differently.

Some students raised the issue of validity. For example, Simone felt compelled to express her concern about the validity (and subsequent fairness) of her assessment of one of the group:

... to mark member E highly on 'Task Commitment and Accomplishment' because she attended every meeting, yet not reflected on the rubric is that whilst she was at meetings, her attention was focused elsewhere. (She also) contributed a number of creative ideas (scoring highly on the rubric for this category) however her timing was such that they could never be implemented and seemed to destabilise the existing status quo.

Although the quality of the rubric was considered by most students to be high, it was apparent to many that it was not the panacea of all assessment.

Conclusion

This paper focuses on our attempt to identify what our students appear to have learnt about the potential of a collaboratively constructed rubric as an assessment "tool". Four key themes emerged, namely in the way the rubric

In evaluating our 'assessment for learning' strategy, how did we "measure up"? (Herman, 1997). There have been some heartening extracts from students' reflection papers on what Wiggins and McTighe (1998) conceptualised as "enduring understandings" which are those things that anchor a unit of study. Jackson and Davis (2000) took the concept further by asserting that "these are the things we want students to remember after they have forgotten the details" (p.54). David appears to have reflected this sentiment when he said,

Throughout the year each person in this course has been taught how to work effectively in groups. This has contributed to the success of completed group assignments such as this one and provides an important insight into how group work should be handled in the middle school classroom, that is students must be taught how to work in groups.

It was also our aim to "mirror best practice". We decided before the unit commenced that one of us would be responsible for evaluating the assignment, and one of us would act as a moderator. We consider this to be have been a sound strategy, especially as it enhanced "technical accuracy" (Herman, 1997, p.200). Specifically, it enabled one lecturer to: orchestrate, monitor, and feedback to students on the progressive construction of our collaborative rubric; engage in the full kaleidoscope of team processes from a variety of group member perspectives; and collate their peers' evaluation of their performance and poster. It is apparent to us that this strategy improved the quality of the process as a whole.

As teacher educators we had been prepared to position ourselves as master learners who valued improving the quality of teamwork, and had faith in our own, and our students' ability to construct, refine, and use the rubric effectively as reflective practitioners. Most importantly, for us the process had served to build our community of learners "to prepare students to embrace what Peter Ellyard (1998) terms the 21st century learning culture, and capitalise on opportunities it provides" (Leggett, Lichtenberg, Newhouse-Maiden & Harvey, 2001, p.1).

References

Blair, G. (1998). Starting to manage: The essential skills. UK: Chartwell-Bratt.

Bennett, B., Rolheiser, C. & Stevahn, L. (1991). Cooperative learning: Where heart meets mind. Ontario: Educational Connections.

Curriculum Council, Western Australia (1998). Curriculum framework from kindergarten to year 12 education in Western Australia. Perth, WA: Author.

Edith Cowan University (2003). Orientation Activity Notes for the Graduate Diploma in Education Middle years Orientation Program. Joondalup, Perth, Western Australia.

Edith Cowan University (2002, March). ECU Graduate Attributes. Perth, Western Australia: Academic Board Attachment 6 Executive Summary: Author.

Flinders Univ@2001 - Group Work. http://www.flinders.edu.au/teach/teach/groupwork.htm

Graves, N. & Graves, T. (1990). A part to play: Tips, techniques and tools for learning cooperatively. Victoria, Australia: Latitude Publications.

Goodrich, H. (1996/1997). Understanding rubrics. Educational Leadership, 54(4), 14-17.

Harvey, M., Leggett, B., Newhouse-Maiden, L. & Lichtenberg, A. (2001). The 'ready, fire, aim' curriculum strategy of a university teaching-learning team: The challenge for teacher educators of curriculum responsiveness in schools. Paper presented at the Annual Conference of the Australian Association for Research in Education. Notre Dame University, Fremantle, 3-6 December.

Herman, J. (1997). Assessing new assessments: How do they measure up? Theory into Practice, 36(4), 196-204.

Jackson, A.W. & Davis, G.A. (2000). Turning points 2000. Educating adolescents in the 21st century. A Report of Carnegie Corporation of New York. New York: Teachers College Press.

Johnson, D. & Johnson, F. (2003). Joining together: group theory and group skills. (8th ed.). Englewood Cliffs, NJ: Prentice Hall.

Learning Connections (2003). Learning Guides: Working-Teams. University of South Australia. http://www.unisanet.unisa.edu.au/learninfconnections/learningres/learning/index.html [URL not found 20 Jun 2004, see http://www.unisanet.unisa.edu.au/learningconnection/]

Leggett, B., Lichtenberg, A., Newhouse-Maiden, L., & Harvey, M. (2001). Curriculum integration as process and product: Authentic learning in teacher education. Paper presented at the Australian Curriculum Studies Association Biennial Conference. "Education futures and new citizenships" 29 September - 1 October.

Mannison, M. (1994). Interactive teaching strategies: 38 cooperative learning strategies with classroom examples. Queensland, Australia: Nice Publications.

McTighe, J. (1996/1997). What happens between assessments? Teaching for Authentic Student performance. Educational Leadership, December 1996/January 1997, pp. 5-12.

Miles, M.B. & Huberman, M.A. (1994). Qualitative data analysis. Thousand Oaks, CA: Sage.

Muth, D.M. & Alvermann, D.E. (1999). Young adolescent development in teaching and learning in middle schools. (2nd ed.). Chap 2, 12-47. Needham Heights, MA: Allyn & Bacon.

Newhouse-Maiden, L.P. (2002). Middle schooling: Does it meet the needs of the young adolescent or is it just a passing fad? St Mary's College Fellowship Lecture 2002. Durham, UK: St Mary's College and The Society of Fellows of Durham University Research Foundation

Stowell, L., & McDaniel, J. (1997). The changing face of assessment. In J.L. Irwin (Ed.), What current research says to the middle level practitioner (pp. 137-150). Columbus, OH: National Middle School Association.

Webb, N.M. (1997). Assessing students in small collaborative groups. Theory into Practice, 36(4), 205-213.

Wiggins, G., & McTighe, J. (1998). Understanding by design. Alexandria, VA: Association for Supervision and Curriculum Development.

Appendix A

Appendix A graphic


Appendix B

Appendix B graphic

Authors: Dr. Lesley Newhouse-Maiden and Dr. Terry de Jong School of Education, Edith Cowan University, Perth Western Australia
Tel: 6304 5485 Fax: 6304 5850 Email: l.newhouse_maiden@ecu.edu.au, t.dejong@ecu.edu.au

Please cite as: Newhouse-Maiden, L. and de Jong T. (2004). Assessment for learning: Some insights from collaboratively constructing a rubric with post graduate education students. In Seeking Educational Excellence. Proceedings of the 13th Annual Teaching Learning Forum, 9-10 February 2004. Perth: Murdoch University. http://lsn.curtin.edu.au/tlf/tlf2004/newhouse-maiden.html


[ Proceedings Contents ] [ Teaching and Learning Forum 2004 ]
This URL: http://lsn.curtin.edu.au/tlf/tlf2004/newhouse-maiden.html
Created 19 June 2004. Last revision: 19 June 2004.