Teaching and Learning Forum 95 [ Contents ]

Small group peer assessment of the human computer interface: A case study

Jean Hall
Faculty of Science, Technology and Engineering
Edith Cowan University
The design of the human computer interface is a major determinant of the usability of computer software. At Edith Cowan University, second year Computer Science students learn software application development by developing a small software project in an experiential environment. This paper reports on the initial exposure of a small, close knit group of students to peer and self assessment when peer, self and academic supervisor assessment were all used to assess the human computer interface of software projects.

A comparison of student opinions about peer assessment canvassed before and after the assessment exercise, shows a trend to changing attitudes. Assessment scores awarded by the academic supervisor, the student themselves and their peers are compared. Students tend to assess themselves lower than they were assessed by their peers. The academic supervisor tends to assess students slightly higher than the students' peers. Finally some issues relating to the use of small group peer assessment as an aid to student learning rather than solely an assessment tool are discussed.


Introduction

This case study describes a small group of Computer Science students undergoing assessment by their peers. The research group ETALICS (Effective Teaching and Learning in Computer Science) funded data collection and entry by student P. Smith. Their help is gratefully acknowledged.

The "Human Computer interface" (HCI) is that part of a computer system with which users interact. ie input data to, or read information from. Preece (1993:14) states that the function of the HCI is to allow users to carry out their tasks safely, effectively, efficiently, and enjoyably. The HCI primarily involves computer screens and hardcopy printouts but also multimedia elements sound, video etc. This case study was restricted to the assessment of student project work in designing computer screens and paper reports.

Second year Computer Science students on the Bunbury Campus of Edith Cowan University studying a unit in Application Development were involved in this study. This was a small and closely knit all male but otherwise not particularly homogenous group of 12 students. One third of the students were part time and one half mature aged, most were Australian and a couple were fairly recent immigrants. These students were highly competitive academically and had worked together closely for two years and knew each other well. Socially they tended to divide into two subgroups. Considerable trust had built up over time between the students and a good rapport with their female lecturer was evident. They had worked together before.

There were several reasons for undertaking this project at this time. On completion of this unit students would be moving into the final year of their course where they would be required to undertake a year long group project with a peer assessment component. There was an unfortunate history of previous students misunderstanding the requirements of that particular peer assessment exercise with appeals and reappeals in two previous years. Bunbury campus students are a small close knit group and all students were well aware of this sad history. An opportunity was awaited to shed a more positive light on the prospect of peer assessment and to give students experience and confidence in the validity of this activity before commencing their final year project.

For some time, the lecturer had considered issues probably as old as education itself and no doubt similarly mulled over by many academics in other disciplines. What is art and what is good design? How do you teach these? What enhances functionality? Does it all depend on the eye of the particular beholder? What part do these considerations play in educating IT professionals to produce better HCIs?

The final trigger for this study came from a student who presented a screen designed for a University course information system in a workshop. The lecturer considered the HCI inappropriate but the student strongly disagreed. He pointed out that throughout his course he had been encouraged to profile the prospective users of his computer systems (age, gender, culture, education, expertise etc) and then design screens considering that profile. He stated that he had "designed for a fellow young student user just like him, not for a middle aged female Computer Science lecturer". Gender and age bias aside, he had made a valid point if not won a complete rational argument. Thus was the peer assessment project born.

Method

Students and lecturer discussed the possibility of peer assessment contributing to part of their final project mark. It was agreed to restrict the peer assessment to the HCI component of the project (5% of the final mark for the unit). Student project HCI screens and reports were assessed by the student designer, his peers and the lecturer. Only the average of the peer's ratings contributed to the final mark.

Student opinion about the peer assessment exercise was surveyed in a pre assessment survey Students raters were then given a marking guide / feedback instrument to use for the peer assessment exercise. See Fig. 1. This guide was carefully explained to them. Student time-keepers and marks collators were appointed. The projects were assessed over a continuous four hour period with regular breaks. Each ratee demonstrated his project and the raters assessed it. After each project demonstration, all student raters present handed their ratings to the student marks collator who tallied, discarded the highest and lowest, averaged and recorded the individual peer ratings before passing the marking guides / feedback forms on to the ratee providing written feedback. The ratee had in the meantime completed a post assessment survey and recorded his self rating of his own HCI.


NameMark / 5

Feedback Comment
Screen designEGSPM
Field placementEGSPM
Report designEGSPM
Menu designEGSPM
Use of GUI objectsEGSPM
Use of colourEGSPM
NavigationEGSPM
Task oriented functionalityEGSPM
IntuitivenessEGSPM

Rating Guide: E excellent, G good, S satisfactory, P poor, M missing

Figure 1: Peer rating marking guide / feedback instrument

Students were required to attend and rate at least the demonstrations before and after their own presentation but most students chose to rate all the demonstrations. The third year final project students also showed considerable interest and many attended some demonstrations and filled in rating forms. At the request of the ratees, these ratings were not reflected in the final scores. However they were passed on to the ratees and provided additional and welcomed feedback.

Quantitative data from the two surveys was processed in SYSTAT a statistical analysis package. An Access database was used to record qualitative survey comments and the ratings were compared and charts produced using an Excel spreadsheet.

Results - comparison of self, peer and lecturer assessment

The average ratings of peers, self rating and lecturer rating were compared - see Fig 2. The ratings obtained by the three methods were similar. On average, self assessment was 1.6% lower than peer assessment which in turn was 1.8% lower than Lecturer assessment. These similarities demonstrated the validity of the peer rating process to the students.

Figure 2

Figure 2: Comparison of Peer Average, Lecturer and self rating of the HCI

Results - student attitudes pre and post the assessment exercises

The surveys conducted pre and post the peer assessment exercise showed plenty of student confidence in the process and some movement in attitudes over time.

Students confidence in their own ability as assessors was 100% in the pre survey dropping to 81% in the post assessment survey. Pre survey student comments were optimistic: "I reckon I can give a reasonably fair review", "Don't know art but I know what I like". In spite of this 100% confidence, no student reported being aware of how to assess someone else's work in the pre assessment survey but 73% knew how to do this in the post assessment survey. The post survey also reflected some more realistic views familiar to most teching Academics. "It's harder to put a mark on someone's work than you might think", "Consistency is a problem".

Student confidence in their peers' competence to rate work grew with their own experience of the rating process. In the pre survey ie before the assessment exercise only 54% of students were confident of their peers' competence to review their work. This increased markedly to 81% in the second survey. Student comments supported this view. "Other students are a pretty good judge of how good your work is", "Students know a good or bad piece of work when they see it", "Lack of competence in student markers is outweighed by other advantages".

Possible peer bias issues which might concern students were canvassed in the pre survey. This concern was unfounded as students were generous in their expectations of their peers. 82% of students were confident that their peers' competitive instincts and wish for high grades would not influence their ratings. A similar 82% felt that friendships, personal likes and dislikes would not be a consideration.

In general this group was very positive about the peer assessment process. In both surveys, 27% of students were strongly positive and the remainder hopeful. No student reported that they were neutral, concerned or very worried about the peer assessment exercise.

Other results

Students supported the validity of peer assessment. 55% of students felt that this was a worthwhile way to assess their work with 45% unsure and no negative responses. I consider this is a positive response as students were surveyed after their demonstrations but before they had seen their peer's ratings.

Many students commented favourably on the broadening of the marking base: "Receive views from larger user group", "An averaging of many opinions - good", "Get a lot of different opinions of your work", "Everyone knows what is difficult more than the lecturer (no offence meant)", "See my work through the eyes of others" and "All ideas shared are bound to be beneficial either critical or complimentary".

Students gained other experiential learning outcomes. 82% reported that this was a worthwhile way to learn about peer review, thus fulfilling the covert goal of this exercise. Nearly half the students reported that this exercise would influence the way they prepared future assignments whether for peer or lecturer assessment. Student comments included: "Learning process for marker", "This is a good learning experience of what happens in the real world.", "I saw students trying to sell their product" and "I learnt capabilities I was previously unaware of".

Factors contributing to the success of peer assessment

The students in this case study were very confident both in their own abilities and the fairness of their peer raters. Montgomery (1986: 19) suggests that given certain procedural conditions, peer assessments are indeed highly reliable. She found them also to be valid. Bias of the rater accounted for only a very small portion of her rating variance i.e. the raters reflected differences in ratees' performances rather than any biases about the ratees.

It is possible that the group cohesiveness and student familiarity with each other may have been contributing factors. Ramsay and Lehto (1994: 40) report that in a manufacturing organisation, peer assessment teams had difficulty evaluating new employees whom they do not know well. No such difficulty was reported in this case study. Trust in the integrity of the University system and the Lecturer may be other important factors contributing to the successful outcome of this peer assessment exercise. Hughes and Grote (1993: 57) consider similar factors an important success determinant in peer review in industry.

Project outcomes

Students benefited from this exercise and will enter the final year of their course with a better understanding of what is involved in peer assessment. The validity of this was demonstrated to them with comparable ratings returned by their peers, their lecturer and themselves.

This exercise was not just a means of assessing students but also an experiential learning activity. The educational insight gained by the process itself was a valuable outcome. Students felt that in assessing others' efforts they learned a great deal about HCI design. They acknowledged that this experience would be valuable both in the workplace and in the design of future material for assessment.

One of the students deserves the final word.: "By assessing, you learn to do better".

References

Hughes, P. A. and Grote, R. C. (1993, Jun.). Peer review places trust with employees. HR Magazine, 38(6), 57-60

Montgomery, B. M. (1986, Feb.). An interactionist analysis of small group peer assessment. Small Group Behaviour, 17(1), 19-37.

Preece, J. (ed.) (1993). A Guide to Usability. Human Factors in Computing. Wokingham, England: Addison Wesley.

Ramsay, M. L. & Lehto, H. (1994, Jul.). The power of peer review. Training and Development, 48(7), 38-41.

Please cite as: Hall, J. (1995). Small group peer assessment of the human computer interface: A case study. In Summers, L. (Ed), A Focus on Learning, p111-115. Proceedings of the 4th Annual Teaching Learning Forum, Edith Cowan University, February 1995. Perth: Edith Cowan University. http://lsn.curtin.edu.au/tlf/tlf1995/hallj.html


[ TL Forum 1995 Proceedings Contents ] [ TL Forums Index ]
HTML: Roger Atkinson, Teaching and Learning Centre, Murdoch University [rjatkinson@bigpond.com]
This URL: http://lsn.curtin.edu.au/tlf/tlf1995/hallj.html
Last revision: 16 Apr 2002. Edith Cowan University
Previous URL 18 Feb 1997 to 15 Apr 2002 http://cleo.murdoch.edu.au/asu/pubs/tlf/tlf95/hall111.html