Teaching and Learning Forum 2000 [ Proceedings Contents ]

Online assessment: The use of web based self assessment materials to support self directed learning

Mary Peat
School of Biological Sciences
The University of Sydney
    It has been accepted that well documented formative feedback for students, especially early on in a course, facilitates their learning. This, of course, is expensive and in recent years in Australia (as around the world) there has been an overall reduction in resources forcing the abandonment of certain learning experiences in order to constrain the costs. This is particularly so in the sciences which traditionally have laboratory classes of several hours per week and with associated written reports and assignments. Thus science teachers with large student numbers continually confront the problem of how to give sufficient feedback to help students gain a better understanding of their abilities. Self assessment opportunities offer one solution towards this problem. If students are offered materials to help them identify both their level of understanding and those areas that they might need to concentrate on, then we are helping their academic development and, indirectly, their self esteem. The design of self assessment materials needs to take into account the diversity of student learning styles, so that opportunities presented to students ensure maximum participation by them. A set of self assessment modules (SAMs) designed to help students identify their learning outcomes will be presented and discussed. The rationale for the design of SAMs is grounded in Bloom's Taxonomy, and the materials are developed on four levels of difficulty. The lowest level of difficulty tests content and knowledge, followed by levels testing application and analysis with the highest level requiring synthesis of information. Student use of these resources will be discussed and their perceptions of the value of these materials will be presented.
Teaching and Learning Forum 2000 Home Page


Introduction

Feedback on performance that provides opportunities for students to gain insight into their understanding of the course content is a valuable tool in the learning process (e.g. Zakrzewski and Bull, 1998). In the past early formative feedback was considered a normal component of science based courses (as in the use of weekly quizzes, paper based laboratory reports, solutions to questions, etc.) and much of it was followed up with face to face encounters between teachers and students. Increasing student numbers and decreasing staff resources expedites the need to develop self help methods for students. The whole issue of giving early feedback to first year students needs to be revisited. The judicious use of self assessment on the web is a viable option that can provide valuable information for students about their progress (e.g. Edwards, 1989). This paper will discuss self assessment materials available on the web to a large first year biology class, and in particular will concentrate on the special features of a set of self assessment modules designed to test different cognitive levels of learning.

The problem

First year biology students at the University of Sydney form a large, highly heterogeneous group, characterised by varied academic backgrounds (with a large range in the incoming entry grades, and some without biology), with varying interest in biology, and with a range of incoming generic skills (writing, computer, team work, etc.). In addition many of the students have an expectation of being spoonfed, a residue of a high school surface approach to learning. With reducing resources, giving students sufficient appropriate support and feedback is becoming more difficult. One solution is to use the web to deliver self assessment materials that allow the students to gain a measure on their understanding of course materials.

The solution

For over a decade, within the first year biology courses at The University of Sydney, there has been a move to change from teaching students to facilitating student learning, especially within the laboratory experience. Early changes involved organisational and structural issues, such as the design of laboratory classes, or the review of activities suited to a student centred learning approach. Small learning communities in large classes were created that offered group activities designed to facilitate learning (Franklin and Peat, 1996). The introduction of computers in 1994 as a permanent feature in the laboratory, and the development of computer assisted learning modules (CAL) to target specific learning difficulties, led to a focus on the use of computers in the learning process. In particular, the use of revision and self assessment materials was targeted. Materials were moved online from the middle of 1997. Students are able to visit a drop in centre (Resources Room) and a Virtual Resources Room (VRR, 1999). Within the Virtual Resources Room (VRR) special attention was paid to the development of online self assessment materials which include weekly quizzes, a mock exam and self assessment modules.

Self assessment modules (SAMs)

Anecdotal evidence from students had indicated that many (possibly those at the lower end of our intake) found it difficult to understand key associations and concepts. To facilitate this understanding we developed appropriate self assessment materials. The aim of a SAM is to draw together related parts of the course and to promote a deeper learning strategy, whilst providing an enjoyable feedback and reinforcement session. Whilst the courses are thematic, e.g. the first semester course covers the requirements of life, basic ecology, role of microbes, cells and energy (metabolism), cells and chromosomes (mutations, evolution), the SAMs are organised around plants, animals and microbes. Thus in using a SAM students are taken down a lateral pathway and so encouraged to see the relationship between the materials.

Each SAM tests the students on four levels of increasing difficulty, using Bloom's Taxonomy (Bloom, 1956). The content of the questions can be re-used (from level to level) with an increasing cognitive requirement. Appropriate question types have been developed for each level of difficulty with Level 1 testing content and knowledge, Level 2 testing application of content, Level 3 testing analysis and Level 4 testing synthesis of information (see Table 1 for details).

Table 1: Question levels in self assessment modules

SAM
level
Cognitive
level
Question types Number of questions
Level 1Content and knowledge Use multiple choice questions and drag and drop scenarios - the answer is always on the screen. 7-8
Level 2Application Use some multiple choice and drag and drop, but mostly use format that expects text input from the student - answer not always on the screen. 4-5
Level 3Analysis As for Level 2 but with question formats expecting students to build diagrams, flowcharts etc. 3-4
Level 4Synthesis of information Expect student to be able to synthesise information in response to a question. May involve writing prose - use of example answers here for student to score her/himself on a 0-5 scale bar. 1-2

SAMs are used by students individually or in groups (allowing for peer discussion, cooperative learning and team work). There is an entry point for each SAM which is designed to act as a barrier to progression if the student in unable to successfully complete it. The entry point consists of a set of five true/false questions allowing students with 80% correct to proceed and denying further access to those students unable to achieve this level of performance. Once past the barrier students can choose to do any level in any order and the program allows the students to know where they have been and how they are performing. At any time they can ask for a progress report. The design of templates for the questions is discussed elsewhere (Peat, Franklin and Mackay-Wood, 1997).

Evaluations

Evaluation of the SAMs during their development and of the final product during use has included students and staff in surveys, focus groups, unsolicited emails and reviews. Feedback on the use of the SAMs has been on a voluntary basis, using an optional logout feedback form which is automatically generated after completion of the module. This asks about the best aspects of the module, the worst aspects of the module and how the module could be improved. Analysis of responses has given useful information for improvements. With respect to the best aspects, the student comments are of a generic nature and not about the content. The students like the immediate feedback, the clarity of material, the variety of question types, the interactivity of the questions, the ease of use and the availability (any time, any place, any pace). In addition they indicate that such a package is motivational (not intended in the design), fun to use and good for revision. The responses to the 'worst aspects' highlight two areas - technological problems (mostly to do with hardware problems, e.g. insufficient RAM, and not in our control), and details about content and in particular criticism about the restrictions imposed by an entry requirement (80% pass for entry).

Many students do not see the entry test to have any value and these students are using the module for revision and not self assessment and so want to go through the entire module even if they are not going to perform very well. The entry test may, therefore, be inappropriate. Some students indicated that they repeat the entry level test until successful rather than doing any revision and that an 'inability' to pass the entry test does not mean they cannot answer the questions in the main test. This is to be expected from the design with four levels of increasing cognitive requirement and the entry test on a Level 2-3 degree of difficulty. The hardware problem will be addressed with additional information online for the students about the technical requirements of the SAMs. The entry requirement will be removed, especially as it appears students are using the SAMs as a revision aid as well as a performance indicator.

The SAMs are accessed from a Virtual Resources Room (VRR) on the web. A recent academic peer review of this virtual learning environment (Jones, 1999) strongly recommended this site, and in particular the self assessment materials.

Discussion

It has been shown (Entwistle, Hounsell, Macaulay, Situnayake, & Tait, 1989) that an important contributing cause of failure of first year students is an absence of feedback on progress, and this is also cited as a reason to discontinue (McInnis, James & McNaught, 1995). In larger classes it is often difficult to provide individual feedback to students. However, it is not always necessary for academic staff to give feedback if another option is available and self assessment is a suitable option (Ramsden, 1982). The self assessment materials in this study offer students a variety of opportunities to test their understanding, and to identify their cognitive working level without the need to seek face to face assistance from the teaching staff. The students in the present study found the materials challenging, helpful and beneficial. This is supported by other studies on the use of self assessment materials. For example, in a similar study with first year physiology students 98% of students thought that the self assessment had made them think more and 71% felt they learnt more (Orsmond, Merry, & Reiling 1997).

The design of the SAMs does not utilise the full range of Bloom's taxonomy (Bloom, 1956). Of Bloom's six categories, evaluation - the ability to evaluate the worth of material, theories, methods, information etc for a given purpose, was not included in the design of the SAMs, as it was considered too difficult to portray using the computer and was best assessed by other means. A criticism of the SAMs is that we have not given the students sufficient support in using them. Fazey (1993) argues that students may need more instruction in the use of self assessment materials, and should be provided with guidance and given opportunities to practice these skills. A future development, therefore, will be to include a set of performance outcomes for students to identify with, based on some real data. For example a high performance at Level 1 and 2, with poor performance at Level 3 and 4 might indicate good rote learning skills but poor application and analysis.

Conclusion

The use of SAMs is one way that students in first year biology at The University of Sydney are being encouraged to take control of their learning. Student responses show that the mix of materials online for them to use is helping them to learn, in a way that suits their lifestyle and which we hope will enhance opportunities for participation in higher education. University teaching and student learning are moving through transition processes, driven by many factors including changing student requirements and economic forces. In the next few years we all need to adjust our mix of online and face to face teaching experiences and develop measures to evaluate the appropriateness of them. The next few years will show us the ways in which students will want to learn and we need to be flexible in accommodating their preferred learning strategies.

References

Bloom, B. S. (1956). Taxonomy of Educational Objectives: Handbook 1, Cognitive Domain. Longman, New York.

Edwards, R. (1989). An experiment in student self assessment. British Journal of Educational Technology, 20(1), 5-10.

Entwistle, N. J., Hounsell, C. J., Macaulay, C., Situnayake, G. and Tait, H. (1989). The Performance of Electrical Engineers in Scottish Education. Report to the Scottish Education Department, Centre for Research on Learning and Instruction, Department of Education, University of Edinburgh.

Fazey, D. M. A. (1993). Self assessment as a generic skill for enterprising students: The learning process. Assessment and Evaluation in Higher Education, 18(3), 235-250.

Franklin, S. and Peat, M (1996). Mechanisms for facilitating group learning in first year biology: Assisting the transition. Proceedings of the 2nd Pacific Rim Conference on the First Year Experience in Higher Education, Melbourne, 3-5 July, 1996, pp.233-242.

Jones, S. (October, 1999). First Year Biology Virtual Resources Room. http://science.uniserve.edu.au/newsletter/vol13/jones.html [Accessed 23 Dec 1999]

McInnis, C., James, R. and McNaught, C. (1995). First Year on Campus: Diversity in the initial experiences of Australian undergraduates. Centre for the Study of Higher Education, University of Melbourne.

Orsmond, P., Merry, S. and Reiling, K. (1997). A study in self assessment: Tutor and students' perceptions of performance criteria. Assessment and Evaluation in Higher Education, 22(4), 357-369.

Peat, M., Franklin, S. and Mackay-Wood, R. (1997). The development of self assessment modules: Use of tailor made templates. Virtual CUBE'97 at http://www.liv.ac.uk/ctibiol/vCUBE97/html/rob_mackay-wood.html [Accessed 23 Dec 1999]

Ramsden, P. (1992). Learning to Teach in Higher Education. Routledge Press London and New York.

VRR (October 1999). Virtual Resource Room. http://fybio.bio.usyd.edu.au/SOBSFYB/FYB_StuRes.html [Accessed 23 Dec 1999]

Zakrzewski, S. and Bull, J. (1998). The mass implementation and evaluation of computer based assessments. Assessment and Evaluation in Higher Education, 23(2), 141-152.

Please cite as: Peat, M. (2000). Online assessment: The use of web based self assessment materials to support self directed learning. In A. Herrmann and M.M. Kulski (Eds), Flexible Futures in Tertiary Teaching. Proceedings of the 9th Annual Teaching Learning Forum, 2-4 February 2000. Perth: Curtin University of Technology. http://lsn.curtin.edu.au/tlf/tlf2000/peat.html


[ TL Forum 2000 Proceedings Contents ] [ TL Forums Index ]
HTML: Roger Atkinson, Teaching and Learning Centre, Murdoch University [rjatkinson@bigpond.com]
This URL: http://lsn.curtin.edu.au/tlf/tlf2000/peat.html
Last revision: 20 Feb 2002. Curtin University of Technology
Previous URL 23 Dec 1999 to 20 Feb 2002 http://cleo.murdoch.edu.au/confs/tlf/tlf2000/peat.html