Teaching and Learning Forum 2006 Home Page

Category: Professional practice
Teaching and Learning Forum 2006 [ Refereed papers ]
A case study of the effectiveness of WebCT as a student-learning tool and platform for structured assessment

Yamin Ma
School of Earth and Geographical Sciences
Sandra M. Saunders
School of Biomedical, Biomolecular and Chemical Sciences
The University of Western Australia

WebCT (Web Course Tools) is an online learning system used at the University of Western Australia, Australia. This current study examines how effective the system has been in aiding student learning for three different levels of an undergraduate science degree. In addition, the use of WebCT as a platform for online assessment was tested on a large first year science class. The results of this current study can give some guidance for teachers on the construction of a WebCT, online study environment for students at different levels in their science degree. Briefly, we found that in setting up an online environment in WebCT, instructors will have to consider both class size and level of the undergraduate degree. We conclude that although science students at all levels preferred face to face lectures and tutorials to studying in an online environment, these same students found WebCT to be a valuable resource in aiding their studies, much the same as traditional study aids, such as library resource books. However, the extent to which students can make use of WebCT and other online environments will ultimately depend on how much time staff are willing to expend on developing such an environment. With regards to developing online assessment tools for large first year science classes, the initial testing of a structured multiple choice format proved amenable to the students.


Introduction

Online study aids have been successfully implemented in disciplines such as dentistry and have proven to be valuable for students (Browne et al. 2004; Gupta et al. 2004). On the other hand, it has been shown that e-learning does not make a dramatic improvement in students' academic performance (Keasar et al., 2005; Mackay, S, 2005). Further, research has shown that students, given the option of having an online course component, preferred it to supplement traditional face to face lectures rather than replace them (Gupta et al., 2004; Keasar et al., 2005). Thus, do online study aids actually "aid" students in their study? Few parallel studies outside of dentistry and medicine have been carried out on the effectiveness of online study aids in general science degrees. Particularly, few studies have looked at how students at different levels in an undergraduate degree would use such online study aids. A study by Gupta et al. (2004) on the use of computer aided learning for dentistry students has shown that although the students found the e-course a good supplement to traditional methods of learning, the staff on the other hand, expressed negative views on e-learning. The experience at the University of Western Australia, Australia, is such that it is up to the staff to develop their own sites (on the online platform, WebCT (Web Course Tools)). Hence, the quality of the e-learning experience for the student could depend on how much staff value e-learning and hence how much effort they are willing to put into developing such a web site. In addition, it is a recognised criticism in the core science disciplines such as chemistry that assessment levels at greater than 80% in final exams are far too heavy. Online assessment could be a possible solution to alleviating such heavy weightings on assessment by breaking course assessments into smaller, more manageable mini-tests, especially when dealing with larger class sizes.

Background

WebCT is an online learning system used at UWA which provides academic staff with an opportunity to post their coursework and manage communications with their students. It is an optional tool that is provided for academic staff to help achieve their course outcomes (Diana Jonas-Dwyer, personal communication, 24 April 2005). WebCT is currently used in various science courses, particularly at the first year level, where large class sizes mean that WebCT provides an efficient communication tool between staff and students. The students surveyed in this study were from science courses at various levels in their undergraduate degrees. WebCT is provided in these courses, albeit with different intended outcomes. In the first year science course, WebCT is used as a vehicle for information (ie. the dissemination of notices, recorded lectures (i-lectures) and the provision of answers to laboratory exercises). At the second and third year level, the focus shifts towards providing additional resources to supplement lecture material in addition to lecture notes. These resources may include, for example, links to the latest research articles and unassessed self quizzes for revision purposes.

The following research questions were raised:

  1. Does WebCT aid student learning?
  2. Can WebCT be used as an assessment tool for science course coordinators handling large class sizes (more than 300 students) at the first year level?
The aims of the study were then to firstly, obtain a survey of how undergraduate science students at different levels of their undergraduate degree utilise WebCT. Guidelines could then be developed to enable academic staff to design WebCT sites aimed at specific levels in the science undergraduate degree. A further aim of the study was to determine if WebCT can be used as a platform for developing online assessment for a large first year science class. This was carried out by implementing a "dry run" online problem solving task in the form of a tutorial based multiple choice questionnaire. Our final aim was to gain a preliminary idea of staff attitudes towards implementing e-learning in their courses.

This present study grew out of a need to assess how useful WebCT has been to third year Soil Science students, after two years of implementing WebCT in that particular unit. First and second year undergraduate science students were also included to give us a more complete picture of how useful students were finding WebCT overall. The inclusion of first and second years also gave us a good contrast of WebCT usage within different levels in undergraduate science courses.

Methods

A questionnaire was prepared for first year science students (sample size and number of responses = 123) in the School of Biomedical, Biomolecular and Chemical Sciences, and second (sample size and number of responses = 56) and third year (sample size and number of resources = 23) science students in the School of Earth and Geographical Sciences at UWA. The questionnaire was designed to assess how effective WebCT was in aiding student learning for students in the science streams. A sample of the questionnaire is shown in Appendix 1. A second questionnaire was designed for the first year students to provide further feedback on using WebCT as a platform for a tutorial style problem worksheet. This second questionnaire was also designed to provide further feedback as to whether students continued using WebCT as a study aid during the term. A sample of the second questionnaire and online worksheet are shown in Appendices 2 and 3. Data collection and statistical testing were carried out in Microsoft Excel. Preliminary face to face interviews with two members of staff in the schools of Biomedical, Biomolecular and Chemical Sciences and Earth and Geographical Sciences were conducted to obtain staff perspectives of implementing such online study environments.

Results and discussion

First year science students

Of the first year students surveyed 91% indicated that they used WebCT in the particular unit (Figure 1). The main reason cited by students for not using WebCT was that they had found it unnecessary. From the available online content, first year students used WebCT to obtain answers to the laboratory questions and tutorial problem sheets and to access past exam papers (Figure 2). In addition to WebCT, students cited attending lectures and/or interaction with lecturers, reading recommended texts and their lecture notes as the top three resources they used as study aids (Figure 3). When asked if WebCT had enhanced their interest in the subject, the majority answered in the negative (62%). However, a 96% majority said that they found WebCT to be useful overall. Further, of the first year students, 77% generally preferred face to face methods of study like attending lectures and tutorial sessions rather than having an online alternative. However, when asked if they preferred online environments like the one provided in WebCT to more traditional study aids such as library references, 70% answered in the positive, possibly because of the ease of access to the Internet. Overall, students found WebCT to be useful (95%).

Figure 1a

Figure 1b

Figure 1c

Figure 1: WebCT usage at different undergraduate levels showing
high usage of WebCT by students at all levels in undergraduate science courses.

Figure 2a

Figure 2b

Figure 2c

Figure 2: WebCT content accessed by students at different levels in undergraduate science degrees. The bar graphs show a trend of a preference of higher level students (3rd Years) for using WebCT as a source for further resources.

Figure 3a

Figure 3b

Figure 3c

Figure 3: Additional resources (in addition to WebCT) quoted by students as study aids. The graphs show that students at all levels in undergraduate science degrees still preferred using traditional study aids such as attending lectures and reading lecture notes and recommended texts or references.

Second year science students

The majority of second year students said they used WebCT (96%). As was the case for first years, the main reason cited for second year students not using WebCT was because they found it a hassle. Students mainly logged into WebCT for obtaining access to lecture notes (86%), with the second most popular reason for logging in being to access the "self quizzes". Of the students surveyed 77% indicated that they used WebCT for catching up on lecture material due to illness or lecture clashes. Although 89% of students preferred attending lectures, for obtaining additional information on the subject, they preferred using online methods rather than going to the library (66%). Overall, 97% of the students thought WebCT was useful in aiding their studies.

Third year science students

Of the 22 students surveyed, 96% said they used WebCT for that unit, mainly to access resources to supplement the lecture material (Figure 2), although 82% said they also used it to catch up on lecture material, possibly for revision purposes. Of the third year students, 78% responded positively when asked if they preferred using online study methods to obtain further information on lecture material than more traditional methods like sourcing library books. 95% said they preferred attending lectures and face to face interaction with lecturers than WebCT as a source of additional information. Finally, 96% of students found WebCT to be useful overall.

Comparison within undergraduate years

Overall, judging from the preference of students for attending face to face lectures and that greater than 95% found WebCT useful in aiding study, students at all levels seem to be using WebCT as a supplementary tool to aid their learning. Further, the data seem to suggest that deeper learning becomes more evident as students progress to the third year level. At the third year level, students are more independent and seem to use WebCT as a vehicle for further resources to that given in lectures, eg. online links and research papers. Comparing their responses to that of the first year students, we can see that the first years were mainly accessing WebCT for answers to the laboratory questions, access to past exam papers and the online tutorial. This fits in with MacKeracher's (1998) model of learning development in young adults. As young adults make the transition from secondary to the post-secondary education system, they make the transition from seeing knowledge as absolute in first/second year to one in which it is subjective and independently sourced in third year.

Further, staff interviewed expressed the concern that accessibility may limit the extent that WebCT maybe useful to students. However, on the whole, all students surveyed have some form of access to WebCT because all of them were able to comment on it. Of this number, 91% of all students surveyed had access to WebCT outside of campus. In addition, students can easily access the internet on campus. Hence, accessibility is probably not an issue in implementing WebCT as a student learning tool. The issue here seems to be more that of a matter of choice. All respondents who did not use WebCT chose not to use it with the main reason being cited as they found it unnecessary. Responses at all undergraduate levels showed that 59% said that WebCT had not enhanced their interest in their subject area, although 87% had said that it had enhanced their understanding of the subject, with similar responses at all three undergraduate levels. Interestingly, a study by Keasar et al. (2005) reported similarly negative responses (53%) to such online study aids in two undergraduate biology courses. This begs the question as to why such online study aids do not increase a student's interest in a subject and promises to be an area for further in depth research.

Developing online assessment

With regards to developing online assessment tools the initial survey posed questions on the students approach to the tutorial and problem solving work. While 47% said they would attempt in on their own, 25% said they preferred to attend a tutorial session, and 16% said they preferred to use the WebCT option. The remaining 8% of respondents who expressed a preference said they would not do any work until the exam period as using WebCT is not a requirement for assessment. For the new structured multiple choice problem format within WebCT, only 39% of the returned surveys had used the resource. This could have been because the item was introduced in the early weeks of the semester when many students had not accessed the WebCT area at all. However, the responses to the structured multiple choice problems were very positive, finding it at this stage a useful study aid which they could work on independently (97% responded saying they worked on these problems on their own). In contrast, it was interesting to note that 84% of the returned surveys indicated a preference for face to face tutorial sessions (Figure 4). A final question was posed asking the students to indicate their preferred method of assessing their tutorial problem solving skills, as a way to reduce the 80% loading of assessment in the end of semester exam. In this regard the structured multiple choice option returned the highest percentage of responses with 43% indicating that they would elect to have structured multiple choice assessment within WebCT and only 16% in favour of a simple multiple choice format. Finally, 32% indicated they would prefer to hand in a hard copy sheet for marking.

Figure 4

Figure 4: First year science students responses to aspects of
the structured online assessment module and tutorial sessions.

Staff interviews

Generally, preliminary staff interviews revealed that they thought WebCT was a valuable tool to be included in a course. Although, when asked to state what value they saw in implementing e-learning into a course, one responded that he did not see it as being essential to student learning, although it could be valuable when implemented well, while the other respondent said that it was an important aspect of today's learning environment and encouraged good IT skills. One consistent issue both staff members raised during the interviews was the time restraints they faced in developing the WebCT sites for their respective courses. In addition, both staff members expressed concerns that having material available and freely accessible could translate to students to not turning up to classes. However, across the board for all students surveyed, this is not likely to be the case, since students preferred the face to face contact in lectures to accessing lecture material online.

As mentioned above, staff also raised the concern of accessibility to WebCT, however, judging from the responses, this was not an issue (see discussion above). When asked where they would like to take e-learning given more time and resources, one staff member replied that they wanted to enhance the interactivity of their WebCT sites. Although he realised that there was no stepping around the large amount of time investment for staff because developing quizzes and providing good resources required academic input. The other staff member replied that they wanted to provide a more professional e-learning product to enhance contact teaching and enable more dynamic and exciting features to be implemented. These initial interviews illustrate that although staff recognised the importance and value that e-learning has in contemporary higher education, the time restraints encountered in setting up such learning environments can significantly reduce the quality of the final product. Wiegel (2002) has stated that many e-learning programs fell into the trap of being merely current educational models "repackaged" into digital formats. Our interviews with staff show that perhaps, this is not so much a result of a desire for meeting marketing goals but one of time constraints faced by staff, given the work load of today's faculty. Further interviews with a larger number of staff members would be useful in determining if similar sentiments are held by teaching staff across the science faculty.

Conclusions and further work

The results of this current study can give some guidance for teachers on the construction of a WebCT online study environment for students at different levels in their science degree. Briefly, we found that in setting up an online environment in WebCT, they will have to consider both class size and class level. We conclude that although science students at all levels preferred face to face lectures and tutorials to studying in an online environment, these same students found WebCT to be a valuable resource in aiding their studies, much the same as traditional study aids, such as library resource books. With regards to developing online assessment tools for large first year science classes, the initial testing of a structured multiple choice format proved amenable to the students. They expressed a preference for this format if there were to be some form of additional assessment in the unit through the semester. An online assessment would provide a more workable option than having to manually assess hard copy worked sheets from the students with class sizes in excess of 300. Further, if students were given the choice to have WebCT as a study aid in a course, they are likely to embrace it readily.

However, the extent to which students can make use of WebCT and other online environments will ultimately depend on how much time and energy staff are willing to expend on developing such an environment. Although the sample size for the staff interviews were small due to restraints in both time and resources, the concerns of staff members regarding the investment of time in setting these websites up are not surprising, given the similar responses reported in the literature (Gupta et al., 2005).

Acknowledgments

We would like to thank the following people for their contributing to the project in one way or another: Diana Jonas-Dwyer, Dr Allan Goody, Dr Andrew "ACC" Rate.

References

Browne, L., Mehra, S., Rattan, R. & Thomas, G. (2004). Comparing lecture and e-learning as pedagogies of new and experienced professionals in dentistry. British Dental Journal, 197, 95-97.

Gupta, B, White, D. A. & Walmsley, A. D. (2004). The attitudes of undergraduate students and staff to the use of electronic learning. British Dental Journal, 196, 487-492.

Keasar, T., Baruch, R. & Grobgeld-Dahan, E. (2005). An evaluation of web enhanced instruction in college level biology courses. Australasian Journal of Educational Technology, 21(4), 533-545. http://www.ascilite.org.au/ajet/ajet21/keasar.html

MacKay, S. (2005). E-learning myths dispelled. InTech, 52(1), 48.

MacKeracher, D. (1998). Development of young adults. University of New Brunswick Bulletin on University Teaching, 25, January. [verified 19 Jan 2006] http://www.lib.unb.ca/Texts/Teaching/bin/get.cgi?directory=JAN98/&filename=mackeracher.html

Wiegel, V. B. (2002). Deep learning for a digital age: Technology's untapped potential to enrich higher education. Jossey-Bass, San Francisco.

Appendix 1: Sample questionnaire given to undergraduate science students

Please answer the following questions by ticking the appropriate box and providing brief written feedback.

1. Do you use WebCT for the Unit? (If yes, please go to Q 3.; if no please go to Q 2.)

_ Yes _ No
2. If you answered no for the above, why don't you use WebCT?







Please go to Qn. 4

3. What parts of the unit's WebCT site do you mostly use?

lecture notes _ Yes _ No
discussion board _ Yes _ No
self-quizzes _ Yes _ No
additional reading/resources _ Yes _ No

4. How often do you access WebCT? (Please check one)

_ Never _ Daily _ A few times a week _ Weekly _ Monthly
5. Do you have access to WebCT outside of campus (e.g. via home internet connection)?
_ Yes _ No
6. Where do you most often access WebCT from?
_ Campus _ Home
7. Does a slow internet connection from home discourage you from accessing WebCT?
_ Yes _ No
8. I use WebCT to catch up on material from classes missed due to clashes, illness etc.
_ Strongly Disagree _ Disagree _ Agree _ Strongly Agree
9. I prefer to study using resources on WebCT (an online study environment) than more traditional methods like accessing library reference books.
_ Strongly Disagree _ Disagree _ Agree _ Strongly Agree
10. I find the current layout of the WebCT homepage easy to access/use.
_ Strongly Disagree _ Disagree _ Agree _ Strongly Agree
11. WebCT has enhanced my interest in the subject:
_ Strongly Disagree _ Disagree _ Agree _ Strongly Agree
12. Using WebCT has enhanced my understanding of the subject:
_ Strongly Disagree _ Disagree _ Agree _ Strongly Agree
13. Apart from WebCT, what other resources do you use to help your study in this Unit? (e.g. reference books; lecturers; lecture notes)







14. I prefer attending lectures and interacting (asking questions) with my lecturers face to face than using WebCT to get extra information on the subject.

_ Strongly Disagree _ Disagree _ Agree _ Strongly Agree
15. Overall, WebCT has been useful to me in aiding my study in the Unit:
_ Strongly Disagree _ Disagree _ Agree _ Strongly Agree
16. Is there a particular aspect of the WebCT site for this unit that you find needs improving? If so, do you have any suggestions to improve the current WebCT site for this unit?








Appendix 2: Sample questionnaire given to first year science students at the beginning of semester for feedback on the structured online assessment

Please answer the following questions by ticking the appropriate box(es) and providing brief written feedback.

1. Which of the following best describes your approach to the assigned tutorial work?

_ I won't do it until it is time for exam revision.
_ I attempted to have a go as a self-test.
_ I worked on the tutorial problems at the face-to-face tutorial sessions.
_ I worked on the tutorial problems in WebCT.

2. I preferred working through the questions on my own without using tutorial session or WebCT.

_ Strongly Disagree _ Disagree _ Agree _ Strongly Agree
3. Did you use WebCT to work on the structured multiple-choice tutorial?
_ Yes _ No (If you answered No, proceed to Qn.11)
With regards to the structured multiple-choice questions posted on WebCT:

4. I found it too easy to cheat my way through the questions by clicking the answers until I got the right one.

_ Strongly Disagree _ Disagree _ Agree _ Strongly Agree
5. I found the hints useful in helping me work my way through the problem to get the correct answer.
_ Strongly Disagree _ Disagree _ Agree _ Strongly Agree
6. How did you work your way through the problems on WebCT?
_ I worked on my own.
_ I worked through them with my classmates.
7. What did you think was good about the structured multiple-choice questions on WebCT?







8. What did you think was bad about the structured multiple-choice questions on WebCT?







With regards to the tutorial session:

9. I found it useful having the lecturer there to help me work through the problem sheet.

_ Strongly Disagree _ Disagree _ Agree _ Strongly Agree
10. How did you work your way through the problems?
_ I worked on my own.
_ I worked through them with my classmates.
11. I preferred using WebCT to attending the tutorial sessions.
_ Strongly Disagree _ Disagree _ Agree _ Strongly Agree
12. If assessment of the tutorial work were to be included as part of the course requirement, how would you like for it to be implemented?
_ I prefer using the structured multiple choice tutorial format in WebCT.
_ I prefer a simple multiple-choice question.
_ I prefer to hand in the worked problem sheet for marking.
13. Do you have any other comments on the questions using WebCT, the tutorial sessions or the tutorial assignments?







Appendix 3: Sample of online tutorial worksheet in WebCT

Appendix 3 screen picture

Authors: Yamin Ma is currently a postgraduate student with the School of Earth and Geographical Sciences, University of Western Australia.
Yamin Ma, School of Earth and Geographical Sciences MO87, The University of Western Australia, 35 Stirling Highway, Crawley Western Australia 6009. Email: may01@student.uwa.edu.au

Sandra M. Saunders is a lecturer in the School of Biomedical, Biomolecular and Chemical Sciences. She teaches and coordinates level 1 chemistry and level 3 environmental chemistry, with teaching contributions in level 3 green chemistry and environmental change. Her research interests encompass wide environmental domains and aims to determine anthropogenic impacts, to develop practical tools for environmental impact assessment.

Please cite as: Ma, Y. and Saunders, S. M. (2006). A case study of the effectiveness of WebCT as a student-learning tool and platform for structured assessment. In Experience of Learning. Proceedings of the 15th Annual Teaching Learning Forum, 1-2 February 2006. Perth: The University of Western Australia. http://lsn.curtin.edu.au/tlf/tlf2006/refereed/ma.html

Copyright 2006 Yamin Ma and Sandra Saunders. The authors assign to the TL Forum and not for profit educational institutions a non-exclusive licence to reproduce this article for personal use or for institutional teaching and learning purposes, in any format (including website mirrors), provided that the article is used and cited in accordance with the usual academic conventions.


[ Refereed papers ] [ Contents - All Presentations ] [ Home Page ]
This URL: http://lsn.curtin.edu.au/tlf/tlf2006/refereed/ma.html
Created 11 Jan 2006. Last revision: 20 Jan 2006.