Teaching and Learning Forum 98 [ Contents ]

How does one evaluate students' participation and interaction in an Internet-based unit?

Dorit Maor
Science and Mathematics Education Centre
Curtin University of Technology
This paper will introduce some dilemmas that arose during the implementation of a postgraduate Internet-based distance education unit, Computers in Education. The participants in this unit are secondary science and mathematics teachers from around Australia. The use of the Internet aims to help overcome intellectual isolation.

The main feature of this unit was the Activity Room in which the teachers, lecturer and invited visitors interacted asynchronously. These interactions focused on the major content areas of the unit which were examining current theories of teaching and learning with computers, and evaluating educational software. The use of the Web/Internet also provided the framework to focus on the learning process as well as on the content area. The challenge was to involve teachers in computer-networked communication in order for them to become familiar with the technology and to utilise it for their own communicative learning. The teaching approach involved social constructivist framework. Each week a different member of the group led the discussion by posing questions and initiating further discussions. The role of the discussion leader enabled the delegation of responsibilities to encourage participation of all the members. However, some of the participants were more active then others.

Three interesting dilemmas will be canvassed.

  1. How does one assess students participation and contribution in the activity room?
  2. Should the participation be compulsory and therefore be integral part of the assessment?
  3. How should the interaction be qualified?

Background

A postgraduate distance education unit called Computers in Education, was developed to enhance the understanding of science and mathematics teachers concerning the effective use of computers in their classrooms. Teachers in general, are reluctant to use the computer during their science and mathematics classes although there are many advantages associated with this practice, especially in the area of simulation. In order to incorporate educational software with teaching teachers should be able to choose educational software for their students, be able to utilise the tool themselves, and understand problems related to the use of computers for learning. To achieve these goals, I decided to engage the teachers themselves in learning experiences which required the use of computers. The availability of the Web as a tool provided the framework for the interactions and enabled us to utilise the technology, to focus both on the process of learning, as well as on the content area of evaluation of educational software and learning with computers. Salomon (1996), suggested that teachers need to experience a novel learning environment as learners so that they have some appreciation for challenges facing their students. Therefore, teachers were involved in computer-networked communication in order for them to become familiar with the technology and to utilise it for their own communicative learning. The unit was designed to promote interactions and social negotiations between the participants in the group, while I planned to act as a moderator.

Therefore, the unit was re-designed to match the dynamics of using the Web. All students had to have internet access. The home page was interactive and flexible enabling the on-line resources, Activity Room, additional readings and Guide to be modified weekly. The rigid component was the reading materials which for copyright reasons were mailed to the students.

This unit was supported by a Teaching Learning Group (TLG) grant which enabled me to plan the unit and to employ a research assistant to support its implementation on the Web. This was my first experience of "teaching on the Web", therefore, a technically skilled research assistant was greatly appreciated by myself and the participants. I was relying on my research assistant to resolve technical problems and consulted with him about design problems associated with the display of the unit on the Web. A web-site manager from the TLG managed the site which left me dependent on him for any changes that I wanted to make or would make in the future. This highlights one of the dilemmas of the new technologies. There is a tension between the empowerment of being able to add, remove and alter aspects of the site and the need for very high levels of technical skills to be able to develop interesting and sophisticated materials.

My aim in conducting this unit on the Web was to create a community of learners in which the common goal was learning with computers.

Objectives of the unit

The proposed objectives of the unit were to engage learners in activities which should enable them to:
  1. develop a sound knowledge of different methods of evaluating educational software;
  2. reflect critically on their own understanding of the evaluation process;
  3. conduct action research involving evaluation of educational software;
  4. use the Web to access information and resources; and
  5. participate in on line discussions in the Activity Room and communicate through the discussion list via email.
Based on the literature, it appears that most distance education programs use technology to deliver instruction emphasising the transmission of the content rather than collaboration (Duffy & Cunningham, 1996). In this unit the main goal was to involve teachers in using the internet for discussion, clarification and extension of ideas.

There were eight science and mathematics teachers from various places in Australia who participated in this unit. Most of them were high school teachers who were interested in enhancing their use of computers in their classroom as well as gaining credit for their postgraduate studies. I was enthusiastic about the opportunity of using Computer Mediated Communication to facilitate conversations with the students and expected that they would engage in discussions, collaboration and reflections.

Research component

Evaluation of the unit was conducted formatively during in the Activity Room discussion, and summatively via a questionnaire at the end of the semester. As part of the evaluation of the course, I decided to conduct an interpretive research study (Erickson, 1986) to investigate the opportunities that learning via the Web provided. The teachers were expected to conduct action research in their classrooms and to present it to the other learners in the group via the Web. This created opportunities to discuss problems in the classroom and provided additional on-line resources. I anticipated that one of the primary benefits of this unit to the participants would be the regular feedback from both peers and lecturer.

Outcomes

In analysing the data I examined students' interactions via the Activity Room. I was looking for unique experiences, or critical incidents, emphasised by the students during the unit. The evaluation indicated the strengths and limitations of using the Web through the perspectives of the participants and myself.

Limitations of using the web

The major problem of conducting this unit was to get everyone started and communicating on a regular basis. The communication at the start of the semester was very slow as students experienced many technical problems in connecting to the internet. The research assistant, and the TLG staff provided help. However, it took three weeks for all the students to be connected and start to communicate. Two students pulled out during this time, one of them feeling very frustrated because he thought that he missed a lot and would not be able to catch up with the program. Previous studies (Eastmond, 1995) suggested that this period of overcoming technical obstacles is sometimes the most crucial for the learner and could determine the success of the unit. Following are some examples of students experiences and difficulties at the beginning of the semester.

Students' expectations

Dan had problems starting the unit. He had to get to a computer laboratory to use the computer. He had little experience with computers and therefore expressed his concern:
...the beginning of the unit was quite frantic, and it took me time to settle in. This had the effect of compressing the main activities towards the end, resulting in a rushed job in the Action Research project. (Dan, November, 96).
Jan who lives in Newman, West Australia, had problems finding a network provider, she finally found one but had to pay STD rates. Her next obstacle was to repair the phone line connected to the modem, and then to have another version of the network software. "After nearly two weeks of rude thoughts about computers modems and external courses I was finally on-line" (Jan, 14.9.96).

Her personal objective for taking the unit:

I feel a little bit left out being 1200 Km away from Perth so I am hoping that doing this course through the net will lessen that feeling of isolation. (Jan, 14.9.96)
Students expressed their expectations at the beginning of the semester:
... like most of you, I am looking forward to the opportunities that this course will provide in integrating and using computer resources within the classroom. (Robert, 10.9. 96)

Advantages and opportunities in using the web

The Activity Room

The Computer in Education Web-site provided electronic copies of the Unit Guide, Unit Plan, Resources (which included the references for the copyrighted readings) and links to other Web sites of interest. The initial list of resources could be increased by students adding their own links to new Web sites they discovered.

Students met in the Activity Room and most of the discussions took place there. The list group was functional and was used for messages at the beginning of the semester. However, when the Activity Room became the central place for "meeting", the list group become redundant and was used only occasionally to remind people to go and visit the Activity Room.

In my view, the Activity Room served a more combined purpose than the list would, i.e. communicating and systematically filling the discussion inputs by individuals. However, we need to practice both for a balanced means of communicating. (Dan, 3.12.96)
The teaching approach that I was using in this unit involved a social constructivist framework (Tobin, 1990; Tobin, 1993). The teaching process involved learners and lecturer engaged in weekly discussions based on the particular readings for the week. Each week a different member of the group led the discussion by posing a question and encouraging any further discussions which emerged. By having a discussion leader, I wanted to delegate responsibilities and ensure full participation. However, depite this, I found it difficult to encourage all of them to take part consistently in the discussions.

To initiate the first conversation, I referred students to a web page which discussed the personal use of computers and told the group about my own personal experience. This acted as a trigger for other students to introduce themselves and talk about their own experiences. The participants who led the discussion for the week usually produced good summaries followed by questions to the other participants. However, the initial conversations did not flow and the interactivity was low and participants were sometimes slow to respond. This made me wonder how to stimulate a good discussion and involve all the participants. Some topics created more interest consequently more reflection and negotiation.

One of the students' comments in his final feedback was:

For me it has been a road to learning, not only about using new technologies, but also learning to attach value to the human touch behind these technologies. (student, 26.11.96)
A major advantage of this unit was that many of the features of the unit could changed and evolve over time. Web sites were added, new topics were discussed and this led to new and relevant issues being initiated by the students. The hub of the unit was the Activity Room. It was a place to meet, to interact, negotiate and clarify ideas and constituted an innovative use of the web.

For example, the following questions generated by one of the discussion leaders illustrates the level of conversations and the reflective process that took place in the Activity Room:

What reflections do you have on the dimensions? (part of the evaluation model?) How has your critical thinking about the software evaluation been extended? (2.11.96)
These questions created further discussions and responses such as Michael's on the 13.11.96:
Although half wishing that it had not happened so that I could continue in ignorant bliss about the effectiveness of the check list approach, I have been exposed to an enormous paradigmatic shift in understanding in relation to software evaluation. The possibilities are certainly more broad than I ever imagined, and the process much more involved than I thought.
Joan, for example, replied to the second question: [the critical thinking]
Extended - it has been elongated and broadened beyond belief. Coming into this unit I was very happy to take someone else's checklist about (as I know now) basically irrelevant criteria and use this as something on which to base my software purchases. I would want a much closer inspection now before I was prepared to part with my or my institution's money. (16.11.96)
At the end of the semester, Michael wrote:
Thanks for your input on this course, it has been a real eye opener for me in terms of effective evaluation of computer software. This is definitely relevant to a teacher in my position. (Michael 24.11.96)
A guest visitor was an excellent idea which always provoked a good discussion. The invited 'guests' in the Activity Room were people whose papers we read and with whom we had stimulating discussions. Talking with the experts and being able to direct questions to them was a valuable event.

Students view on the Activity Room

In reply to my informal evaluation of the Activity Room, student replies were positive and indicated that they welcomed the opportunities provided. Some of their replies:
Studying at a distance has its own unique problems and contact with the lecturer/tutor is very important. For those students with Email access this has become a very efficient method of receiving help quickly in answer to queries (Joan, 21.9.96)

I found the Activity Room great because of the ability to follow a thread of argument, and the reduced pressure of time because you could come back later and complete some missed work. (Robert 21.11.96)

In addition to the discussions in the Activity Room, the participants had to conduct action research and had the opportunity to present it as a web page. The research was conducted in the teachers' classes and then most participants submitted it as a written assignment. Only a few managed to present it on their web page for further discussion. I envisage that in the future most of the teachers enrolled in this unit will present their work on the Web and will get the technical support to do so.

Assessment of students participations and contributions

The main dilemma that I faced was how to assess student participation in the Activity Room. The idea of having a discussion leader each week increased the participation of an individual for a specific week, however, did not increase the participation in general. This led to other questions: How do you measure the contribution and quality of students' contributions? How to qualify the participations in the Activity Room? In order to quantify the participation of the individual in the Activity Room, I created a database which included all the interactions in the Activity Room. This database was sorted by: 1) participant, 2) discussion leader and 3) discussion topic. One of the columns in the database was the learner's involvement. This section had categories from "learner initiated redundant conversation" to "poses question and presented URL", "explicitly lead the discussion" or "responding to questions". This qualitative analysis of individual contributions to the communicative interactions provided me with an overview of the type of responses in which the individual learner engaged. It was possible, although difficult, to quantify the number of responses of each participant. However, students' self-reflection of the value of the communicative interactions was the most valuable one, and I found it extremely difficult to assess the extent to which each participant contributed to the dynamic of conversations and to the depth of the reflections.

In another attempt to resolve the assessment dilemma I decided to give the students' shared control of the assessment issue and asked their opinions about what it could be. My message to them was as follows:

All the active participants will pass this unit .... In addition, the second goal required your own self-assessment (ie. reflection on your learning) because you are the best one to know how far you have come since the beginning of the unit. I appreciate your self assessment....Please note that I am more interested in the quality of the unit and what you have gained from the unit rather than the more technical issues. (Dorit, 19.11.96)
Suprisingly, students were slow to respond. One student expressed concern as she claimed that this assessment policy was not announced in advance. Other students were happy with my suggestion and said:
I think the breakdown is fine and the type of assessment is good too. I think it's fair and reasonable and offers scope for our assignments as well. (Jacqui, 21.11.96)

An emerging issue: Establishing the boundaries of the unit

The task of facilitating this unit was very time consuming. I spent a few hours every week reading students' email, their comments and discussions in the Activity Room. Also I reflected a lot on their work, my feedback, what to do next and my own learning process. Students also claimed that: "it was more [work] than in other units".

One of the reasons for the excessive amount of work for a relatively small unit of 15 credits was the that the unit evolved and developed during the semester. Because the topic was "Computers in Education", and because it was taught on the web it was easy to frequently add new and relevant resources. For example, web sites on evaluation of computer software or learning with software: pedagogies and practice; as well as articles about constructivism and technology.

I was faced with the problem of how to keep realistic boundaries to the unit when I discovered a new website that I thought was of great value and relevance. Learners had opportunities to comment on problems and to evaluate the unit. One of the initial criticisms was that I was not responding quickly enough to the email. This was because I too was experiencing, for the first time, this mode of learning. The development process was very time consuming and much more demanding then a face-to-face class curriculum.

Final remarks

Anecdotal evidence from this semester showed that students, in general, were happy using the Web for this unit and benefited from the interactions in the Activity Room. Most of them used the Web for the first time, some had used it as Email only.

As a facilitator of this unit, I feel that I am on a steep learning curve which hopefully will improve. Feedback from students and my experience as a facilitator of the unit assisted me to restructure the unit which will enhance the quality of learning opportunities provided to future participants.

In summary, in implementing this unit on the Web we achieved the following:

  1. Familiarised teachers with wide range of possibilities of using Computer Mediated Communication (CMC) technology,
  2. Introduced teachers to educational computer issues,
  3. Provided a tool for the learners and the lecturer for interactions, and foremost,
  4. Created a community of learners who engaged in collaborative discussions through Computer Mediated Communication.
Although I have been through a great learning experience, I still struggle with some dilemmas. Firstly, how to encourage active participation of the learners; secondly, how to assess students' participation and contributions; thirdly how to become more technologically independent and finally how to set appropriate boundaries to the unit.

References

Duffy, T. M., & Cunningham, D. J. (1996). Constructivism: Implications for the design and delivery of instruction. In D. H. Jonassen (Ed.), Handbook of research for educational communications and technology. New York: Simon & Schuster Macmillan.

Eastmond, D. V. (1995). Alone but together. Cresskill, New Jersey: Hampton Press.

Erickson, F. (1986). Qualitative research on teaching. In M. C. Wittrock (Ed.), Handbook of research on teaching, 3rd ed, 119-160. New York: Macmillan.

McDougall, A. and Squires, D. (1995). A critical examination of the checklist approach in software selection. Journal of Educational Computing Research, 12(3).

Salomon, G. (1996). Technology's promises and dangers in a psychological context: Implications for teaching and teacher education. Paper presented at the The Second International Conference Teacher Education: Stability, Evolution and Revolution. Wingate Institute, Israel.

Tobin, K. G. (1990). Social constructivist perspectives on the reform of science education. Australian Science Teachers Journal, 36(4), 29-35.

Tobin, K. G. (Ed) (1993). The practice of constructivism in science education. Washington, DC: AAAS Publications.

Please cite as: Maor, D. (1998). How does one evaluate students' participation and interaction in an Internet-based unit? In Black, B. and Stanley, N. (Eds), Teaching and Learning in Changing Times, 176-182. Proceedings of the 7th Annual Teaching Learning Forum, The University of Western Australia, February 1998. Perth: UWA.http://lsn.curtin.edu.au/tlf/tlf1998/maor.html


[ TL Forum 1998 Proceedings Contents ] [ TL Forums Index ]
HTML: Roger Atkinson, Teaching and Learning Centre, Murdoch University [rjatkinson@bigpond.com]
This URL: http://lsn.curtin.edu.au/tlf/tlf1998/maor.html
Last revision: 12 Mar 2002. The University of Western Australia
Previous URL 31 Dec 1997 to 12 Mar 2002 http://cleo.murdoch.edu.au/asu/pubs/tlf/tlf98/maor.html