Teaching and Learning Forum 2000 [ Proceedings Contents ]

Improving the effectiveness of tools for Internet based education

Martin Dougiamas
Centre for Educational Advancement
Curtin University of Technology
    This report presents results of a study of a class of high-school teachers learning about the Internet in an Internet-based course called "Internet Overview".

    The course was implemented using a prototype of a new web course development tool called Moodle that uses constructionist referents to model engagement of the participants with course content and each other. The intention of the research was to identify areas to improve Moodle as a tool to build and run effective internet-based courses.

    The action research focused on the factors affecting the level and quality of participation in the course, as well as the quality of reflective thinking, interactivity and cognitive support. An online survey of the students, combined with voluminous texts and records created by all participants of the course, led to interviews with students selected for their interesting responses. These texts provided the data with which I attempted to understand how various factors affected the experiences of learning and teaching in this internet-based course.

    The results suggest that Moodle as it stands is relatively successful as a tool to produce structured content with workbook-like responses. Two areas needing the most improvement are internet knowledge and student interaction. This encourages me to continue development on two fronts: firstly the Internet Overview course as a tool for students to learn about the Internet; and secondly, functions within Moodle to encourage and manage educational discourse among a class of students within its content-based framework.

Teaching and Learning Forum 2000 Home Page

Table of contents


1. Introduction

1.1 Background

In my work as an Internet Consultant at Curtin University of Technology, I have spent five years developing courses and helping academics develop their own online courses. As a student I've also participated in some. I've noticed a tendency for newcomers to teaching online to use frameworks limited by metaphors such as textbooks or lectures. As a result the full potential of using the Internet to cross-communicate and develop knowledge does not seem to be reached, and general satisfaction with the results appears anecdotally low. This is problematic to me, as the oft-cited advantages of the Internet (decreased dependence on time and space, and cost-effectiveness) should help make education available to more people than it is today. This has encouraged me to look for ways to improve online courses in terms of the satisfaction of both teachers and students, by improving the interaction between technology and people. There are two main approaches to this: helping people learn how to use the technology; and improving the technology to make it more effective as a tool. Naturally these depend on each other and need to be developed in tandem.

For many teachers, creating an effective online course site using general-purpose tools such as web page editors, discussion groups, mailing lists and commercial office software can be a complex and expensive task. Becoming proficient in the use of these tools is beyond most teachers, who have limited time and resources. As a result, most online courses are very simple, consisting basically of static 'lecture notes' (text and graphics) and, less often, a generic 'threaded discussion area'. Even when funding is obtained to create a more ambitious site using hired labour, the cost of maintenance may still be too high for the teacher to support, and these web sites easily stagnate as the course, and the teacher, changes from year to year. This situation has occurred at my university even when using commercially-supported online course-building environments such as WebCT [www.webct.com]. The difficulty of creating and maintaining web site resources often makes it appear to be the main part of creating an online course. Improvements to the course are very often seen in terms of rewriting the content, or adding more media to the web site, such as audio and video. A major issue of concern in these cases is often that of protecting intellectual property, so that courses aren't "stolen" and downloaded by students who haven't paid.

Pedagogically, this approach perpetuates a traditional metaphor of teaching as delivering packages of knowledge, rather than as a purposeful collaborative activity promoting active construction of meaning (Wilson, 1995). Although recognition of the importance of discussion may mean a bulletin board is added to the course, focussing on the quality and type of discussion in there is often not a priority. When it is, it can be limited by the software used to implement the discussions. Resource-heavy web courses are best suited to capable self-learners, who are able to contact a largely passive teacher when they need to, or seek out alternative Internet resources or textbooks and teach themselves. They are not well-suited to students who do not respond well to this style of learning such as those harbouring deep misconceptions, or who skim the material lightly. The teacher may not find out that problems exist until they see the first (and sometimes only) large assessable essay or test - by this time it is too late to help students. This style of teaching does not cater for diverse styles of learning.

Newer ideas in learning theory, especially constructivism, has been successfully applied as a referent to improve the efficacy of online courses (LeFoe, 1998). Constructivism in its most trivial form states that "Knowledge is actively constructed by the learner, not passively received from the environment." (von Glasersfeld, 1990). Constructionism asserts that constructivism occurs especially well when the learner is engaged in constructing something for others to see:

"Constructionism shares constructivism's connotation of learning as 'building knowledge structures' irrespective of the circumstances of the learning. It then adds that this happens especially felicitously in a context where the learner is consciously engaged in constructing a public entity, whether it's a sandcastle on the beach or a theory of the universe... If one eschews pipeline models of transmitting knowledge in talking among ourselves as well as in theorizing about classrooms, then one must expect that I will not be able to tell you about my idea of constructionism. Doing so is bound to trivialize it. Instead, I must confine myself to engage you in experiences (including verbal ones) liable to encourage your own personal construction of something in some sense like it. Only in this way will there be something rich enough in your mind to be worth talking about." (Papert, 1990)

Social constructionism is a closely related set of ideas that focus on the individual development of meaning through communicative action and the construction and sharing of social artifacts, including texts (Gergen, 1995). When applied to the study of students who are interacting within a mostly text-based environment, these referents keep the focus on the importance and quality of dialogue between participants as a tool to construct knowledge.

A dialogue is an exchange affecting two parties, either directly via a semiotic medium such as language and other signs, or indirectly via tools such as computer interfaces , drawn from the socio-cultural context (Duffy and Cunningham, 1996, cited in LeFoe, 1998). Computer software, such as web sites, can be both a tool and a language - as a medium for dialogue it can be very powerful. Learning dialogue in online learning situations can be divided into four types:

Many teacher/researchers using the Internet for education are aware in some way of these issues, but can be restrained by inadequate tools to participate in the many dialogues possible within a single class in a manageable way. Taylor et al (1999), for example, describe the "astonishing intensity of discussion room activity" that resulted from making discourse an assessable activity, forcing the researchers to delay their research analysis until after the completion of their teaching. They conclude that their challenge lies in redesigning their web-based pedagogy to engage students in better-focussed and more manageable online discourse. In another sense, the metaphors of current tools contain their own pedagogies, and can constrain how a teacher thinks of an online environment and what is possible. The common metaphor of a "discussion room" or "bulletin board" masks some of the differences that such an environment has when compared with discussions in a physical room. Current tools, being among the first generation of Internet tools for computer-mediated communication are based on real-world metaphors, but opportunity exists to create new tools unconstrained by these conventions.

The tool of the interactive web site itself is only part of an "online class". From each student's perspective, the factors that make up the total learning environment of such an online class can be divided into nine parts:

Each of these may vary dramatically in quality, and they are all affected by the availability of tools and each other. In particular, though, a web site can affect all but number 2 in this list.

1.2 Objectives

The broad objective of this research is to contribute towards improving the quality of learning over the Internet so that it can be widely accepted as a valid or perhaps even preferred alternative to traditional learning environments. If this can be achieved, then the inherent flexibility and cost-effectiveness of the medium could allow education to be available to those disadvantaged by distance, disability, culture, lack of wealth or numerous other restrictions.

More specifically, the objective of this research was to examine the educational performance of an existing internet-based course and the tools used to make it, in order to identify areas of relative success and failure. This knowledge is intended to be applied to the next iteration of both the course and the course-building tools. Through action research, answers were discovered for the following research questions:

In this course, which of the nine aspects of this learning environment compared the most favourably to other learning experiences that the students and teacher have been involved in? In this course, which of the nine aspects of this learning environment needed the most improvement? Why do they need improving, and how can they be improved?

1.3 Overview of this report

Section 1 - provides background to this study of internet-based learning and my motivations as a researcher.

Section 2 - explains the design and development of the internet-based course used for this study.

Section 3 - discusses the theoretical and practical methodology of my actions as a teacher-researcher for the duration of the course.

Section 4 - discusses the processes and results of of my summative evaluation of events after the course.

Section 5 - presents conclusions and directions for further work.

2. Teaching before the course started

2.1 Developing the course

My intention with the Internet Overview course was to introduce the Internet to beginners, with the major goal of improving their "internet literacy". I started with the assumptions that participating students would be adults who know enough to get to the course web site, and have basic skills with moving a mouse and manipulating windows. From this, the course helps students to: read about and apply basic concepts and terms; explore current issues such as censorship and ownership; develop skills for using and searching the web; sample communications of all kinds; and learn about the basics of publishing their own web pages.

I started developing the web site in 1995, as a tool to use in conjunction with my regular 2-hour "Internet Overview" seminars. The web pages were designed to be projected behind me for demonstration and illustration, so that I could navigate through them while speaking. The first iterations of the web site were a simple set of heirarchically-organised descriptions with diagrams. Over the following years, the site was refined further in response to the types of questions I received in the seminars, by reorganising, adding and rewriting explanations. Each page attempted to anticipate the questions I was being asked in my seminars. However, the web site on its own was not as successful as it was when used with the seminars.

In 1998, I started exploring educational theory to find directions for improvement. Constructivism, and particularly constructionism, was immediately attractive to me as a tool to help understand learning and the nature of knowledge (Dougiamas, 1998) as were theories of how reading and writing could be used in learning (Dougiamas, 1999). This is because the Internet as a medium is well-suited to interactive constructive activites, especially using text and signs.

It was from this point that I approached this study, and the teaching it involved, with a new set of referents promoting interactivity between teacher, learner and course content. Technically, a dynamic web site that implements customised changeable pages can be a lot more difficult to create than a traditional static web site - it requires ways to store and recover texts from a number of people in a database, as well as keep track of access, ownership and quality. This can be an extremely time consuming process, requiring skills in programming and system design. This is one reason why such sites are not yet common. Existing tools such as WebCT attempt to simplify this process, but, being commercial software embodying particular pedagogies, can be restrictive in the types of dynamic web sites that are possible, making it difficult to experiment with new learning theories. In addition, as Salomon et al (1991) describe, a flexible tool can evolve in partnership with the people using it, each affecting each other.

As a result, I was inspired to create a new web-based tool I could use to implement, modify and experiment with an interactive internet-based course as easily and as flexibly as possible. I called this tool Moodle (Modular Object-Oriented Dynamic Learning Environment) and set up a server called moodle.com on the internet to provide an easy-to-remember location for it. As a researcher using constructionism as a referent, I needed to develop Moodle to a usable state, then use Moodle itself to develop Internet Overview from a teacher's perspective. Through interaction with students and the course, I intended to develop new insights as a researcher and perhaps the nature of constructionism. See Figure 1.

relationships of teaching and research

Figure 1: Diagram of affective relationships between my roles as teacher
and researcher, course software, the course and students.

2.2 Moodle

My starting point for Moodle was the old Internet Overview. Through iterative redevelopment, Internet Overview had already evolved to the point of being a resource consisting a large number of short pages (concepts) organised hierarchically (as topics and sub-topics) and linked together. To define Moodle I examined Internet Overview from the dual perspective of a student and a teacher, and started with the features that already seemed to work well. Finding an optimal design was not an issue, as this was just a first prototype, but I tried to concentrate on a limited number of important features.

2.2.1 Student perspective

Drawing from experience and literature, I identified the three most important features of Moodle from the student point of view as follows:

2.2.1.1 Navigation

To facilitate the varied learning styles of students, the structure of concepts within the course must be easy to see and navigate. This includes the strategies listed by Oliver et al (1996): placement cues that indicate the location of the learner within the material; heirarchies and indices to provide access to pages within the system together with an overall structure that is reinforced through use; and semantic nets, where connections and associations between related information are recognised and made specific using links within the content. From any page within the content, there are links to the next concept, the previous concept, the current topic and the overview. The hierarchical structure of these concept pages form a concept map (Novak, 1996). Links to related information outside of the course should also be included as much as possible.

2.2.1.2 Page design

Web pages in Moodle are restricted in length, with good spacing and clear fonts. For speed of download, pages have a simple layout and a minimum of graphics. Pages work predictably and consistently throughout a course to help students focus more on the content and less on how it is presented.

2.2.1.3 Interactivity

Each concept can require a response from the learner - an activity or exercise that involves the learner in dialogue to promote construction of the knowledge related to the concept (Jonassen, 1994). These responses are stored in the course database and can be recalled later for a variety of uses. After some thought I isolated the following types of responses as suitable for the first prototype, because they seemed useful and varied:

1. Writing an answer to a question

The student writes directly onto the web page, into a form text field. For the student, this text will stay there for the duration of the course, but will not be editable again. Each student will see their own text embedded on the page, and not that of other students. This activity is similar to a test, or a survey and can be used to capture an answer or an opinion. The exact activity in each case is directed by the teacher. It might require a summary of a text, restricted to a limited number of words, which promotes development of idea relation and revision (Glynn and Muth, 1994). It might require an answer to an open-ended question such as "From this, I learned..." (LeGere, 1991). In this case, the student is required to confront their own learning and organise their thoughts (Johnson 1983, cited in LeGere, 1991). The text remains as part of that page for that student, like an annotation, and can be useful to the student when reading it later to compare the change of their own understanding over time (Borasi and Rose, 1989). To help this, a function is provided for a student to extract all such entries in the course and display them on a single page, providing a record of thoughts throughout the course. These texts remain private between the individual and the teacher, to encourage educative authenticity (Glynn and Muth, 1994) that might not be present in comments that are visible to peers.

Teachers, when looking at the same page, don't see the text field. Instead, they see a button that displays the texts from all students, on one page. In this way they can easily read the responses from the whole class and by reflecting on them, learn of misunderstandings, problems, or trends in the class (Tobin, 1993). If required, they can be assessed, by checking off boxes next to each entry. By clicking on a name in the list, the teacher can also view all entries by that student from the whole course, in order, to reflect on that student's progression through the course.

2. Writing a reflective response into a journal

This is similar to the previous activity, except the text remains editable throughout the duration of the course. When prompted by the content, the student has a space on the page to reflectively write about their reaction to the content. Doing so can help them to form their understanding of the topic (LeGere, 1991), and to give feedback to the teacher. Later, the student may update the text to reflect changing opinions.

As before, the teacher can view all responses to this page on one page to gain a cross-section of the class.

3. Answering multiple choice questions

In this case, the student answers a multiple choice question by clicking a button. If the teacher has specified a "right" answer, then this functions as a self-test, and the system provides immediate feedback (right or wrong) - otherwise it simply stores the response.

For the teacher, a bar graph is automatically produced, listing names of students next to each answer, providing a cross-sectional view of the class in relation to the current concept.

4. Constructing and discussing an opinion

In response to a question within the content, the student is asked to construct a statement of their opinion on a topic, to be read by their peers. The text is sent to a focus bulletin board (discussion area) where it forms the basis of threaded discussion on that topic among the class. Only at that time can the student read other posts - they must post before reading. In each thread, students may alternate in the roles of actively teaching and learning with their peers, promoting active engagement with the concepts (Glynn and Muth, 1994). Everything is recorded and labelled, and it is possible for participation to be assessed either informally in a glance, or formally by assigning marks to each posting. The teacher can provide instructions or participate to help guide the nature of the discourse (Moore, 1994, p291). For example, they may promote certain values such as students being oriented towards others in an empathic and mutually supportive manner which invites disclosure of their experiential worlds and encourages their active interest in others' lived experiences.

When a student knows they are writing for a peer audience in an authentic task, responsibility may be felt by each student to write their initial statement as well as they can (Glynn and Muth, 1994). This may involve some redrafting (Connor et al, 1994), some research (Glynn and Muth, 1994) and a consideration of the audience (Glynn and Muth, 1994). The discussion can be used to stimulate a number of different styles of debate or information sharing which can facilitate learning. For example, discussions may centre on comparing different opinions of texts to help the participants refine their own opinions (Glynn and Muth, 1994). Participants also have an opportunity to "test" statements or frame questions. This active social engagement with a topic encourages more effective learning, as the discussion texts are co-constructed (Gergen, 1995).

5. Other options can be added

As other types of response arise, they can easily be added to the system in a modular way, allowing the software to adapt to the needs of the courses it implements and the development of the teacher. In this way the construction of the system itself is revisable and promotes constructivist ideals (Tobin, 1993).

On the surface this overall model may appear similar to the traditional model of textbooks with exercises or worksheets at the end of each chapter, however there are several major differences. Firstly, the granularity of content and responses is very fine, so that each concept/response pair can be highly focussed. Secondly, the types of activities is broader, allowing different types of interaction with other participants in the course than is possible in a distance education setting without the Internet. Thirdly, and most importantly, the individual texts and signs are stored in a database, allowing the teacher instant access to them in a variety of ways that promote quicker analysis and understanding of the students in that class. This in turn could help the teacher maintain a higher quality of dialogue with the students throughout the course.

2.2.2 Teachers Role

One of the most important goals of Moodle is to minimise the work a teacher needs to do to free up time for teaching and reflecting on their teaching. From my experience in teaching as well as literature, I identified three desirable features of Moodle from the teacher's point of view, listed below.

2.2.2.1 Teacher learns more about the content and can modify it easily

Navigating the course and adding or changing texts throughout the course is done directly and easily, using buttons embedded in the web pages. The content is typed in a simple built-in word processor. As the teacher works within the course environment, small changes are very easy to make on the spot. This ease, when combined with constant interaction with the class through writing, provides a dynamic set of written evidence upon which the teacher can reflect and make sense of the curriculum, then adjust or reconstruct the course as necessary. Tobin (1993) talks about this as an important part of a constructivist perspective:

What emerges as having important implications for teacher education is the range of objects for reflection. [...] Reflecting in this manner enables teachers to modify their visions of what the curriculum would be like and compare what is happening in their classes to the vision of what they would like to happen. (Tobin, 1993, p 225)

There are four main ways Moodle helps teachers monitor and modify content:

1. Page content editing

The content of each page consists of page (concept) consists of standard HTML text, which allows graphics, text styles, links like any other web page. This HTML can be edited directly on the web itself using the Edit button on every page, much like a word processor. This capability is restricted to the teacher (students cannot modify this primary content).

2. Response design

For each page, the teacher can choose to set one of the four types of responses listed in 2.2.1.3, by selecting from a menu. This automatically sets up the system to collect, tag and store student responses. For responses such as multiple choice, a text box allows all the options to be specified and the right one to be marked.

3. Automatic structure and navigation

The creation of new pages within the hierarchy is achieved using two small buttons next to every page represented in the hierarchical index listing. The first button creates a new sibling page to the one currently selected. The second opens a new sub-branch (topic). Once created, each page in the index listing can be moved up or down in the sequence, as well as up or down in the hierarchy.

Moodle keeps track of the structure, and when displaying full pages, is responsible for creating internal navigational links on the fly to relevant pages such as next, previous, and current topic. This frees up the teacher to concentrate more on the content.

4. Collation of responses

The response feedback from students can be collated in a number of ways. Firstly, each concept page has a button that displays the whole class response to that page. For journal entries, this will show a long page with all the bits of text that have been entered in reponse to that page, which for multiple choice, a bar graph is displayed showing the distribution (and names) of students across the choices. The discussion appears as it does for students, with the addition of a table showing numbers of posts per person.

Another form of collation is by student. For each student, a collation of all the entries made throughout the course can be generated, as a record of their activity in the course. This is available to each student, as well as the teacher. The display can be sorted by time, or by the order of the concepts in the course material.

Finally, a display can be generated of all students currently studying in the course, along with the time spent using it, number of pages read, and number of responses made.

2.2.2.2 Teacher learns more about each student

Through the use of the capabilities listed in the last section, a teacher can actively sift through the feedback and learn more about the personality of each student, particularly when a course encourages open and honest discourse. As well as deepening the teacher-student relationship which will affect later communications, this process can identify students who may have problems on social, academic or technical levels. The teacher may choose to contact these students directly in order to try to help.

2.2.2.3 Teacher learns more about the class

By examining the whole-class response, as well as combining knowledge about individual students, the reflective teacher may identify evidence of misconceptions, or patterns in the response. These may prompt the teacher to alter the content, start a new discussion, or search for other ways to engage the students in their learning of that part of the course.

2.2.3 Construction of Moodle

The building of the first prototype of Moodle took me approximately one month of full-time work to complete, using free open-source software tools. These included: Linux (www.linux.com), for the operating system; Zope (www.zope.org), as the application server; Python (www.python.org), as a lower-level programming environment; Apache (www.apache.org), as the web server gateway; and a number of Unix shell scripts to control the system.

In order to finish the prototype on time, I had to simplify down to the bare essentials the discussion forum in the prototype used in the research project, removing most of the management interfaces.

2.3 Internet Overview

Once Moodle had been constructed, I started work on using it to build the course content, basing it heavily on my previous version. Immediately it was apparent that it would be much easier to create this version of the course than the last had been. Editing pages in-place was very quick, so that each page passed through more versions as I moved around the course, reorganising concepts and refining the language. It was a very liberating feeling to be using my own tools to do this and I can say it was a thoroughly enjoyable experience to be able to concentrate more closely on the content, and less on technical details. Occasionally I ran into difficulties adding content, but I was always able to resolve it by modifying Moodle in some way.

By the end of my editing, it consisted of 85 "concept" pages, organised as 4 main topics with 16 subtopics. For each page I tried write as clearly as possible, avoiding ambiguous grammar and jargon, and using simple language with a friendly tone. Most pages contain a diagram, screenshot or at least a small cartoon to either illustrate the concept or simply make the page more memorable. Despite this, attention was paid to keeping overall page sizes small.

The four main topics build on each other in a logical progression. The introduction starts with a broad view of the Internet, examines current issues and explains basic terms such as 'server' and 'email'. The next major topic covers 'web surfing', or how to use the web to find things. The third section covers all of the many possibilities for communicating with other people on the Internet. The last major section introduces basic web publishing concepts and techniques.

About half of the pages required some sort of response, including three discussion forums for discussing the more contentious issues of censorship, ownership and privacy. The process of designing the responses for each concept was particularly challenging, forcing me to focus on the learning outcomes rather more closely than I had before, a highly reflective exercise. In addition, I was also anticipating some of the research outcomes, and trying to plan activities that would yield interesting information about the students. Due to a lack of time before the course started, some short cuts had to be made: there was a lot less discussion than I had originally intended, and I was not able to design a response for every page.

Eventually, the course reached a useable state, and I was ready to study how it was used in a real educational setting.

3. Methodology

3.1 The Teacher-Researcher

This study was guided by the metaphor of 'research as bricolage' which allows the interpretative inquirer to employ a range of research methods to pursue issues as they unfold (Denzin and Lincoln, 1994). My main method was to inhabit the dual roles of a teacher and an action researcher reflecting critically on my own practice as a teacher, both during and after the course. There was also some statistical analysis and case study research.

During a two-month period, students were guided to interact textually with the course content, each other, and the teacher in a variety of types of discourse. From this, a large amount of textual data was collected. These field texts included:

After this time the collected data was analysed to determine trends and anomalous behaviour. From this analysis, three students were selected as interesting case studies. Holt et al (1996) discuss the way in which case studies relate to research, and they quote Kemmis:

Despite its ambiguities, its imperfections and its simplifying interpretations, a case study can create conceptual stabilities which are platforms for understanding and for action. The authority of case-study work, like the authority for all science, does not derive from theoretical or logical elegance. It derives from the purchase it gives us on the real world of action and experience (Kemmis, 1980, p 131).

A short narrative of each student was developed for each case study, based on synthesising collected texts. For credibility (Guba and Lincoln, 1989) each narrative was checked with the student via email, which led to further clarification and rewriting.

Throughout these processes of interpretive action research, an awareness of the five authenticity criteria of Guba and Lincoln (1989) was maintained, in order to help assess the process and any results that were obtained. These criteria are:

Fairness: the extent that each stakeholder's constructions and values are honoured within the study.
Ontological authenticity: the extent that the researcher's own knowledge is improved by the ongoing process of research.
Educative authenticity: the extent that an individual's understanding of the constructions of others is enhanced.
Catalytic authenticity: the extent to which action is stimulated and facilitated by the evaluation processes.
Tactical authenticity: the extent to which stakeholders are empowered to act on new knowledge.

3.2 Ethical Issues

It is important that educational research be guided by ethical principles, as it has potential to affect the lives of participants in adverse ways. Cohen and Manion (1994) provide useful guidelines that I used to try and shape my role as teacher/researcher in the context of this study.

Most important is the principle of informed consent, which promotes an implicit contractual relationship between the researcher and the researched. An information page and consent form was part of the course web site, and included a summary of the research and how it might affect them, a statement of ethical guarantees, a link to this paper for more detail, and a small form to signal consent. The guarantees are of confidentiality, optional anonymity, and constant opportunities to verify or keep in touch with my findings as the research proceeds. See Appendix A.

In the case of any ethical decisions that needed to be made during the course of the research, I tried to use Strike's two guiding principles of benefit maximisation and equal respect (cited in Cohen and Manion, 1994). Benefit maximisation holds that the best decision is that which results in the greatest benefit for the most people. Equal respect demands that we respect the equal worth of all people, requiring us to treat people as ends rather than means, and to regard them as free, rational, and entitled to the same basic rights as anyone. As well as the participants of the class, this applies equally to any other stakeholders involved in the research.

3.3 An account of events

The Internet Overview course was conducted as a module within a larger course called Learning Technologies, Technology for Teachers (HREF A) . This larger course was for the professional development of around 100 high school teachers interested in learning how to apply technology to their teaching and was funded as a project by the Education Department of Western Australia (EDWA). EDWA purchased new laptops for each of the high-school teachers involved in the scheme, and assisted their connections to the Internet if they weren't already connected. One reason they were selected for this study was the fact that these students were also teachers with an interest in using the Internet as part of their own teaching. This suggested that they would be a particularly responsive and reflective group to study as they used Internet Overview.

Another major feature of this class was the extreme physical isolation of the students. Their locations were diverse, including such places as Christmas Island, Wingellina, Port Hedland, Queensland, Carnarvon and Perth. Physical meetings were unlikely, forcing the class to interact solely through the Internet, via the main Learning Technologies web site as well as the Internet Overview web site. In addition, the remote nature of many of these locations meant that their connections were via satellite or poor quality phone lines, and thus relatively slow and/or expensive.

Students entered Internet Overview via a link within the Learning Technologies web site. The first page required them to create their own username and password as well as provide some contact details (name and email address). After that they had access to the full Internet Overview site, and their activity was tracked and managed by their username. Once inside the course, one of the first links there was to a page that explains this research and allowed each student to give their consent to be involved. Of the total 83 students that used the course during the study, 54 gave their consent to be studied, 6 refused and 23 students didn't respond. Most of the students visited Internet Overview between June and September of 1999, with most activity around August. Of the consenting students, each spent an average of about two hours using Internet Overview, viewing an average of about 200 dynamic web pages. Use tended to concentrate around the mornings and evenings, but rarely was there more than one person in the course at a time.

The Moodle server on which the Internet Overview course ran was on my desk where I work during the day. In order to stay closer to what was happening in the Moodle site, I set up a monitoring program to watch the server activity logs and trigger sounds and displays according to events in progress. The steady stream of sounds in the background kept me informed of people who were, reading pages, using the chat room, or watching my web camera. I could monitor the sounds for most of the day, and turn to the displays for more information when required. The displays gave me more detailed information such as the name of the student and which pages they were currently using, as well where they were coming from and what sort of computer they were using. This helped me "be there" if they tuned in to watch my web camera, or be ready to talk with them if they entered the chat room.

Despite this, my own direct involvement with students was fairly minimal, compared to a face-to-face course. During the course I had only four chat sessions and about ten email exchanges, mostly helping students with minor technical problems and directing them to other resources that might be helpful.

Indirectly, I was involved in the course by reading the collated responses to various concepts, looking for trends and students having problems. In the main this was useful, and led to some email exchanges with students as well as changes to the content, but because the students weren't very synchronised in study time (being spread over three or four months) it was difficult to identify changes in the student population over time. For example, when I looked at a collated response in July, and then later at the same page in September, there was an increase in number of responses, but no way to identify recent additions, or to rank the responses according to any criteria. As I had updated the content in August (as a response to a problem found by looking at the feedback) it was impossible to tell which version of the content each entry had responded to. I could clearly see the need for an improved management interface to solve such problems by flexibly sorting, categorising and marking student responses. My experience in using the tool from a teaching perspective was directly contributing to some new insights into its operation from a research perspective.

At the end of September an accident with the server rendered the entire database unusable and the course was "off-air". It took me nearly two weeks to recover the data successfully and continue. Fortunately, as most students had already finished, this did not impact the operation of the course very much.

Towards the end of the course I developed an online survey to collect information about student perceptions of the learning experience of using Internet Overview. The survey was largely based on the Constructivist On-Line Learning Environment Survey - Preferred (COLLES-P) survey by Maor and Taylor (1999). The COLLES-P survey asks the students to rate 30 statements about practices which could take place within an online unit using two scales - what they actually experienced and what they would prefer. The COLLES-P statements are grouped into six sections: relevance of learning, reflective thinking, interactivity, cognitive support, affective support, and interpretation of meaning. For my survey I removed the last two sections, as they dealt with issues less relevant to this study, and added two new sections: one that asked about the nine factors of the learning environment listed in Section 1.1; and an open-ended field for comments. The COLLES-P sections support my factors section by probing for more detailed, but related information. (See the form in Appendix B).

I then wrote a script to mail each of the 54 students who agreed to participate in the research with a customised invitation to fill out the survey. Of these, 20 students returned to the course and successfully filled out the survey. While there, some of the students took the opportunity to revisit much of the course to refresh their memories and check on the progress in the discussion areas. The data was collected and collated in the Moodle database.

After two weeks, when the submissions had stopped, I transferred the data to a spreadsheet and analysed it (using a combination of simple statistics and visual inspection of colour-coded data), looking for trends as well as anomalous responses. It was particularly useful to look for large differences between actual and preferred scores, as they indicated what students felt were the greatest shortcomings in their experienced.

From these results, I chose four of the most interesting cases to develop into case studies. Two students were chosen for their similarity to the general trends, and two were chosen because their answers ran contrary to the trends in some way. I interviewed these students via email, sending them a narrative of my impressions gained from studying the information I had so far, and asking for feedback, clarifications and corrections. From this emerged several email conversations that allowed me to get a better picture of their personal motivations as well as confirm some of the broader conclusions I had reached.

The results of this summative analysis are described in the next section.

4. Summative analysis

This section analyses the reponses received from students during the course. All names have been changed to protect anonymity.

4.1 Case Studies

Here are short case studies of four of the participants. Barbara and Florence were chosen for their closeness to the majority of opinions in the survey, while Mary and Kim were chosen for somewhat anomalous statements. These case studies were developed after analysing all twenty survey respondents, but they are presented here first as they will help to frame the analysis in the next section.

Barbara

Barbara liked to study in the afternoon. In August, she checked for the first time on a Sunday afternoon, and spent an hour and a half browsing the course. The next day she connected again and filled in more of the blanks. Then she had a holiday and some time off, and it wasn't until October later that she came back and went through it all again. To her pleasure, she found it a lot easier to understand the second time around. Overall she was happy with the experience, but as well as more interactivity with other students, she would have liked a more direct connection to her professional practice.

Florence

Florence was quite experienced with computers, and considered herself a fairly critical user. She skimmed through the course quite quickly but still found it interesting, as she could see it being applied in her high school to help other staff learn about the Internet. The "issue" forums were most interesting, especially the post-before-read idea, and she would have liked to see more use of discussion in the course, as well as a little more participation from the teacher. One major problem she had with the course was that the forums were buried too deep in the course structure which made them hard to find, another was that they were cumbersome to use.

Kim

Kim had been using the Internet for about two years, mostly to find resources for her own students to use in the classroom. Primarily, she was more interested to learn at her own pace how to create her own web pages than to spend time chatting in discussion forums, so she was happy with the limited amount of discussion that she took part in. She covered most of the course in a single stretch of three hours. When she does communicate online, she would prefer more meaningful and useful communication with people she knows. She felt that was still some room for improvement in the course itself, but the main obstacle for her was a bad connection to the Internet from her remote town, forcing her to use it in the morning when the lines were less congested.

Mary

Mary had already been using the Internet for 6 years, though mostly just email. She was the Indonesian teacher at her school, and was looking to contact Indonesia in particular. She didn't see much need to interact with the teacher or other students, but would have liked an asynchronous forum for quick question-answer technical support. Overall she found Moodle more productive than other sites, because it was clearly written and fast to download. However it could have benefitted by offering a lot more of the exercises in order to consolidate her learning.

4.2 Factors of the learning environment

My analysis of the data involved counting numbers of students that indicated they desired a reasonable improvement for each of the factors mentioned in the survey. The factors most liked are defined as the ones with the highest mean 'actual' scores and the fewest numbers of students wanting improvement. The factors that needed most improvement are defined as those with largest numbers of students wanting improvement and lowest mean 'actual' scores. Each is listed below, illustrated with unsolicited quotes from the survey comments and interviews that support each result. I will conclude this section with a discussion of some external factors that may have helped or impeded the research process that produced these results.

4.2.1 Factors that participants liked

1. The quality of the web site, resources and navigation.

The factor with which students were most happy overall (expressing the least desire for change) was the quality of the web site itself, in terms of speed, navigation and features such as post-before-participate and the exercises throughout. I have combined the two factors of "web site and resources" with "web site navigation" into one factor because their scores were similar, and they appeared to be understood by students as being the same thing.

Pat posted this comment on the Learning Technologies discussion area:

I have just completed the tutorials on the Internet in Moodle. Just wanted to make a quick comment about how easy the site was to use. The explanations were terrific and after sitting on the computer for [three hours] I don't feel like I've been working at all. Thanks to Martin for making learning so easy. - Pat, July 12.

Helen compared her experiences with a commercial provider of similar courses:

I have enjoyed the tutorials I have done on Moodle much more satisfying than ZDU
A) the language was clearer in the tuts
B) the site worked more efficiently, time waits were shorter
C) the site was better organised easier to navigate
D) I felt I was learning lots in the right sized chunks to remember and be able to use. - Helen, survey.

Jenny found the explanations helped her teach the subject to her own students:

Found the site easy to navigate and learnt many worthwhile tips on how to explain the internet to my class of year sevens. For me further study is to better my knowledge and skills to pass onto our future generation, not for necessarily bettering my personal self. My aim is to engage my students and knowledge of technology is a way to do this. I successfully introduced the Internet to my year 7 class and they are learning to use the net more purposefully. This unit helped me to do this. - Jenny, survey.

Maria liked the style and the fact there was a real teacher available:

I found that the language used in the units was simple, clear without being patronising. There was an opportunity to ask questions (through email) with the writer. - Maria, survey.

In general, it seems the "exercises" were welcome, though some would have liked to have seen more:

Some of the readings that did require exercises were good and the ones which didn't require exercises sometimes left me unsure if I understood it all. - Mary, survey.

Perhaps more interaction (practice) might have made some of the things I've forgotten stick better? - Mary, interview.

...more hands on examples could be included in the Moodle course. - Kim, interview.

I loved the fact that you could not read other posts before you made your post about an issue - it made you be very careful about your position on something. - Florence, interview.

Some students mentioned they preferred the self-paced asynchronous nature of the course:

I love the opportunity this gives me to work at my own pace and think about what I learn and do. - Rosemary, survey.

I found it easy to use and basically skipped through the questions and readings quickly. One advantage of Moodle is that self-regulated learners can pace themselves as appropriate. [...] I like the ability to access effective PD cheaply from isolated locations, the flexibility that Moodle provided in terms of locating the information and progressing through the areas it covered quickly based on background knowledge. - Florence, interview.


I can't think quickly enough to take part in another [realtime] reality online!!!! Call me a scaredy-cat or stick in the mud, I even have trouble keeping up with the Nintendo and Sony Playstation game characters when I'm controlling them! - Kim, interview.

One negative comment was that the discussion forums were hard to find and use, being buried amongst the content:

... links to the comments page would have made it easier to re-visit regularly. - Florence, survey.

It disappointed me that I could not provoke more of a response - I wonder if that was because it was time consuming to expand threads? - Florence, interview.

2. The speed of student computers

Not surprisingly all students indicated that they were fairly happy with the speed of their computers, since they had just received modern laptop computers from EDWA for taking part in the course.

3. The quality of the thinking the course helped students achieve

Overall the students seemed to find the experience a very rewarding one. This is supported by quotes such as:

Topics have forced me to think more critically and form my own opinions. - Colleen, survey.

I have learnt so much. As you are probably aware, I began this course then had several weeks break, and then came back to it. In the time I had "off", I continued with the other elements of the course, via WWW and CD ROM. When I came back to your course, I found I had to go back and re-do/re-read all the modules to refresh my memory. When I came to do this, I found that I had a much greater understanding and appreciation of what I was reading. It helped me to do lots of reading and other course stuff. - Susan, survey.

4. Cognitive support from the teacher

The average score for actual cognitive support from the teacher was quite high (between 'good' and 'excellent'), despite my own impressions that I had hardly participated in the course. It seems the content prepared before the course was all the cognitive support they needed from the teacher. If so, then this factor may be closely tied to the quality of the web site.

4.2.2 Factors needing most improvement

1. Their own knowledge about the internet

Almost unanimously the class still wanted to learn more about the Internet after the course, rating their current knowledge on average as 'lacking' and 'good', but preferring 'excellent'. This is not surprising as it was the reason why they enrolled in the course, and the course was only a short introduction designed to start developing concepts. However, from my experience in education I do believe it reflects a general desire to know more about the new technologies, to take advantage of them and to avoid "getting left behind". An interesting comment from Mary reflects the dilemma of saving time in a busy world while also knowing that deep knowledge comes from experience:

What could be useful (for beginners like me) would be a less public forum where you could ask dumb technical questions...and get answers straight away. Help NEVER seems to cover my problems....and I sometimes spend hours working out something that is so easy if you know. (you don't forget THOSE bits, once you know!) - Mary, interview

Stacey (1997) mentions the effects of previous knowledge of computer-mediated communication as a critical factor in the success of online learning.

2. Participation with other students

Oddly, this did not rate highly as a deficiency in the list of environmental factors, but did rate very highly in the COLLES-P questions that broke this down into specific activities: being asked to explain by other students, asking other students to explain, explaining to other students, and other students explaining. Most students said the current level of participation was between 'seldom' and 'sometimes', but that they would prefer discourse occurred 'often'.

This is supported by research such as Stacey (1997), who concluded that collaborative learning and behaviour made the most effective use of an online environment for education.

However, some students were happy with the level of participation as it was:

I've found that I am the type of person who studies to fulfill my own needs. I'd rather talk to the people I know than get too involved in discussion with other students. Besides online courses allow me to leave the situation and mull over what I've learnt before coming back to reply. - Kim, survey.

I believe that if I take the time to "converse" with people in a particular setting then it is the beginnings of developing an online community. - Kim, interview.

And some felt shy of participating in a public forum:

It forced me to express my opinions in writing, which made me a little uncomfortable. I was not embarassed about my opinions, but rather the way I express myself in writing. I feel more articulate when speaking face to face. Maria, survey.

I don't know what kind of online participation would be beneficial, but I do know that I feel very inadequate when reading the EDWA course BB and am not inclined to contribute and make a fool of myself. Mary, interview.

3. A better quality connection (speed and uptime)

Despite the aid from EDWA in getting the Internet connected, bandwidth (speed) is still generally poor in regional areas, even with a fast modem. Depending on the area, though, these complaints can apply even in urban centres:

Technical problems like slow connections, ZDU and filtering at work are frustrating. Rosemary, survey.

It would be wonderful to have 100% connectivity. Christine, survey.

I have found that the main obstacles to my online learning have been interruptions to my connection and bottlenecks at peak times when everyone else is online. Thus I try to use my connection very early in the morning whenever possible. Kim, interview.

4. A better quality physical location

Some students would have preferred better physical surrounds to study in. In some cases this may have meant they dislike working for long periods on a small screen, but in the more extreme cases it was difficult to study at home among family and work commitments:

Finding time in between work, family and sickness is sometimes an issue. Rosemary, survey.

The best PD I have ever done but very very time consuming. I think it would be very beneficial for some time off of class to be built into the course in order to complete it. Karen, survey.

It can all get a bit frustrating when you are studying in snatches of time after a long day at school. Mary, interview.

4.2.3 Factors that may have hindered the research

While monitoring the course, and while analysing the activity later, I uncovered a number of factors related to this particular study that may have confused some of the students and caused some inaccuracies in their answers to the survey.

Firstly, there was some confusion about the difference between the larger Learning Technologies course and the Internet Overview. It was apparent that some students treated them all as a single web site, not realising that the smaller web site was completely separate entity linked to the main course by a single hyperlink. For some students this misconception appears to have extended to other linked web sites as well. It is unclear what their perceptions of the course boundaries are - this could be an interesting avenue for further research. Secondly, there was widespread confusion between Internet Overview (the course) and Moodle (the system used to implement the course). Explanations were provided, but would be difficult to understand without a much deeper understanding of the internet than most participants had when they started the course. Most people called the whole course Moodle.

4.3.4 Factors that may have helped the research

One factor in the success of this research is that the students were teachers themselves, interested in using the Internet for teaching, and so probably more reflective, mature and helpful than an average class.

A second important factor is that because they had all been given a laptop by EDWA with which to do the course, there was a guaranteed baseline across the class of a quite high-quality computer. This allowed a more even experience of the course, as well as better support. When one person had a problem with their computer it was often the case that they all had the same problem - so that help was more readily available.

4.3 Summary of results

Factors that participants liked

  1. The quality of the web site, resources and navigation.
  2. The speed of student computers
  3. The quality of the thinking the course helped students achieve
  4. Cogitive support from the teacher

Factors needing most improvement

  1. Their own knowledge about the internet
  2. Participation with other students
  3. A better quality connection (speed and uptime)
  4. A better quality physical location

5. Conclusions

Drawing on the student results in Section 4 as well as my own experiences and research in Section 2 and 3, I feel confident that the basic idea of Moodle as a tool to manage a content-based framework supporting embedded educational discourse is an acceptable and useful idea.

As a broad generalisation the students indicated that they liked the web site and the content. They were also happy with the medium to high level support they received from the teacher, even though it was via the pre-written web content rather than direct participation on my part. This suggests that the processes that produced Internet Overview as an extension of my teaching (see Section 2.3) were as successful from the students' point of view as it was from mine.

An important result was the students' strong desire to learn more about the Internet. This suggests that Internet Overview itself can be useful tool for learning about the Internet, particularly if it addresses the major shortcoming of the current version: student interactivity. It is clear that the current discussion areas of Moodle are too clumsy to be really useful. The major work to be done now is to define and implement the next generation of tools for students to interact discursively in a focussed, non-threatening, asynchronous environment that promotes learning.

6. References

Borasi, R. and Rose, B.J. (1989). Journal Writing and Mathematical Instruction. Educational Studies in Mathematics, 20, 347-365.

Cohen, L. and Manion, L. (1994). Research Methods in Education. London, Routledge.

Connor, W., Prain, V. and Hand, B. (1994). Improving science through writing in secondary science: Two examples. Australian Science Teachers Journal, 40(1), 32-47.

Denzin, N. and Lincoln, Y. (1995). The handbook of qualitative research in education. Newbury Park, CA: Sage.

Dougiamas, M. (1998). A journey into Constructivism. http://dougiamas.com/writing/constructivism.html [accessed 19 Jan 2000]

Dougiamas, M. (1999). Reading and Writing for Internet Teaching, http://dougiamas.com/writing/readwrite.html [accessed 19 Jan 2000]

Duffy, T.M. and Cunningham, D.J. (1996). Constructivism: implications for the design and delivery of instruction, In D.H Jonassen, (Ed.), Handbook of Research for Educational Communications and Technology. NY: Macmillan Library Reference USA.

Gergen, K.J. (1995). Social Construction and the Educational Process. In L.P. Steffe & J.Gale (Eds), Constructivism in education (pp 17-39). Hillsdale, New Jersey: Lawrence Erlbaum.

Glynn, S. M. and Muth, D. (1994). Reading and Writing to Learn Science: Achieving Scientific Literacy. Journal of Research in Science Teaching, 31(9), 1057-1073

Guba, E.C. and Lincoln, Y.S. (1989). Fourth generation evaluation. Newbury Park : Sage Publications.

Holt, D., Rice, M., Seagrave, S. and Thompson, D. (1997). The case of the case: The interplay between research and practice on the Web. Proceedings of RIDE '96, 18-33, Deakin University Press.

HREF A: Learning Technology, http://webct.curtin.edu.au/public/EDWA1 [accessed 19 Jan 2000]

Jonassen, D. (1994). Towards a constructivist design model. Educational Technology, 34(4), 34-37.

LeFoe, G. (1998). Creating constructivist learning environments on the web: The challenge in higher education. Proceedings ASCILITE'98 http://cedir.uow.edu.au/ASCILITE98/asc98-pdf/lefoe00162.pdf [accessed 19 Jan 2000]

LeGere, A. (1991). Collaboration and writing in the mathematics classroom. Mathematics Teacher, 84, 166-171

Moore, R. (1994). Writing to learn biology: Let's stop ignoring the tool that works best. Journal of College Science Teaching, 23(5), 289-295.

Novak, J.D. (1996). Concept Mapping: A tool for improving science teaching and learning. In Treagust, D.F., Duit, R., & Fraser, B.J. (Eds), Improving teaching and learning in science and mathematics (pp. 32-43). New York: Teachers College.

Oliver, R., Herrington, J. and Omari, A. (1996). Creating effective instructional materials for the World-Wide-Web. Proceedings Ausweb 97.

Papert, S (1991). Preface. In I. Harel & S. Papert (Eds), Constructionism, Research reports and essays, 1985-1990 (p.1), Norwood NJ.

Salomon, G., Perkins, D.N. and Globerson, T. (1991). Partners in Cognition: Extending Human Intelligence with Intelligent Technologies. Educational Researcher, 20(3), 2-9

Stacey, E. (1997). Collaborative learning at a distance. Proceedings of RIDE '96, 18-33, Deakin University Press.

Taylor, P. and Maor, D. (1999). COLLES-P: Constructivist Online Learning Environment Survey - Preferred.

Taylor, P.C., Geelan, D., Dawson, V., Stapleton, A. (1999, in press). Can teaching in a virtual classroom enhance real learning? FID Review.

Tobin, K. (1993). Constructivist perspectives on teacher learning. In K. Tobin (Ed), The Practice of Constructivism in Science Education (p 215-226). Hillsdale, New Jersey: Lawrence Erlbaum.

Von Glasersfeld, E. (1990). An exposition of constructivism: Why some like it radical. In R.B. Davis, C.A. Maher and N. Noddings (Eds), Constructivist views on the teaching and learning of mathematics (pp 19-29). Reston, Virginia: National Council of Teachers of Mathematics.

Wilson, B. G. (1995). Metaphors for instruction: Why we talk about learning environments. Educational Technology, 35(5), 25-30. http://www.cudenver.edu/~bwilson/metaphor.html [accessed 19 Jan 2000]

Appendix A - Sample screens from the course

Appendix B - The end-of-course survey

Please cite as: Dougiamas, M. (2000). Improving the effectiveness of tools for Internet based education. In A. Herrmann and M.M. Kulski (Eds), Flexible Futures in Tertiary Teaching. Proceedings of the 9th Annual Teaching Learning Forum, 2-4 February 2000. Perth: Curtin University of Technology. http://lsn.curtin.edu.au/tlf/tlf2000/dougiamas.html

This article is available also at http://dougiamas.com/writing/tlf2000/


[ TL Forum 2000 Proceedings Contents ] [ TL Forums Index ]
HTML: Roger Atkinson, Teaching and Learning Centre, Murdoch University [rjatkinson@bigpond.com]
This URL: http://lsn.curtin.edu.au/tlf/tlf2000/dougiamas.html
Last revision: 19 Feb 2002. Curtin University of Technology
Previous URL 20 Jan 2000 to 19 Feb 2002 http://cleo.murdoch.edu.au/confs/tlf/tlf2000/dougiamas.html