Teaching and Learning Forum 99 [ Contents ]

Student feedback collected using groupware

Dr Doug Atkinson
Information Systems, Curtin Business School
Curtin University of Technology
At the last forum I reported on the use of GroupSystems to collect student feedback on a large first year unit. Since then I have collected data from two further cohorts of students. The most recent cohort includes feedback specifically about lectures and tutorials. In particular there is analysis of the effects of anonymity and breakdown into national culture groupings including Australian, Singaporean, Malaysian and Indonesian students.

GroupSystems is an example of software that supports face-to-face meetings. Key features include anonymity, parallel input, recording and structure. The structured modules provide support for idea generation, idea organisation and idea evaluation. Whilst there are numerous business applications the software can be useful in the classroom for structured discussions including collecting student feedback.


Introduction

Student feedback is one way of evaluating the effects of our teaching on student learning. Most teachers are interested in how the curriculum and their implementation of it is received for their own job satisfaction and professional development. Fuelled by quality concerns, scarce public funds and increasing fees being paid by students, University management also has a keen interest in student feedback. Formal end of unit and course questionaires are a common feature of higher education with names such as "student appraisal of teaching", "student opinion questionairre", "course experience questionaire" and "student evaluation of education quality". Last and by no means least students are interested to hear how their lecturers and courses are perceived by their colleagues and in what ways the University is responsive to their concerns.

The decrease in government funding for higher education has also seen a commensurate rise in the intake of international students particularly from the South-East Asian region such as Singapore, Malaysia and Indonesia. The increasing diversity of the classroom increases the need for teachers to be aware of cultural differences. Examining student feedback is one way of identifying differences in student experiences.

In the last forum (Atkinson, 1998) I reported on an alternative approach to traditional questionairre methods. This was based on a focus group approach supported by software generically known as Groupware. A series of pilot sessions were held for small groups of a first year information systems unit known as IS100. In the past semester this pilot project was extended to a larger cohort of students. The cohort was divided into national culture groups including Australian, Singaporean, Malaysian and Indonesian. Furthermore an investigation was carried out as to the effects of anonymity on the productivity of each cultural group.

In the first section of this paper groupware is briefly reviewed. Research on national culture is then discussed. The research design is described followed by a discussion of findings and implications for future research.

Groupware

Groupware is software designed to support the work of groups. This work includes same time, same place work and different time and or place work. Groupsystems is a software designed for the familiar same time same place environment - the face-to-face meeting. It provides support for generating, organising and evaluating text based ideas. This is the focus for the research in this study. Software such as Lotus Notes/Domino provides support for geographically distributed group members or members who work asynchronously. In particular it provides support for email, discussion groups, document databases, workflow and web site development. Whilst a face-to-face environment permits an intensity of interaction where verbal interaction and human facilitation can be used to complement the text based exchanges, much of the student feedback described in this paper could be generated in the distributed environment using internet based software.

The Groupsystems software is based around the familiar meeting paradigm. The software enables an agenda to be developed consisting of modules that support brainstorming, categorisation and voting. A meeting room environment consists of participant workstations in a U shaped network with a workstation that supports the agenda creation and the running of the agenda. A data projector displays the public aggregate data and a printer is available for the timely creation of reports. Either one or two persons help facilitate the meeting taking the participants through the agenda and the supporting software modules.

The main features of the Groupsystems environment are anonymity, simultaneous input, structured modules and reporting. Anonymity allows individuals freedom through alleviating fear of evaluation by peers or superiors. This can result in richer and more honest ideas. Simultaneous or parallel input allows many persons to enter ideas at the same time thus promoting productivity - more ideas generated in a given time. The structured modules of Groupsystems help to keep group members focussed on task and provide support for activities such as multi-criteria decision making which are not commonly undertaken at traditional meetings because of the difficulty of structuring and summarising inputs.

The GroupSystems software was developed in a North American cultural context and as such reflects values such as technology, democracy, anonymity, structure, productivity and recording. Given the globalisation of business it is of interest to see how other cultures appropriate the software. First a framework for comparing cultures is required.

National culture

Hofstede (1978) defined culture as "the collective programming of the human mind, obtained in the course of life, which is common to members of one group and opposed to another". Culture is acquired knowledge that people use to interpret social behaviour. Many problems facing businesses on the international stage may be caused by a lack of understanding or familiarity with cultural differences.

In a series of studies at IBM, Hofstede (1983; 1985) identified five attitude or value dimensions differentiating national cultures and then positioned particular cultures on those dimensions. These five dimensions are labelled individualism/collectivism, power distance, uncertainty avoidance, masculinity/femininity, and time orientation.

Individualism refers to a preference for a loosely-knit social network in which individuals are expected to take care of themselves and their immediate families, whereas collectivism refers to a preference for a tightly-knit social framework in which individuals can expect their relatives or other closely-related groups to look after them (usually in exchange for unquestioning loyalty).

Power distance is the extent to which a culture accepts the fact that power in organisations and families is distributed unequally. In a culture with high power distance (such as most Asian cultures), subordinates defer to superiors and do not question their authority, whereas in a culture with low power distance (such as in most Western countries), subordinates are relatively free to disagree with their superiors.

Uncertainty avoidance refers to the extent to which people within a culture are uncomfortable with situations they perceive as unstructured, unclear, or unpredictable. It indicates the extent to which members of a culture feel threatened by uncertain and ambiguous situations and try to avoid those situations by adopting strategies such as seeking greater career stability, establishing more formal rules, not tolerating deviant ideas and behaviours, and believing in absolute truths.

Cultures that exhibit masculinity show a preference for achievement, heroism, assertiveness and material success, as opposed to a culture exhibiting femininity which shows a preference for relationships, modesty, caring for the weak, and quality of life.

Time orientation is considered to be of importance to decision-making situations and lies on a "continuum with long-term orientation at one pole and short-term orientation at the other" (Watson and Raman, 1994). Cultures with a short-term orientation are characterised by people who expect quick results, and in group decision making in this culture problems are considered independently and quick solutions sought. Cultures with long-term orientation (such as Japanese) consider the long-term future more important and are more likely to accept that benefits may take longer to be achieved.

Western and Asian societies differ on a number of these cultural dimensions, but especially in relation to power distance. This cultural characteristic can be observed through the concept of face where in Asian cultures (which have high power distance) 'loss of face' would occur for both superior and subordinate if open disagreement occurs between them. This would not be an issue in Western cultures where power distance is low and 'face-saving' is less important.

Hu (1944) identified two dimensions of the concept of 'face', namely, lien (the confidence others have in your integrity) and mien-tzu (outside face or reputation based on one's personal efforts). Maintaining face is seen as being of paramount importance in business dealings with Asian cultures (Redding and Ng, 1982).

Research question and hypotheses

Given the preceding discussion of Groupware anonymity and national culture the research was designed to answer the question:
Do groups from high power distance (and correspondingly high concern for 'saving face') national cultures derive greater productivity from Groupware anonymity than low power distance national cultures?
To account for differences in group abilities it was decided to measure productivity as the productivity gain between an identified and an anonymous treatment. Hofstede's power distance index orders the national cultures as Malaysian > Indonesian > Singaporean > Australian (Hofstede, 1985). Hence the expectation is that productivity gains from identified to anonymous treatment would be in this order. Thus the hypothesis to be explored is:

Ha : The productivity gain from identified (I) to anonymous (A) treatment is in the order Malaysian > Indonesian > Singaporean > Australian.

Research design

An experimental design was chosen because of the desire to isolate the relationship between national culture and Groupware anonymity. Future research methods could include field studies. Here however the ability to measure productivity and make comparisons between anonymous and identified conditions is problematic. The field study is better suited to discovering perceptions and behaviours of participants with regard to the whole Groupware activity.

The availability of at least four national cultures amongst the Curtin University student population represented am opportunity for implementing an experimental design. In addition to the research objective the experiment provided an opportunity to gain student feedback on a core first year subject, and to involve students in Groupware as part of their curriculum.

The key feature of the design was that each national culture group would undertake two Groupware sessions, one under the anonymous treatment and the other under an identified treatment. This repeated measures design would allow comparisons of productivity to be made whilst attempting to control for a key variable - that of group composition. The details of the design are described below.

Measurement of dependent variables

Productivity of the groups was measured by counting the number of words generated during an idea generation step. The number of words generated was chosen rather than the number of comments to reflect the varying length of comments from a single word to a five line sentence statement. To account for varying group size the measure was adjusted to be the number of words generated per group member. The time over which the comments were generated was set to be ten minutes.

To control for natural group variability the productivity was measured as a gain between the identified and anonymous treatments. It was expected that productivity would be greater under the anonymous condition. However, if it wasn't, a negative gain would be recorded. The productivity measurement was conducted via the Groupware system electronic record from each session and the use of a word processor word count feature.

Perceptions of participants with regard to the anonymous and identified treatments were collected via a questionnaire which included a scale and an opportunity for open comment. Given the repeated measures design where each participant had experienced both conditions it was possible to ask participants directly to compare the treatments. This questionnaire was administered at the end of the second Groupware session.

Participants

The participants were drawn from first year undergraduates at Curtin University. These students were enrolled in the Bachelor of Commerce and were studying a core unit in Information Systems. The Bachelor of Commerce attracts a large number of international students particularly from Singapore, Malaysia and Indonesia. Understanding the needs of these and other national cultures is thus of direct interest to the successful internationalisation of the Curtin Business School.

Students were invited to participate in two Groupware sessions in lieu of their normal tutorial/laboratory sessions. Course credit of 5% was given for those students who participated in both sessions. In addition to the course credit incentive, Groupware was being discussed in lectures and thus was potentially of interest. A further incentive was that students were to be given the opportunity to provide feedback on the Information Systems course to date.

Four national cultures were identified from participant supplied information. These were Australian, Singaporean, Malaysian and Indonesian. Based on the relatively low enrolment numbers in this semester it was not possible to form a statistical sample size of groups, rather there was enough for a pilot study. The actual number of groups was Australian (7), Singaporean (3), Malaysian (1) and Indonesian (1). Unfortunately the group size for the Malaysian group was only three and was thus omitted from the analysis. The intention was to form group sizes of ten commensurate with the seating capacity of the Curtin Groupware Facility. Due to varying availability of participants the actual group sizes varied. Furthermore participants were allocated to groups based on timetabling information supplied.

Tasks

To suit the two session design two tasks were constructed. The tasks were intended to be controversial so that there would likely be a treatment effect, of sufficient interest to motivate the participants to be involved, and equivalent for experimental control. It was difficult to satisfy these competing criteria. The issues chosen were "feedback on the lectures" and "feedback on the tutorials".

The Groupware process design for each task was similar consisting of idea generation, idea organisation and discussion, and finally idea evaluation. The key measurement of productivity (words generated per participant) was based on the ten minute idea generation step. It was felt that measurements thereafter would be less likely due to culture and anonymity and more to do with group dynamics and facilitation. Each session was designed to be completed in one hour.

Controls

As described previously the two session design helped control for the key variable of group composition. Another major source of variation in Groupware activity is facilitation. Consistent with the facilitated mode of Groupware use adopted by Curtin academics and practitioners a facilitator was used in all the sessions. The facilitator acted as both process and technical facilitator as it was felt that the tasks did not necessitate two persons.

To control for facilitation variation, the four facilitators involved had all undergone the same amount of education and training and were given a script to follow for all the sessions. The participants from Singapore, Malaysia and Indonesia were in their first year in Australia and thus could be considered relatively free of Australian cultural influences. Additionally given that they were at least of eighteen years of age their native cultural influences would have been well developed. Thus the national culture identification was considered to be quite strong. It was noted that there are sub cultures based on race (for example Chinese Malays) or geographic location (Country Malay), that could be the subject of further research.

Further controls included the room environment, Groupware technology (GroupSystems for Windows) and the GroupSystems tools used (topic commenter, categoriser and vote). All the participants were GroupSystems novices.

Treatments

The identified and anonymous conditions were implemented through the Groupware technology. The software allowed the full name of a participant to be tagged to any comment he or she typed into the system. A list of names was known before the session to the facilitator. Depending on the type of session the facilitator would switch on the identified condition or switch it off to allow anonymous generation of ideas.

Results

For each national culture the productivity gains between the identified and anonymous treatments are given in Table 1. These have been ordered from greatest gain to least gain.

Table 1: Productivity gain from identified to anonymous

National culture Treatment#Words# Participants#Words per
participant
Productivity
gain (A - I)

Singaporean
N=3 groups
Anonymous
Identified
1413
982
17
16
83
61

+22
Indonesian
N=1 group
Anonymous
Identified
440
273
10
9
44
30

+14
Australian
N=7 groups
Anonymous
Identified
3258
3488
38
37
86
94

-8

The hypothesised order based on power distance of Indonesian > Singaporean > Australian, is supported for two of the three national cultures. Surprisingly the Singaporean groups have derived greater productivity than the Indonesian group. This may be because of the small sample size we are yet to observe the natural variability in group productivity. This caution could be applied to all of the results indicating the tentative nature of this pilot study.

The implicit expectation of a gain in productivity for all cultures is supported for the Singaporean and Indonesian groups- indicating a generally positive effect of anonymity over the identified condition. However, suprisingly the Australian groups were more productive in the identified condition. Aside from the issue of small sample sizes a possible explanation is that learning effects from one session to the next had a more positive effect on productivity for Australian groups compared to the othe cultures.

Open ended comments generally supported the preference for anonymity. A sample of comments is given in Table 2 below. These comments illustrate the diversity of individual views on anonymity both for and against. There was no obvious differences in the content of the comments across cultural groups. A tentative proposition is that the benefits and drawbacks of anonymity are perceived similarly across these cultural groups. Field study research could be used to investigate this proposition.

Table 2: A sample of comments on preference for anonymity or identified

Prefer anonymity

  • "It usually boils down to what others would think of me."
  • "People can contribute more."
  • "Probably people feels more open and dare to put their opinion when their identities are unknown."
  • "Because sometimes you feel constrained on how your opinions contrast with other people's feelings."
  • "Fair when deciding the topics, everyone is equal. There is not racism, sexism or skin colours."
  • "I guess being anonymous gives you less fear of speaking up. More security and less embarrassment to a certain extent."
  • "Its easier for people to be totally honest."
  • "A lot more ideas were generated in the anonymous session and this gave us more to work with."
  • "I feel people are more comfortable in presenting ideas if it is anonymous - especially for self-conscious or quiet people. It's a great advantage to Groupware."

Prefer identified

  • "You can deal with the personality so its more understandable."
  • "...identified has a good pt so that we'll know what the person is trying to say."
  • "I would like to be anonymous at times but certain times I would like to be identify. Especially, when giving good ideas."
  • "It's good to know people's personal feelings and viewpoints rather than always being anonymous spokesperson."

With regard to the content of the feedback received on lectures and tutorials the results were presented in a written report to the unit controller. Whilst the Groupsystems process allowed a large number of comments to be efficiently collected a comment from the unit controller was that "we have only scratched the surface of the issues here". This indicated the need to gain a deeper understanding of student issues possibly through one-to-one interviewing. The process however was perceived as a useful supplement to the quantitative surveys which are currently used.

The students were generally positive about the Groupware experience in that it gave them a hands on experience of technology they were reading about and an opportunity to provide feedback on the course. Additionally students were able to see what their colleagues views were giving them a better understanding of the variety of feedback and whether their own views were shared or idiosyncratic.

Conclusion

For the four national cultures studied and limited by small sample size, this pilot study tentatively supports the propositions that: The availability of up to 1000 students (100 groups) in the next semester of this unit will provide a statistical sample suitable for measuring the variability in productivity and perceptions of anonymity. Whilst desirable, the two session design may not be practical with this number of groups.

To complement the experimental work aimed at measuring productivity, field studies with different culture groups are necessary to find out how Groupware in general (and anonymity in particular) is perceived by business. It may be that despite demonstrated productivity in experimental groups, perceptions of Groupware and anonymity are such that traditions of identified contribution without Groupware technology prevail.

The larger question is how Groupware values and technology are appropriated by different cultures and also by groups of mixed cultures. With increasing globalisation and internationalisation of business the interaction of mixed cultural groups may be based on Groupware models that have not yet been considered.

References

Atkinson, D. (1998). Evaluating Teaching/Learning via Groupware Focus Groups. Proceedings 1998 Teaching and Learning Forum, University of Western Australia, Perth February 4-5th pp22-27. http://cleo.murdoch.edu.au/asu/pubs/tlf/tlf98/atkinson-d.html

DeSanctis, G. and Gallupe, R.B. (1987). A Foundation for the Study of Group Decision Support Systems. Management Science, 33, 5, May, pp589-609.

Connolly, T., Jessup, L.M. and Valacich, J.S. (1990). Effects of Anonymity and Evaluative Tone on Idea Generation in Computer-mediated Groups. Management Science, 36, 6, June, pp689-703.

Gallupe, R.B., Bastianutti, L.M. and Cooper, W.H. (1991). Unblocking Brainstorms. Journal of Applied Psychology, 76, 1, pp137-142.

Gray, P. and Olfman, L. (1989). The User Interface in Group Decision Support Systems. Decision Support Systems, 5, pp119-137.

Ho, T.H., Raman, K.S. and Watson, R.T. (1989). Group Decision Support Systems: The Cultural Factor. Proceeding of the 10th International Conference on Information Systems, Boston, Massachusetts, December 4-6, pp119-129.

Hofstede, G. (1978). Culture and Organisation - A Literature Review Study. Journal of Enterprise Management, 1.

Hofstede, G. (1983). The Cultural Relativity of Organizational Practices and Theories. Journal of International Business Studies, Fall, pp75-89.

Hofstede, G. (1985). The Interaction between National and Organisational Value Systems. Journal of Management Studies, 22, 4, pp347-356.

Hu, H.C. (1944). The Chinese Concept of Face. American Anthropologist, 46, pp45-64.

Jessup, L.M., Connolly, T. and Galegher, J. (1990). The Effects of Anonymity on GDSS Group Process with an Idea-generating Task. MIS Quarterly, 14, 3, September, pp312-321.

Nunamaker, J.F., Jr., Applegate, L.M. and Konsynski, B.R. (1987). Facilitating Group Creativity: Experience with a Group Decision Support System. Journal of Management Information Systems, 3, 4, Spring, pp5-19.

Rawlinson, J.G. (1981). Creative Thinking and Brainstorming. Gower Publishing Limited, Westmead.

Redding, S.G. and Ng, M. (1982). The Role of 'Face' in the Organizational Perceptions of Chinese Managers. Organizational Studies, 3, 3, pp201-219.

Tan, B.C.Y., Watson, R.T., Wei, K.K., Raman, K.S. and Kerola, P.K. (1993). National Culture and Group Support Systems: Examining the Situation where Some People Are More Equal Than Others. Proceedings of the 26th Annual Hawaii International Conference on the Systems Sciences, Hawaii, January, Vol. IV, pp132-141.

Valacich, J.S., Jessup, L.M., Dennis, A.R. and Nunamaker, J.F., Jr. (1992). A Conceptual Framework of Anonymity in Group Support Systems. Group Decision and Negotiation, 1, pp219-241.

Vogel, D.R. (1992). Group Support Systems in International Contexts. Proceedings of the 2nd International Organisational Behaviour Teaching Conference, Curtin University, Perth, Western Australia, December 16-19.

Watson, R.T. and Raman, K.S. (1994). Culture: A Fourth Dimension of Group Support Systems. Communications of the ACM, 37, 10, pp45-55.

Please cite as: Atkinson, D. (1999). Student feedback collected using groupware. In K. Martin, N. Stanley and N. Davison (Eds), Teaching in the Disciplines/ Learning in Context. Proceedings of the 8th Annual Teaching Learning Forum, The University of Western Australia, February 1999. http://lsn.curtin.edu.au/tlf/tlf1999/atkinson-d.html


[ TL Forum 1999 Proceedings Contents ] [ TL Forums Index ]
HTML: Roger Atkinson, Teaching and Learning Centre, Murdoch University [rjatkinson@bigpond.com]
This URL: http://lsn.curtin.edu.au/tlf/tlf1999/atkinson-d.html
Last revision: 24 Feb 2002. The University of Western Australia
Previous URL 10 Jan 1999 to 24 Feb 2002 http://cleo.murdoch.edu.au/asu/pubs/tlf/tlf99/ac/atkinson-d.html