Traditional evaluation methods such as self report questionnaires with numeric scales are useful for providing summative indicators of teaching performance. However questionnaires are less useful as a guide to making changes to curriculum design as the statistical summaries are lean on meaning.
Group interviews or focus groups are a means of obtaining a broad range of opinion and a rich range of ideas through interaction of participants. The challenges in conducting focus groups are to ensure broad participation and freedom of expression with appropriate recording of ideas. Whilst focus groups can be conducted with cards, butchers paper and whiteboards there is software (groupware) which can be tailored to suit.
This paper reports on some groupware focus groups conducted to evaluate a large first year unit in information systems.
In the first section of this paper, teaching and learning evaluation is reviewed via personal experience of survey questionnaire approaches. The focus group interview is then described as an alternative or supplementary approach. Groupware is then introduced as a means of supporting the facilitation of focus groups. Following description of an application, the paper is concluded with a discussion of lessons learnt.
My own introduction to evaluation came when as a naive young tutor I attended an induction course for new staff and was informed about the survey questionnaire SAT (Student Appraisal of Teaching). This was intended as a professional development tool. Staff could use it voluntarily, information was confidential and it could also be used for staff promotion purposes.
As I diligently collected SATs I pondered what the average ratings meant about my teaching and whether and how I should change the way I did things. For example I appeared to rate poorly on 'feedback' the reality however was that I was tutoring class sizes that went from 16 to 32 in a 1,000 student unit where the assessment had been paired down to the minimum of a mid-semester test and a final exam. I also pondered what sort of feedback students expected. Did they want me to mark something each week? Ask them more questions? Or just spend more time with them?
As the years rolled by there appeared to be a greater emphasis on doing SATs. Emails would come from Heads of School and reminders from office staff to undertake SATs. No one had gone as far as saying they were mandatory. However, expectations had gone from SATs being the exception to being the rule. When I went for a promotion, SATs were suggested as a performance indicator and I diligently filled out the 10 years worth in a summary table provided by the University. I'd heard stories of people who had got "5s" on their SATs and wondered whether my "4s" would be enough when competing against them.
Over the ten years I'd also started doing my own evaluation forms based on my experiences of qualitative research. On these forms I would ask open ended questions such as "Best things?", "Worst things?" and "Improvements?". Responses were a little richer than the SAT forms and gave me a few ideas for improvements, however they could still be lean on meaning and open to interpretation. In places where I provided a mix of quantitative and open ended questions, respondents filled in the quantitative responses in preference, possibly because it took less thought and they thought this is what the lecturer "really wants".
The SATs have now been replaced by a larger survey instrument called SEEQ (Student Evaluation of Education Quality). This instrument includes a set of 44 questions (the SAT consists of six questions) and space for open ended comments on the back. My recent experience of SEEQ is similar to my mixed survey forms where students fill out the front page quantitative scale questionnaires and leave the open ended comments fairly brief.
Whilst the survey questionnaires have continued to satisfy institutional requirements of accountability, the lack of depth in providing guidelines to changes in curriculum has caused me to consider interviewing students. To get some representativeness and efficiency I considered interviewing groups rather than individuals. My own research into groupware provided a means of conducting focus groups in a novel and efficient manner. Given that groupware is a part of the information systems curriculum I could also get some synergy from allowing students to experience groupware whilst providing evaluation of their experience in a given unit.
Focus groups are applied in market research, public policy development, and political opinion assessment as well as being a general technique for undertaking social research.
The purpose of a focus group is to emerge through group discussion, perceptions which will be useful to the broader research objective. These perceptions will have been subjected to scrutiny and debate and to some extent be representative of individuals' views in the presence of a group. The latter point emphasises the social quality of the perceptions. Social research is often concerned with the influencing effect of group members given that individuals' beliefs and actions are moderated by their perceptions of others. Thus the purpose of the group interview is not primarily to save time in conducting individual interviews but rather to discover perceptions within a social context.
As an example, students work in groups including lectures, tutorials, laboratories and project groups thus if discussion of the teaching and learning experience is facilitated in a group the researcher may get understanding pertaining to making the group experience more effective. Concentrating solely on the individual perspective could result in some social aspects being overlooked.
The facilitator, who is sometimes referred to as the moderator, is critical to the success of a focus group. They are involved in the preparation, conduct and follow-up of the focus group. Their role demands an understanding of group dynamics and the use of methods to intervene and facilitate achievement of a rich set of perceptions useful to the research objective.
The objective of the focus group is to emerge the group's perceptions rather than the facilitator's perceptions. Thus a successful facilitator will concentrate on supporting the process of the discussion. Whilst this may include finding participants' views that challenge a given view, the facilitator will avoid putting their own views into the content. A facilitator who has a vested interest in the discussion topic and outcomes finds it more difficult to remain content neutral than an external facilitator. For this reason, an external facilitator is desirable.
Group dynamics such as conflict, domination, free riding and groupthink which are natural group phenomena can impact on a focus group making it difficult to achieve a rich set of representative perceptions in an efficient manner. The facilitator intervenes by using structured group processes and facilitative techniques. The former can include agendas, brainstorming methods, nominal group technique and cognitive mapping. Facilitative techniques include establishing rapport with participants, using eye contact, active listening, and use of ground rules.
Traditionally focus group facilitators have made use of whiteboards, butcher's paper, and index cards as support tools for focus group sessions. The development and availability of software designed for supporting groups offers support for both recording and structuring focus groups.
This paper concentrates on application of groupware (GroupSystems) to the face-to-face environment of focus groups. Support for remote participants is a topic for future research.
GroupSystems consists of modules designed to support groups in generating, organising and evaluating ideas. These modules can be formed into an agenda to structure activities such as information gathering, problem solving, planning, and decision making. An example is the support of a SWOT (strengths, weaknesses, opportunities and threats), a common activity in strategic planning.
The software runs on a network and provides differing facilities for participants and meeting facilitators. The facilitator has the facility to design the agenda, start and stop participants and tailor the participantste functions. Facilitation of meetings is critical to success and may consist of a process facilitator to manage group process and discussion, and a technical facilitator to run the software. If the meeting is not too complex and the facilitator is sufficiently skilled, one person can undertake both the process and technical facilitation.
Whilst the software can support distributed meetings it is commonly used for face-to-face meetings in a purpose built facility or computer laboratory/training room. A purpose built facility will include an appropriate seating arrangement, projection facilities, unobtrusive computers, and a printer.
Each participant has their own computer terminal through which they can enter text, numbers and graphics within a GroupSystems module. The technical facilitator has access that allows him or her to run appropriate modules for the group, and a projector for displaying public data generated in the course of a session.
Key features of the GroupSystems environment are:
Benefits may include:
The following describes the application of the GroupSystems software to support student focus groups aimed at evaluation of teaching and learning in a unit of study. The evaluation was then to be communicated to the lecturer in charge for changes in future semesters.
The author acted as facilitator for the sessions. Whilst I had a general interest in how the unit was being received I was not involved in the teaching of the unit beyond having given a guest lecture on groupware. Thus my neutrality to the content and perception of this by the students was certainly greater than had the lecturer in charge facilitated the sessions.
After gaining agreement from the lecturer in charge to undertake some evaluation sessions, and giving a guest lecture on groupware, the author invited groups of ten students to the GSS Facility for a 50 minute session, to experience groupware and undertake an evaluation of the unit.
The modules of GroupSystems were matched to a three step agenda of
The facilitator then shifted the ideas into a module called categoriser to enable public discussion and organisation of the ideas over about 30 minutes. Participants viewed the projected public screen while the facilitator led a verbal discussion of each idea with the objective of gaining a shared understanding of the meaning, and grouping the ideas into a smaller set of categories.
Having completed public discussion of the ideas, the facilitator shifted the categories into a vote module to enable participants to assess importance. This was defined on a ten point scale as those things which the lecturer in charge should attend to. This module allowed private and anonymous input, and produced a summary distribution.
A brief discussion of the ratings was held. As a reality check, the facilitator summarised the main points and looked for confirmation from the participants. This was done to avoid misinterpretation based solely on the recorded voting results. As with quantitative survey questionnaires, the voting activity is open to misinterpretation however the interactivity of the focus group permits checking that the message has been received as intended.
With the 50 minutes up, the facilitator thanked the participants and confirmed that the participants were happy to have their concerns passed to the lecturer in charge. A report of the session generated by the system was printed and passed to the lecturer in charge for discussion with the facilitator.
Based on the success of these pilot sessions, next semester the groupware focus groups will be held during scheduled laboratory sessions so that all students can participate. To accommodate the large numbers, trainee facilitators from a postgraduate unit on groupware facilitation will facilitate the sessions.
The context of information systems students provides a great opportunity for groupware focus groups because exposure to the groupware can form part of the syllabus. Other disciplines may not have the same opportunity nor the resources to afford groupware. Nevertheless focus groups can still form an additional form of teaching and learning evaluation.
A broader issue of evaluation is the next step of acting upon it. Students providing feedback to academic staff early in semester will quickly become cynical if there is no explanation concerning improvements. Institutions demanding evaluation will also need to be genuinely interested in resourcing improvements to teaching and learning.
Jessup L. M., and Valacich, J. S. (Eds.). (1993). Group support systems: New perspectives. New York: Macmillan.
Johansen, R., (1988). Groupware: Computer support for business teams. London: Collier MacMillan.
Lotus Notes. Lotus Development Corporation.
MeetingWorks. Enterprise Solutions, Bellingham, Western Washington.
Ramsden, P. (1992). Learning to teach in higher education. Routledge: London.
Stewart, D. W. and Shamdasani, P. N. (1990). Focus groups: Theory and practice. Sage: London.
The Australian (p. 40 October 22 1997). Those who can't teach.
|Please cite as: Atkinson, D. (1998). Evaluating teaching/learning via groupware focus groups. In Black, B. and Stanley, N. (Eds), Teaching and Learning in Changing Times, 22-27. Proceedings of the 7th Annual Teaching Learning Forum, The University of Western Australia, February 1998. Perth: UWA. http://lsn.curtin.edu.au/tlf/tlf1998/atkinson-d.html|