Teaching and Learning Forum 2004 Home Page
Teaching and Learning Forum 2004 [ Proceedings Contents ]
Using action research to embed systems thinking concepts in postgraduate IT teaching

Donald C. McDermid and Anita A. Bose
Edith Cowan University

The paper discusses the results of an action research study whose aim was to improve the quality of teaching systems concepts by using a software program that actually executes the models in order to re-inforce the concepts being learnt. The conclusions reached were that using such software in a structured learning environment did indeed improve the quality of teaching and had an impact on the ability of students to model problems more systemically. Further, lessons were learnt in the mechanics of how to structure laboratory exercises to support this goal.


Introduction

All reflective teachers look and re-look at their teaching in a quest to improve the learning context. They search for underpinning concepts and ideas that permeate their discipline or at least a domain of their discipline and try to ensure these ideas are embedded as deeply within the fabric of their teaching as possible.

While much of the teaching that goes on in IT courses is necessarily techno-centric focussing on the software and hardware technologies that underpin the knowledge economy, it has become widely accepted within the profession that there needs to be a balance between the techno-focus and the context or environment in which IT is undertaken (ACS Certification Program, 2003). This latter is usually referred to as the people or organisational context. Many coursework masters degrees in IT, for example those at the School of Computer and Information Science at Edith Cowan University, offer a number of units in this socio-organisational domain to complement the more technically oriented units and thus provide a more well-rounded preparation for working in the IT industry. Such units cover professionalism, ethics, legal issues, service level management, strategic planning and computer consultancy. The unit discussed in this paper is entitled 'software and systems management'. Its brief is to introduce students to a range of methods and concepts that empower an IT professional with respect to facilitating solutions, analysing problems, creating business cases and so on. A major part of this unit is the teaching of systems concepts as a means of modelling the underlying features of many situations that commonly occur in practice. It is important that students work in as 'authentic' a manner as possible.

As an example of the kind of problem discussed and modelled, please refer to Figure 1. Here, a typical organisational problem is depicted involving late orders, expediting orders and order delivery. The diagram should be read as follows. An increase in the number of late orders leads to a need to expedite more orders. Expediting more orders will increase the rate of order delivery. Increasing the rate of order delivery will reduce the number of late orders and thus have a balancing effect on the system. In Figure 1 the symbol 's' indicates 'same' or more formally an increase in X leads to an increase in Y and the symbol 'o' stands for opposite which means that an increase in X leads to a decrease in Y. The symbol B in the figure indicates that the system depicted is in balance. Or at least that is the theory. Figure 2 indicates that an unexpected side effect has occurred because expediting more orders has meant that there has been more disruption to production and so there has been an increase in missed production. Overall this means that the initial solution proposed (Figure 1) has actually had the opposite effect. In fact it has produced a system that will steadily get worse, because as more deliveries are missed, there will be more late orders and thus (as many systems find in practice) more pressure to expedite more orders and so on and so on. The 'R' in the lower part of Figure 2 stands for re-inforcing which means that the system will spiral out of control.

Figure 1

Figure 1: Naive scenario of a typical organisational problem situation

Figure 2

Figure 2: Realistic scenario of a typical organisational problem situation
adapted from Maani and Cavana (2000)

While this kind of modelling is an effective tool that provides significant enlightenment to students and provokes much discussion and debate in class, it doesn't go far enough or deeper in terms of providing understanding of the unique dynamics of any particular situation. For example, from a management perspective it would be valuable to know if it were possible to increase the rate of expediting orders to a certain level before production levels are significantly affected.

Clearly in order to accomplish this, it is necessary to have a quantitative model of the relationship between each of the above components of the 'system' and to be able to execute a running model in a similar way as we execute spreadsheets to answer 'what if' type questions such as the one above. Ithink software and its educational partner Stella (High Performance Systems Inc, n.d.) offer the opportunity for students to explore such problems quantitatively and therefore to obtain a much more intimate understanding of the nature (dynamics) of such situations. It is contended that this juxtaposition between theory and practice (or conceptual model and executable model) is a precursor to deep learning and provides valuable insight into the nature of the domain being studied.

Figure 3 is an example of a model produced by a student using ithink as part of an assignment. While it is outside the scope of this paper to introduce the detailed notation used in this diagram, the reader should note the following

Figure 3

Figure 3: An example of a solution produced by a student using the ithink software

The remainder of this paper describes a research project using action research in which an honours student was supervised in conducting a study to extend the Systems and Software Management unit to incorporate an additional component that, through laboratory exercises, coached postgraduate students in the use of the ithink software and attempted to measure the impact of this experience on their learning. The remaining sections are thus entitled research method, results and reflection.

Research method

Action research was the research method chosen for this project as its characteristics and flexibility were particularly well suited to the needs of this research problem. Hart and Bond's definition of action research is quite appropriate here.

... educative, problem-focused, context specific and future-orientated method for conducting qualitative research, involving individuals as members of social groups whilst aiming at improvement and involvement. It involves a change intervention and a cyclic process in which research, action and evaluation are interlinked. Action research has been founded upon a research relationship in which those involved are participants in the change process. Hart and Bond (1995, pp.256)

It should be apparent from the foregoing discussion that the scope of this action research project was socio-technical i.e. not just involving the technical or functional aspects of the computer software and hardware but also the social and organisational context in which the teaching took place. This context included at one end how the delivered unit actually fitted into the routine day-to-day operational environment of the students through to how systems concepts made sense to students in general.

The main research question was to ascertain the extent to which using the ithink software in a structured learning environment of laboratory exercises enabled students to create better mental models and solutions to complex problems, thus improving the academic achievement of students. A related minor aim of the research was to ascertain the number and nature of laboratory exercises that best supported the academic achievement of students. The research team included supervisor and research student as principal researchers and student group as participants.

The study was divided into five main phases for the purpose of supervising the student and took approximately one year to complete.

  1. Develop the proposal and project plan
  2. Develop the laboratory exercise material
  3. Obtain feedback from pilot students
  4. Obtain feedback from enrolled students
  5. Write the thesis (Bose, 2003)
These phases were then operationalised into the flowchart in Figure 4.

Figure 4

Figure 4: Operationalised research flowchart after Checkland (1991)

Legend

  1. Enter problem situation: Step for negotiating project as a whole. At this stage the lecturer initiates the research problem and proposes this study be undertaken.
  2. Establish roles: Step for agreeing roles and expectations of each participant. At this stage the roles for the lecturer and the student are finalised.
  3. Define scope: Agree the scope of the project. This stage focuses on the aims of the research and the research questions.
  4. Design laboratory exercises: This includes deciding on the number of laboratory sessions that are needed. At this stage the planning and development of the laboratory sessions are established.
  5. Conduct study: This stage is visited twice, first with pilot students who "work through" the seminar materials and provide feedback, and then with the enrolled students undertaking the unit who provide an evaluation of the unit at the end.
  6. Observe changes: Conduct interviews and obtain feedback. At this stage the feedback is collected and analysed.
  7. Exit: Leave project and complete the documentation, i.e., write the thesis.
  8. Reflect and record learning: Critical analysis of achievement which is an essential process of action research.
    Level 1 Learning: Learning associated with obtaining feedback from the pilot students. This may cause a re-design of the laboratory exercises.
    Level 2 Learning: Learning associated with obtaining feedback from the enrolled students.
    Level 3 Learning: Learning associated after completing the entire study and analysing the interviews conducted by the researcher.

The following were used to conduct the research and collect the data.

Results

Results will be reported in three sections as follows. Firstly, questionnaires were completed by all students in the class close to the completion of the part of the unit and in addition a number of students were selected to undergo a semi-structured interview by the research student. Secondly, a content analysis of the surveys and interviews was conducted and emergent themes identified. Thirdly, during the weekly laboratory sessions, feedback was directly obtained by observation and discussion with the students in situ.

Questionnaires

The overall feedback from the questionnaires was very positive indeed. One has to be somewhat wary of this kind of feedback (as a sole measure at any rate), partly because of the Hawthorne effect (Mayo, 1933) and partly because, regardless of assurances and precautions on anonymity, students may feel disinclined to be brutally honest. Nevertheless, the vast majority of what was stated and the consistency of it suggested that there was strong positive feeling about the laboratory exercises and how they supported the theoretical content. So, while the feedback was positive, due caution needs to be applied to a single source such as this. However, combined with the content analysis, the credibility of this source improves somewhat.

Content analysis

A content analysis (Miles and Huberman, 1994) was conducted on open ended questions in the semi-structured interviews and open-ended questions in the questionnaires. Examples of open-ended questions are 'what aspects of this unit did you like or dislike?' and do you have additional comments?'. The purpose of this analysis was to elicit themes that the students may have held that were not prompted by specific question probes. Six recurring themes emerged from the content analysis. These themes were thinking, applying, meeting study objectives, structuring of the course, solving complex problems and interaction. These themes are ordered here in terms of frequency of occurrence in the data.

'Thinking' was the label for those comments made by respondents that refer to some positive cognitive aspect relating to the laboratory exercises directly or the connection between the theoretical content and the laboratory material. Since the main research question concerned intellectual development and modelling, it was re-assuring to receive a high frequency of positive comments in this regard.

'Applying' refers to the emphasis placed on applying the ideas i.e. enabling the conceptual discussion to be taken through to a point where they can be implemented and simulated on a computer. Also, under this theme were comments that related to students finding that they could bring in outside knowledge and apply it to the problems given.

'Meeting study objectives' was the next most frequent theme. This was very much an holistic comment relating to the degree of satisfaction respondents felt about how this component supported and re-inforced their whole of unit study.

'Structuring of the course' was a label used to capture all suggestions for change or improvement with regard to mostly the laboratory sessions but also the unit itself. Details of these are discussed shortly.

'Solving complex problems' refers principally to statements about students' confidence in their new-found ability to embrace and solve complex problems of similar nature to the ones introduced in this unit.

'Interaction' was the last of the six main themes identified and refers to the feelings of involvement and participation felt by students throughout this whole research process. Yes, the Hawthorne effect is alive and well!

The fact that the students were framing the unit in the above terms suggests that they were conceptualising the benefits of the unit at a high level of abstraction. Rather than seeing the unit as 'easy' or 'hard work', the themes from the content analysis demonstrate a degree of intellectualising regarding the unit by the students which is consistent with the hypothesised research questions stated at the beginning of the paper.

Weekly feedback in situ

The nett effect of the weekly feedback in situ was to improve the quality of the laboratory material in terms of volume, format and organisation. Initially the exercises were ambitious even for the top students. It took some weeks before a 'house style' for these sessions emerged and lessons were learned (see reflections section below) concerning how to structure individual sessions and link them with previous and following sessions. After about three or four weeks however, the mechanics of creating and running laboratory sessions had settled down and was generally well accepted by the students and thus non-contentious in terms of feedback.

Reflections

While there was strong 'evidence' in terms of reported statements of the usefulness and value of the laboratory sessions by virtually all participants, it needs to be recognised that much of this is perceptual. It would be very difficult to justify these views in a positivistic manner. That said, in retrospect it is contended that this project facilitated a deeper understanding of systems concepts and ideas. Furthermore, as a consequence of these additions that students are able to model typical organisational situations better and arrive at better solutions and through that process they have, to some extent, developed intellectually and academically. These conclusions are based on having run the unit once without ithink and once with ithink and using the following reflective frames.

Overall performance in the unit was better that previously both in assignments and examination with a greater number of high achievers. Low achievers were breaking up problems into constituent parts better than their previous counterparts and while they were not particularly able in synthesising solutions this represented still a milestone for them in their intellectual development. In general most students were able to discuss their problems openly and how they were overcome and this represented a shift from the previous cohort in terms of how they framed their discussion (previously discussion was more 'matter of fact' reporting usually only the basic logic of the model). Another noticeable difference was the frequency in which students brought in outside knowledge to bear on a problem or solution. For example, one student brought in his knowledge of 'S curves', a standard manpower planning technique common in project management. Perhaps the most remarkable area was high achievers. Of course every class has them, so that in itself is not remarkable. However, it was clear that a number of students were working ahead of the laboratory sessions or picking up techniques that were not part of the planned syllabus. For example, a number of students presented in their assignment a control panel (Figure 5), something which had not been covered up till that time. A control panel is a much more user-friendly and often more powerful means of allowing a user explore 'what if' type questions. In Figure 5, the user can run and reset the model at the touch of a button and move the sliders thus allowing different values and assumptions to be run through the model. These high achievers were exploring this effectively independently.

Figure 5

Figure 5: An example of a control panel submitted by a student

In terms of operating the laboratory sessions, a lot of useful experience was accumulated. Students preferred not to start a brand new problem each week. They preferred to build on previous weeks. That said, it was found prudent to start each new exercise with a revision or even test of previous accomplishments and to provide a starting model (i.e. file) that represented where students should have reached previously (otherwise the weaker students would fall behind). Another lesson was in terms of the manner in which material should be introduced. Fairly quickly we discovered it was wise to start each session by introducing only basics of a concept and get the students to practise that before covering more advanced aspects of a concept.

They say two heads are better than one. It certainly was the case that students brought forth very practical suggestions for improvement which frankly were not identified at the outset. Suggestions included worked demonstrations of aspects of ithink in addition to the traditional 'laboratory exercise'. It was also suggested that feedback and discussion sessions be held the following week in class from the previous week's laboratory exercise.

References

ACS Certification Program (2003). Australian Computer Society Certification Program Website. [viewed 5 Nov 2003] http://www.acs.org.au/certification/index.cfm

Bose A. (2003). Using action research to design a systems thinking module. Unpublished honours thesis, Edith Cowan University, Perth, Western Australia.

Checkland, P. (1991). From framework through experience to learning: The essential nature of action research. In Information Systems Research: Contemporary Approaches and Emergent Traditions. North-Holland: Amsterdam, pp. 397-403.

Hart, E. and Bond, M. (1995). Action Research for Health and Social Care: A Guide to Practice. Buckingham, Open University Press.

High Performance Systems Inc. (n.d). Systems thinking. [viewed 5 Nov 2003 at http://www.hps-inc.com/MoreAboutSytemsThinking.htm, verified 19 Jun 2004 at http://www.iseesystems.com/(mjp534vyd54d50mczvlnu0nd)/Community/STArticles/SystemsThinking.aspx

Maani K and Cavana M. (2000). Systems thinking and modelling: Understanding change and complexity. Pearson Education: New Zealand.

Mayo, E. (1933). The human problems of an industrial civilization. New York: Macmillan. Ch.3.

Miles, M. B., and Huberman, A. M. (1994). Qualitative Data Analysis. Thousand Oaks, California: Sage.

Authors: Donald C McDermid and Anita A Bose
School of Computer and Information Science
Edith Cowan University, Joondalup, Perth, Western Australia
Email: d.mcdermid@ecu.edu.au

Please cite as: McDermid, D. C. and Bose, A. A. (2004). Using action research to embed systems thinking concepts in postgraduate IT teaching. In Seeking Educational Excellence. Proceedings of the 13th Annual Teaching Learning Forum, 9-10 February 2004. Perth: Murdoch University. http://lsn.curtin.edu.au/tlf/tlf2004/mcdermid.html


[ Proceedings Contents ] [ Teaching and Learning Forum 2004 ]
This URL: http://lsn.curtin.edu.au/tlf/tlf2004/mcdermid.html
Created 19 June 2004. Last revision: 19 June 2004.