Teaching and Learning Forum 2005 Home Page
Category: Research
Teaching and Learning Forum 2005 [ Refereed papers ]
Techniques for developing outcome statements

Claudia Amonini, Nicholas Letch and Cherilyn Randolph
The University of Western Australia

Following a Senate decision in 2002 that the University community should embrace the concept of outcomes based education, there is pressure to develop outcome statements at degree, major and unit levels. A project to develop outcomes statements for a major in the Business School, UWA, provides an opportunity to reflect on the curriculum.

Three stakeholders are involved - students, lecturers and employers - from whom it is appropriate to elicit data to support the writing of outcomes statements. Each group has its own view of what a student should be able to perform as a result of studying this major and each view is valid. The varied nature of the three groups means that different techniques are needed. The methodology presented here is generally applicable to any major in any faculty.


Introduction

The work described in this paper is part of a project to develop outcome statements for a major course of undergraduate study in the Business School at The University of Western Australia. The original plan was to hold a facilitated 'lock up' session for the lecturers when existing objectives would be transmuted to outcomes for each unit and coalesce to form outcomes for the major. Immediately lecturers questioned the difference between what they had already written and true outcomes except for sentence structure. We realised that focusing on lecturers belies the concept of outcomes based education in which outcomes must be meaningful to all partners in the learning process. A much deeper reflection on the curriculum for the major was required.

Essentially, there are other stakeholders, students being the most obvious. It could be argued that most students do not have the life experience needed to make a significant input to defining outcome statements. However, because they will be working with them, their aspirations, language, and viewpoint must be incorporated. Universities are increasingly vocationally oriented and students are awake to the need to satisfy the requirements of their future employers. The third group of stakeholders therefore are employers.

Direct inquiry of any of these stakeholders is likely to result in more time spent debating outcomes based education than data collection. We opted to use other techniques, each chosen to be most appropriate for the type of stakeholder being targeted. This paper describes the techniques we have used. Not all of the work is complete but we have some results to present.

Background

In 2001 the Academic Council resolved that outcomes based teaching would be introduced into the University in the academic year 2004, ahead of the 2005 intake of new students whose high school education was structured around outcomes. All faculties within the University were encouraged to implement the resolution, and research grants, from the Teaching and Learning Development Fund, were made available to stimulate and promote the adoption of outcomes based education (CATL, undated). The work described in this paper was supported by one of these grants[1].

The Business School at The University of Western Australia awards two undergraduate degrees, Bachelor of Commerce and Bachelor of Economics. Students of both degrees must satisfy the requirements for at least one major stream of study and most students graduate with two or three majors. A major in Information Management (IM) is popular amongst students with an interest in computer technology (they typically pair it with another Business major) but is also chosen by students of accounting, finance, management and economics who see it as a valuable complement to their primary interest.

The name, Information Management, was chosen to reflect the broad range of topics that could be chosen by students, from the use of statistics to support decision making, through to systems development and social implications of technology adoption. However, Information Management is not a widely used term and does not easily translate into employment prospects as represented by job advertisements in newspapers. Although the academic staff (particularly those involved in teaching the units that make up the major) believe that this is a critically important subject to study, it is believed that this view is not held by prospective students. In essence, there is anecdotal evidence that suggests students do not recognise how their CVs will be improved by undertaking a major in Information Management.

The Academic Council resolution was recognised as an opportunity to review what is taught in the Information Management major and to document this in terms of the desired outcomes - sometimes referred to as 'exit outcomes' (Spady, 1988). The intention was to use the IM major outcome statement as a vehicle to establish outcome statements and competency scales for each of the units. Outcome statements at both the major and unit level, would provide both staff and students with a much clearer idea of what graduates will be able to do as a result of completing an IM major.

In order to include the perspectives of major stakeholders in the major we approached the identification of exit outcomes from several directions. Unit lecturers have perhaps the most tangible stake in the development of teaching and learning outcomes. Curriculum should be developed from outcomes not vice versa, but with units and teaching in place it would be naïve to adopt a green fields approach. The first line of attack, therefore, was to explore the content and objectives for each unit. We started by asking the lecturers in charge what their teaching objectives were.

The second stakeholder group that is likely to have a strong influence on outcomes are the potential employers of our graduates. Research has previously shown that there is frequently a gap between employer perceptions of graduate capabilities and the actual competencies of graduates (Trauth, Farwell, & Lee, 1993; Tye, Poon, & Burn, 1995). While we do not intend to replicate large scale studies of employer perceptions, we do feel that it is necessary to develop outcomes in line with what industry and businesses are looking for.

It could be argued that students are not sufficiently informed with respect to which outcomes are of value to them for their views to be a significant factor. In contrast, we believe that to be effective, the outcome statements we produce must reflect the criteria that undergraduates use to choose their degree and majors. At the very least, the expression of the outcome statements must be easily accessible to young people, particularly in language and emphasis. We therefore also intend to ask current students about their attitudes towards, and expectations from the IM major.

While the units within the major are not specifically driven by curricula recommended by professional and teaching bodies (eg ACM, AIS, ACS, etc), a review of recommendations provides a cross check and confirmation on what is appropriate (ACM, AIS & AITP, 2002; ACS, 1997; IRMA & DAMA, 1997).

This project is designed to stop at the specification of unit outcomes. It is expected that each lecturer in charge of a unit will take responsibility for devising appropriate assessment scales to match their unit outcomes. This is consistent with the 'designing back' principle advocated by Dalziell and Gourvenec (2003), also within The University of Western Australia.

Outcomes based education

Outcome Based Education means clearly focusing and organising everything in an educational system around what is essential for all students to be able to do successfully at the end of their learning experiences. This means starting with a clear picture of what is important for students to be able to do, then organising the curriculum, instruction, and assessment to make sure this learning ultimately happens (Spady, 1994, p1, in Killen, 2000).
The sentiments in this quote are both laudable and achievable but require a different mind view of teaching and learning in higher education. Listing objectives for units is standard practice at this university. Relating those objectives to what students can do after graduation, as a result of taking the unit (outcomes) is the next step. Ensuring that the outcomes from each unit are compatible and complementary with those of the other units in the major is further progression towards a coherent and effective educational system. Applying outcomes to instruction and assessment is outside the scope of our project.

What outcomes based education means at our university can be assessed by reading what is available on the website created by the staff development group (CATL, undated).

Research methodology

Given that we want to include the perspectives of major stakeholders in the development of outcomes, we have chosen a multi method approach to collect data on the current and desired outcomes for the IM major, whereby each method selected was designed to suit the targeted sample and elicit responses. While we see the perspective of each stakeholder group as contributing to the development of outcomes, our objectives and subsequent data collection approach varied across the groups. These techniques are summarised in Table 1 and detailed below.

Table 1: Data collection techniques for relevant stakeholder groups

Stakeholder
group
ObjectiveData collection technique
LecturersIdentify and explore perceptions of what is currently taught and how that contributes to the major.Semi-structured interviews with lecturers using RepGrid technique.
EmployersIdentify the outcomes expected by prospective employers
  1. Survey of IS-related job advertisements
  2. In depth interviews with employers and recruitment agents
StudentsIdentify and explore student expectations of outcomes and their reasons for choosing/not choosing the major.Focus groups with current undergraduate students
  1. Information Management major
  2. Other majors

Lecturers

Each lecturer in charge of units within the Information management major has quite well defined aims and objectives for the unit(s) they coordinate. Although changes to the curriculum may arise as a result of this project, that is not a primary goal. Rather, the project needs to capture what is currently being taught in each unit and why so that a single coherent account of the outcomes of the major can be developed. However, a list of topics like the contents of a textbook does not progress our study towards articulating outcomes. Essentially we needed each lecturer-in-charge to explain the rationale for their unit within the context of the objectives of Information Management major and with respect to the other units in the major. We needed to deeply probe how they understand what they teach, their motivations for doing so, and what they believe that students learn. To facilitate these probing interviews, we made use of a knowledge elicitation technique that has been widely used in psychology - namely, the RepGrid technique.

The RepGrid methodology is described by Kelly (1961) who developed the technique as part of his Personal Construct Theory, which is based around the concept that individuals have a personal view of the world that the clinical psychologist is attempting to reveal [Bannister, 1968 #525]. It is necessary to abstract and generalise personal constructs from what the target does, says, achieves. One useful technique is to explore what the target can be perceived to be and also by what it is not, which is the basis of the RepGrid test [Bannister, 1968 #525]. The requirements on such a test that is to be used in a clinical setting such as the need for 'useable terms', '... reveal those resources ... which might otherwise be overlooked ...' and '... may subsequently be tested and put to use' (Kelly, 1955, in Bannister & Mair, 1968, p.39) particularly resonate with our project.

There are similarities between the psychologist's appraisal of personal constructs of an individual and trying to characterise the Information Management major. Our lecturers need to abstract from the literal content of their unit to recognise the unit constructs, which can then form the basis of outcome statements. To give an example, a unit that includes lectures, tutorials and assessment on the use of presentation skills and PowerPoint is contributing to building communication skills. The relevant outcome is the ability to communicate in large groups. Generic skills such as this are the easiest to articulate. Abstracting a unit that teaches students to draw data flow diagrams is less obvious because the outcome from learning these techniques may be communication skills, software manipulation, work process analysis or all three and there is a flow-on effect to the evaluation of corporate based information systems taught in another unit.

Briefly, the grid in the name refers to a table with the constructs used by individuals to characterise the 'elements' of interest along one axis and the rating made by each individual about that element on the other. In our case the elements of interest are the units in the major, the individuals are the lecturers and the constructs are what we are trying to elicit in the first instance.

RepGrid has been widely used in a number of disciplines outside psychology including Information Systems (Hunter & Beck, 2000; Tan & Hunter, 2002), but its use in the development of outcome statements for majors in tertiary education is new. We applied the technique in two rounds of structured interviews. In the first we were focusing on extracting the constructs in structured interviews. We used the recommended approach of asking the interviewee to consider the units in groups of three and to identify aspects of commonality between two of units that is not shared by the third - the method of triads (Bannister & Mair, 1968; Fransella & Bannister, 1977). Analysis of these interviews reveals the constructs and these will form the basis of the outcome statements.

In the second round of interviews we will attempt to score each unit against the outcome statements as a way of confirming the constructs and reviewing the amount of overlap or complementary material in the units. This should also provide a first step towards building assessment scales.

Employers

There are some difficulties associated with canvassing the views of employers. First, the type of employment into which a new graduate with an IM major goes can range from technical (eg programmer if the degree is Computer Science), to management (Commerce degree with HR or General Management), and to finance (Commerce degree with Accounting or Finance). The value of the IM major sometimes does not emerge until some industry experience has been gained and the graduate can move into such positions as management consultant or business analyst.

Secondly, many UWA graduates take their first job outside the state. The project budget did not run to interviewing employers interstate or overseas.

An additional difficulty is that, in material aimed at new graduates, (job ads, flyers in the University careers office), employers do not articulate their needs beyond task specific (eg knowledge of tax laws if the job is tax accountant) and the more obvious generic skills like team work.

We adopted two approaches to overcome these difficulties. In order to gain an initial appreciation of the types of skills that are associated with job titles, we sampled job advertisements over a two week period and analysed the stated requirements. To expand on our understanding of employer requirements, we also conducted in depth interviews with human resource consultants and executive recruitment agents. These interviews confirmed that as intermediaries between universities and employers, the employment agents had a wider view of what skills, knowledge and experiences are valued both by employers of new graduates as well as those required for later career moves. They also had some influence on employers to help them to recognise the value of less obviously relevant knowledge.

Job ads review
Research into the skill requirements of IT professionals in Australia has previously used job advertisements as an indicator (Young, 1996). In our study, we wanted to examine the titles and skill requirements of published job advertisements as an initial assessment of whether the types of capabilities that we believed our graduates possess are actually in demand in the workplace.

The student cohort in the Information Management major is typically a mix of Western Australian school leavers and international fee paying students (primarily from Singapore). To reflect this mix, job advertisements from the local newspaper (The West Australian), a national newspaper (The Australian) and a Singaporean daily newspaper (The Straits Times) were analysed. In addition, the job advertisements posted on Seek.com were also examined over the same period.

The analysis involved searching for job titles that appeared to represent the softer, less technical skills that are associated with IT professionals (eg. information systems analyst, business analyst, IT support manager).

In depth interviews
Some disciplines have an effective association that provides a conduit between employers and universities with respect to the content of undergraduate courses (eg CPA for Accountants). Although there is an Australian Computer Society (ACS), it does not have the influence of a compulsory professional body as membership is not a requirement for employment in related industries. As an alternative we chose to explore the university graduate careers service, and the Government online Gradlink site[2]. We also approached recruitment agents and employers directly through in depth interviews.

Questions for the interviews included:

  1. For what are the employers of new graduates looking? What looks good on their CV?
  2. What are the characteristics of a new graduate who can easily find a job?
  3. Which of the generic skills are important and how can an applicant demonstrate they have those skills?
  4. What IT related subject knowledge would be expected of accountants, marketing graduates, finance majors, trainee HR managers?
  5. Given a list of the study outcomes for the IM major, which would be the most valuable?
  6. What other factors affect the success of new graduates in getting their first job eg dress, manner, cultural background, casual employment, sports etc.

Students

As mentioned above, it was considered important to understand students' expectations of outcomes from the IM major and each of its units. A survey was deemed unsuitable given a list of outcomes or attitudinal statements to measure had not already been identified. Adapting items from any other similar study also seemed inappropriate as it was felt that educational and other outcomes are very unit specific.

Essentially, this study required exploratory qualitative research to gain an understanding of the attitudes and perceptions of the target population prior to any statistically testing with a survey.

Focus group discussions with undergraduate students were used to explore salient attributes when selecting a major as well as the specific outcome expectations from undertaking an IM major. The focus group is a valuable alternative to an in depth interview as it can often generate more information due to the additional benefits from the interaction and dynamics of the group (Malhotra, 1993). The free flow discussion can trigger thoughts and feelings that are shared with the group. Focus groups are widely used in research to gather a range of opinions and experiences (Malhotra, 1993; Cavana et al 2004). Focus group participants were recruited via email. The participants were randomly selected from the following populations:

Homogenous groups were stratified according gender to explore potential gender specific reasons and requirements of the major. Similarly some groups will be populated with only domestic or only overseas students to investigate any cultural dimensions.

A moderator's outline was developed for use during the group discussions, and was divided into the following sections:

  1. Drivers of major choice - important attributes
  2. Decision-making process - IM major
  3. Value of IM major
  4. Expectations and perceived benefits of IM major
It should be noted that other researchers have investigated the choice criteria for university entrance, and found some common salient factors such as standard of education, cost, campus appearance, peer and parental influence (Joseph & Joseph, 1998). However, this research is different in that it is dealing with a major stream of study (not university selection) which is chosen at the end of the first year at university. Thus, it was decided that the focus group facilitators should approach the investigation with no preconceived notions and encourage a far ranging discussion.

Results

Lecturers

Some lecturers had already been experimenting with outcome statements for their units. Most other lecturers were routinely specifying 'objectives' to their students in the unit outline at the beginning of semester. The interviews provided further detail to the project team who were able to code the interviews (using NVivo) according to common themes that emerged. Taking unit outlines and the interview coding structure a first pass at a set of outcomes for the whole major was identified. It was recognised that this was incomplete and biased towards the units with which the project team members were most familiar.

Clearly this early draft needed verifying. The University has a set of outcome statements embedded in its Academic Structures Plan (Haskell, 2001) . We took these and tabulated them against the outcome statements for the major and the units. (Outcome statements for the two degrees are not yet available.) Now we were in a position to present the framework to the rest of the teaching group. They had each already given time for the interviews so it was important to organise a meeting where exchange of views would be effective.

Employers

While the job advertisements analysis was intended to merely provide some orientation regarding job titles and skills relevant to graduates of the Information Management major, it did, to some extent, confirm our suspicion that there is no clearly defined job title or role for which Information Management graduates could identify and apply. Approximately fifty job advertisements were identified as being relevant and these fell into the categories shown below.

Table 2: Possible job titles

Position titleNumber
reviewed
Business Analyst15
Systems Analyst8
IT Administrator/IT Manager8
Project Support Officer/ Project Manager7
Data Analyst5
Process Analyst2
Business Development Manager3

For most positions, analytical capabilities were deemed essential and several positions required specific technical expertise. All advertised positions required previous experience making them unsuitable for most new graduates. Some results from in-depth interviews include:

Students

The focus groups described above were scheduled for October, in the lull at the end of semester, before exams. The results will be summarised and presented at the Forum.

Conclusions

Creation of outcome statements that are helpful to students, lecturers and employers requires data from all three sources. This paper has described the methodology adopted by a project team to elicit the outcomes for a major in Information Management. Several different techniques have been used and these have been described along with the reasons for choosing them. This project is on going, more data is being collected and the outcomes are still evolving.

References

ACM, AIS & AITP. (2002). IS '02 Model Curriculum Guidelines for Undergraduate Degree Programs in Information Systems. Association for Information Technology Professionals.

ACS (1997). The ACS Core Body of Knowledge for Information Technology Professionals. Sydney: Australian Computer Society.

Bannister, D. & Mair, J. M. M. (1968). The Evaluation of Personal Constructs. London: Academic Press.

CATL (Centre for the Advancement of Teaching and Learning) (undated). Implementing Outcomes Based Education @ UWA. http://www.catl.osds.uwa.edu.au/obe

Dalziell, T. & Gourvenec, S. (2003). Partnerships in outcomes based education. In Partners in Learning. Proceedings of the 12th Annual Teaching Learning Forum, 11-12 February. Perth: Edith Cowan University. http://www.ecu.edu.au/conferences/tlf/2003/pub/pdf/23_Dalziell_TanyaGourvenec_Susan.pdf

Fransella, F. & Bannister, D. (1977). A Manual for Repertory Grid Technique. New York: Academic Press.

Haskell, D. (2001). Academic Structures Plan. [retrieved 14 Oct 2004] http://www.admin.uwa.edu.au/reg/sec/acadstructures/ASWP-Rept-June.pdf

Hunter, M. G. & Beck, J. E. (2000). Using repertory grids to conduct cross cultural information systems research. Information Systems Research, 11(1), 93-101.

IRMA & DAMA (1997). IRM Curriculum Model (Draft). Hershey, PA: Information Resource Management Association.

Joseph, M. & Joseph, B. (1998). Identifying needs of potential students in tertiary education for strategy development. Quality Assurance in Education, 6(2), p90-.

Kelly, G. A. (1955). The Psychology of Personal Constructs (Vol. 1). New York: W.W. Norton & Company.

Killen, R. (2000). Outcomes-based education: Principles and possibilities. [retrieved 17 Jan 2003 from http://www.schools.nt.edu.au/curricbr/cf/outcomefocus/killen_paper.pdf; verified 24 Jan 2005 at http://www.acel.org.au/affiliates/nsw/conference01/ts_1.html

Spady, W. (1988). Organizing for results: The basis of authentic restructuring and reform. Educational Leadership, 46(2), 4-8.

Spady, W. (1994). Outcome-based education: Critical issues and answers. Arlington, V.A: American Association of School Administrators.

Tan, F. B. & Hunter, M. G. (2002). The Repertory Grid technique: A method for the study of cognition in information systems. MIS Quarterly, 26(1), 39-57.

Trauth, E. M., Farwell, D. W. & Lee, D. (1993). The IS Expectation Gap: Industry Expectations Versus Academic Preparation. MIS Quarterly, 1993(Sep), 293-303.

Tye, E. M. W., Poon, R. S. K. & Burn, J. M. (1995). Information systems skills: Achieving alignment between the curriculum and the needs of IS professionals in the future. Data Base Advances, 26(4), 47-61.

Young, J. (1996). The scope of qualifications, experience and skills Australian organisations require of system analysts and programmers. University of Tasmania, Hobart. Paper presented at the 7th Australasian Conference on Information Systems, University of Tasmania, Hobart.

Endnotes

  1. 2002 OBE Scheme Grant Recipients. http://www.admin.uwa.edu.au/reg/sec/obe/obe-2002-recipients.pdf
  2. Australian Job Search: Graduate links. http://jobsearch.gov.au/government/Gradlink.aspx?WHCode=5&TextOnly=0

Contact author: Mrs Cherilyn Randolph
School of Economics and Commerce
The University of Western Australia
M261, 35 Stirling Highway, Crawley, Western Australia 6009
Phone: +618 6488 1829 Fax: +618 6488 1055 Email: crandolp@biz.uwa.edu.au

Please cite as: Amonini, C., Letch, N. and Randolph, C. (2005). Techniques for developing outcome statements. In The Reflective Practitioner. Proceedings of the 14th Annual Teaching Learning Forum, 3-4 February 2005. Perth: Murdoch University. http://lsn.curtin.edu.au/tlf/tlf2005/refereed/amonini.html

Copyright 2005 Claudia Amonini, Nicholas Letch and Cherilyn Randolph. The authors assign to the TL Forum and not for profit educational institutions a non-exclusive licence to reproduce this article for personal use or for institutional teaching and learning purposes, in any format (including website mirrors), provided that the article is used and cited in accordance with the usual academic conventions.


[ Refereed papers ] [ Contents - All Presentations ] [ Home Page ]
This URL: http://lsn.curtin.edu.au/tlf/tlf2005/refereed/amonini.html
Created 23 Jan 2005. Last revision: 23 Jan 2005.