Teaching and Learning Forum 97 [ Contents ]

Teaching literacy in intensive science degrees:
A formal evaluation

Stephen R. Kessell
Science and Mathematics Education Centre
and School of Computing
Curtin University of Technology

Introduction

Most undergraduates in intensive science degrees have no interest in pursuing a broad, balanced programme of study:
Their desire is to "get in, get trained, get out and get a good job". We do a pretty good job of meeting these expectations. But are the students getting an "education" or just very good training? (Kessell 1992: p. 8)

Similar laments were heard from Dr Don Anderson (ANU) in the Senate Standing Committee Report Priorities for Reform in Higher Education (Australia 1990):

The undergraduate curriculum, particularly in economics and the science-based professions, is deficient in that it is producing highly trained, highly competent technicians who are undereducated in the traditional sense of the word. They are not familiar with the society in which they are going to practise ... They do not have good critical capacities and they are not good communicators (p. 1) ... The majority of graduates in medicine, engineering and other science-based professional courses ... are cultural illiterates (p 4).
Recent national employer surveys found that oral and written communications skills were rated as the most important requirement of graduates (Bate and Sharpe 1990; Australia 1992; Australian Association of Graduate Employers 1993; Institute of Chartered Accountants 1994). The Discipline Review of Computing Studies and Information Science Education (Hudson 1992) reports:
The Committee received widespread expressions of concern from employers that the educational preparation of computing students, whilst producing good technical skills, is not always adequate to meet important workplace requirements of communication and interpersonal skills (p. 187) ... A major problem, about which concern was expressed by an overwhelming majority of the employer community, was the level of English language ability and interpersonal skills, which were considered below the standards expected of professionals. IT (Information Technology) professionals (also) need to have an understanding of the cultural, social, legal and ethical issues inherent in these professional areas. There is a need to understand where the IT disciplines have come from and where they are heading and to appreciate philosophical questions and issues relating to the social impact of computing. IT professionals also must be aware of basic legal issues, particularly in the area of intellectual property, and understand ethical values that should be a basis for commercial activity (p. 189).
Sadly, many science faculties have demonstrated little or no interest in addressing these problems (Kessell 1996a, 1996b).

The Solution Attempted

Curtin's School of Computing established three new units, over the period 1988-1992, to help rectify the perceived problem of literacy and breadth (Kessell and Kessell 1993; Kessell 1996a, 1996b). These included: The first year unit also included a mandatory one-hour weekly lecture dealing with the broad issue of "how to survive university".

Concurrently, the School introduced an intensive, interventionist, and often "intrusive", programme of undergraduate advising and counselling, for which every student failing subjects, or otherwise deemed to be at risk, was contacted by the course controller immediately, and required to engage in appropriate and regular study skills and counselling sessions until the problem was resolved (Kessell 1995, 1996a, 1996b).

The new teaching initiatives addressed major issues and goals contained in Curtin's (1995) Strategic Plan for Teaching and Learning. They also dealt with issues subsequently raised by Latchem, Parker and Weir (1995), and followed what is commonly referred to as a "communications-in-context" approach, whereby communication skills are taught in the context of disciplinary studies (in contrast to the teaching of "context-independent" literacy units (Lee 1991; Cowen 1993)). The intensive counselling programme was modelled on the one practised at Amherst College (Massachusetts) (personal experience; Professor R. Rosbottom, Dean of the Faculty, Amherst College, personal communication, November 1991) and several other American liberal arts colleges; it dealt specifically with several of the major issues of "university survival" recently raised by McInnis, James and McNaught (1995) and Fairnie (1996). It was offered as a "first point of contact" for students experiencing personal and/or academic difficulties, and frequently referred students to specialised services available on- and/or off-campus. Further descriptions of these teaching and counselling initiatives are provided in Kessell (1992, 1995, 1996a, 1996b) and Kessell and Kessell (1993).

The Need To Evaluate The Solution

Despite extremely favourable feedback from students, colleagues and employers, on regular formative evaluations, it was obvious that a more rigorous and formal summative evaluation was desirable, especially in light of proposed major cuts to these programmes. During first semester of 1996, I completed an MEd at Curtin; the thesis was a formal evaluation of these programmes (Kessell 1996b). The remainder of this paper summarises the conduct and results of that evaluation.

The major feature of the evaluation was the use of multiple methods to collect some quantitative, but primarily qualitative, information from a range of stakeholders, including current students, graduates, employers, business leaders, and teaching and research staff (Fraser 1984, 1985; Fraser and Smith 1980a, 1980b; Smith and Fraser 1980; Williamson and others 1987; Sanders 1994; Armstrong and Conrad 1995). The results were interpreted in the context of current "best practice" approaches, to the teaching of literacy, including "communications-in-context" (Latchem, Parker and Weir 1995) and "writing across the curriculum" (Cowen 1993) models, the introduction of professional practice issues (Hudson 1992), and the provision of effective student counselling, preparation for study, problem resolution and basic pastoral care services (Fairnie 1996; McInnis and others 1995; Elliott 1995). In particular, the results were evaluated with respect to the specific recommendations regarding communication skills and breadth of education of the Discipline Review of Computing Studies and Information Sciences Education (Hudson 1992).

The specific evaluation methods were applied to three generic areas, including student performance, perceptions and feedback from students and graduates, and perceptions and feedback from employers and academic staff.

Measures of student performance included three major assessment tools. Firstly, comparison was made between samples of students' entry and exit written skills in each literacy unit (to assess the units' success). Secondly, comparisons were made between students' entry written skills (estimated via our written diagnostic test) and their tertiary entry literacy marks (TEE English, TEE English Literature, or equivalent results for interstate or international students), to determine both the utility of entry scores for reflecting actual written literacy levels, and to identify those students in need of further preparation. Thirdly, an analysis of graduates' abilities to meet professional writing and public speaking tasks required in the workforce and/or for higher study was conducted via the assessment of both their ability to perform appropriate "real world" written and verbal tasks and feedback from both graduates and their employers (discussed below). Measures of perceptions and feedback from students and graduates included six assessment tools. The first was responses of students to Student Appraisals of Teaching (SAT), which were administered every time these units were taught. [The SAT was used across Curtin University for end of semester appraisal of teaching; it included six quality of teaching questions on a Likert scale.] Secondly, responses of 1993, 1994 and 1995 graduates to the Course Experience Questionnaire (CEQ) were analysed to determine graduates' response to the statement:

While the Australian Vice-Chancellors' Committee recommends against making inter-course and inter-university comparisons (Johnson, Ainley and Long 1996), the results show that a large majority of the School of Computing's BSc graduates agree with the statement (Latchem, Parker and Weir 1995; Ainley and Long 1995; Johnson, Ainley and Long 1996). Thirdly, formal formative evaluations were conducted during 1994 and 1995, in which students responded to questionnaires including both structured (Likert scale) and open-ended questions addressing student satisfaction with respect to content, teaching quality and perceived benefits. Fourthly, during 1995, six formal two-hour workshops were conducted in which all third year students, in small groups, were asked to design the "ideal" science degree. In 1996, a detailed, open-ended questionnaire, dealing with the quality, content and utility of these units, perceptions of preparedness for the written and verbal requirements of the workplace and/or higher study, the quality of the student counselling programme, and overall quality of teaching, was sent to a demographically representative cross-section of 30 final year students and recent graduates. Lastly, follow-up interviews were conducted with a substantial subset of these and other students and graduates.

Measures of perceptions and feedback from employers and academic staff included five major assessment instruments. Firstly, a detailed, open-ended questionnaire was sent to a representative cross-section of Perth employers and business leaders, including the largest employers of our graduates (addressing perceived strengths, weaknesses, and needs with respect to graduates' communication skills). Secondly, feedback from several community business leaders who have participated in the Technical Writing and Project Preparation unit was obtained via a detailed questionnaire on the unit's merits and potential improvements. Thirdly, feedback and suggestions from a range of professionals, at my School's and other Advisory Board and Board of Study meetings (the Advisory Board members are professionals external to the university), and several other employer forums, were recorded. Fourthly, follow-up interviews were conducted with a subset of these employers and business leaders. Lastly, detailed interviews with and questionnaires from Curtin academic staff were conducted by Fiocco in 1994 (Fiocco 1996a, 1996b); I also conducted interviews with a range of academic staff from Curtin and other Australian tertiary institutions.

Taken together, these three sets of appraisal tools attempted to determine how well the School of Computing's initiatives met the requirements and expectations of students, graduates, employers and the profession.

Overview of the Results

Student Performance

Diagnostic testing of more than 1300 first year students revealed very little correlation between their tertiary entrance (TEE) English or English Literature marks and their ability to meet basic tertiary writing requirements. At the completion of the first semester writing unit (English for Technical Communication), about 80 % of the students demonstrated substantial improvement in their written work; this was most apparent in the structure and organisation of their reports and essays, the clarity of their expression, and their improved sentence construction, use of grammar, and punctuation. Of the remaining (perhaps 20 %) who did not show improvement, it appears that roughly half of this group possessed the required skills upon entering university; the other half still did not possess written skills at a level appropriate for first year university students. Since 1993, when all students who failed the diagnostic test have been required to complete remedial units prior to enrolling in the standard first year unit, the proportion showing little improvement over the course of first year dropped to approximately five percent.

The situation is somewhat different in the third year Technical Writing and Project Preparation unit. Detailed discussions between the lecturers and the principal tutors indicated that perhaps 20 percent of the students who enrolled in this mandatory unit had both written and verbal skills appropriate to their profession and/or higher study (however, many of these better students still lacked confidence and many of the finer points of technical writing and public speaking -- they also lacked experience of specific tasks, including professional resumes, job interviews, sales proposals, tenders, critical reviews, etc.). At the completion of the semester, perhaps another 20 percent of these third year students still did not possess the written and verbal skills that are deemed necessary for their intended profession (many, but by no means all, of these were non-native speakers of English). For the remaining 60 percent or so, Technical Writing and Project Preparation provided the transition from a skill level appropriate to meeting university requirements to a skill level appropriate to their profession and/or higher study.

Graduates' ability to meet professional literacy requirements was judged by surveys of both graduates and their employers, and is discussed below.

Perceptions and feedback from students and graduates

Results included Student Appraisals of Teaching and written formative evaluations from approximately 2000 students enrolled in these units, Course Experience Questionnaires completed by 129 graduates, detailed written evaluations from 26 graduates and final year students, and interviews with more than 100 students and graduates.

The results were extremely supportive of the conduct, content and success of the three units. For example, while very few first year students said they would have enrolled in English for Technical Communication had it not been a mandatory subject, the overwhelming majority were glad to have completed the unit and felt that it had improved their writing skills significantly. The "group dynamics", "interpersonal communications", and "how to survive uni" sessions were deemed to be the most valuable parts of the unit.

Third year students found Technical Writing and Project Preparation to be "a necessary ingredient in my preparation for the professional workforce". The mock job interviews, how to write a tender, and sessions in intellectual property, ethics and business practice were the most popular. Specific comments included:

The lecture was a good insight into what the workforce expects.

The lecturer gave helpful information and spoke from personal experience.

I am more confident about going on to further study because of this unit.

It seems obvious to me that if my literacy skills had not been lifted (by this unit), the documents and presentations I produce regularly (in my employment) would be sub-standard.

If this unit was abolished from the degree, the "quality" of students graduating in 3 years would drop by 100 %.

In my honest opinion, the literacy units formed the most useful component of the degree. If it weren't for these units, completing the research component of the honours degree would have been very difficult. Working on a research project has made me realise that literacy is imperative for getting the right message across.

The most rewarding responses came from students taking the elective Information Technology and Society unit. Frequent comments included "classes were always interesting", "it made me think", "my outlook changed", "it was so much more than rote learning", and "it was the most educational unit that I studied at university". Over 95 % of the students objected strongly to the proposed abolition of the unit.

Student Appraisals of Teaching results were very high for all three units (for example, Information Technology and Society results averaged 4.8 out of a possible 5.0 over a four-year period). The overwhelming majority of respondents noted a positive relationship between my serving as the undergraduate course controller and lecturing units that dealt with communications, breadth of study, and "becoming a professional". The "design the ideal science degree" workshops strongly reinforced students' desire for more literacy skills, breadth of study, and professional practice material in their degrees.

Many students, especially during interviews, noted the wide disparity of teaching and presentation skills, and interest in students, within our faculty. Responses included:

Too many of our lecturers couldn't teach a fish to swim.

Most lecturers are just too stupid to realise that they are not purely "information handybanks".

All students benefit by enthusiasm shown by lecturers.

The mind can only absorb what the seat can endure.

A lecturer needs to be more than an overhead project that can talk.

Perceptions and feedback from employers

The feedback from local employers strongly reinforced the views expressed by national employer surveys, the IT industry review, and our own graduates. Participants included most of the major employers of our graduates, as well as several large employers of tertiary graduates generally, in Western Australia.

All stressed the importance of communication skills and professional literacy, and most suggested that graduates' abilities in this area generally were inadequate. Mr Ron Hutchinson, Managing Director of ESRI - Australia Pty Limited (one of the largest employers of our GIS graduates), noted:

Communication skills are very important to us as our staff are required to not only be GIS technology competent but also able to perform marketing, understand contracts and business law. We have no place for backroom technocrats who cannot present their knowledge, understand user needs (listen) and develop a good cost benefit report. In evaluating potential employees, we put high value on communication, presentation and general knowledge ... Hence, the job does not always go to the graduate with the highest "computing" marks ... Generally, we would find about 80% of applicants can do the job technically but only about 20% (or less) have the communication and presentation skills.
Mr Michael Chaney, Managing Director of Wesfarmers, responded:
Well developed communication skills are of critical and growing importance to employers. Our experience with graduates is variable. On the whole, however, written communication skills tend to be underdeveloped -- in particular, there is too little knowledge of grammar.
Mr Ross McLean, Deputy Chief Executive of the Western Australia Chamber of Commerce and Industry, noted:
I wish to confirm in the strongest possible terms that well-developed communication skills are very important to employers of recent university graduates. In my view, most recent graduates do not possess such skills at an appropriate level ... My staff are required to prepare credible, relevant, and professional public policy papers ... I certainly do not expect to spend my time double checking the presentation aspects of my staff's work. This might sound old-fashioned, but it is a very important element in our assessment of job applicants.
Most employers also stated that the teaching of ethics and business practices was highly desirable. A prominent Perth barrister noted:
I am very much in favour of the teaching of ethics relevant to any occupation ... Although the teaching of prescriptive rules designed to prevent wrong-doing has its place, it is more important that ethics be taught in the context of stressing "doing right" rather than "avoiding evil". A good quality ethics course is highly desirable.
Interviews with academic staff, both at Curtin and seven other Australian universities, reinforced the employers' views.

The Outlook

The detailed feedback from a range of stakeholders suggested that the teaching of literacy, professional practice issues including ethics, and the impacts of technology of society are essential elements of a tertiary science degree; the programme described here was deemed to be highly successful by all stakeholders (Reid 1996; Kessell 1996b).

Despite the enthusiastic reception of our attempts to provide both this "breadth of study", as well as individual and empathetic counselling, to our undergraduates, much of this programme is being dismantled in 1997. It appears that a "research is more important than undergraduate teaching" attitude, more than the tertiary financial crisis, is responsible.

It is my view that science faculties can ignore literacy, breadth of study, ethical issues, and the impacts of science and technology, only at their peril.

References

Ainley, J. and M. Long (1995). The 1994 Course Experience Questionnaire. Graduate Careers Council of Australia, Parkville (Victoria).

Armstrong, J. and L. Conrad (1995). Subject evaluation: A resource book for improving learning and teaching. Griffith (Queensland): Griffith Institute for Higher Education.

Australia (1990) Senate Standing Committee on Employment, Education and Training: Priorities for reform In higher education. Canberra: AGPS.

Australia (1992). National Board of Employment, Education and Training (NBEET): Skills required of graduates: One test of quality in Australian higher education. Higher Education Commission Report No. 20. Canberra: AGPS.

Australian Association of Graduate Employers (1993). National survey of graduate employers. Sydney: AAGE.

Bate, D. and P. Sharpe (1990). Student writer's handbook. Sydney: Harcourt Brace and Jovanovich.

Cowen, K. (1993). Responding to the writing crisis in universities: Writing across the curriculum. Higher Education Research and Development Society of Australasia Conference, Brisbane.

Curtin University of Technology (1995). Strategic plan for teaching and learning: Summary. Perth: Curtin University of Technology.

Elliott, J. (1995). Voluntary study skills seminars: Why they miss the target. Perth: Curtin University of Technology (manuscript).

Fairnie, I. (1996). The first year experience. Report to University Academic Board (Document UAB 36/96), Curtin University of Technology, Perth, 26 April.

Fiocco, M. (1996a). Tertiary literacy: Case study research into the literacy policies, definitions and practices of four disciplines within a university. Conference on Tertiary Literacy, Victoria University of Technology, Melbourne, 15-16 March.

Fiocco, M. (1996b). Tertiary literacy: Definition and practice within one university. What is the relationship to language theory? Higher Education Research and Development Society of Australasia Conference, Perth: 239-244.

Fraser, B.J. (1984). Directions in curriculum evaluation. Studies in Educational Evaluation, 10: 125-134.

Fraser, B.J. (1985). Combining quantitative and qualitative methods: Evaluation of High School Education Law Project (HELP). In: B.J. Fraser, Case Studies in Curriculum Evaluation. Perth: West Australian Social Science Education Consortium (W.A.I.T.).

Fraser, B.J. and D.L. Smith (1980a). Evaluation of development and dissemination activities of the High School Education Law Project (HELP). Canberra: Curriculum Development Centre.

Fraser, B.J. and D.L. Smith (1980b). Application of a change agent strategy in dissemination of an Australian innovation. Australian Journal of Teacher Education, 5: 9-19.

Hudson, H. (chair ) (1992). Department of Employment, Education and Training (DEET). Report of the Discipline Review of Computing Studies and Information Sciences Education (Vol. 2: The Main Report). Canberra: AGPS.

Institute of Chartered Accountants in Australia (1994). Chartered accountants in the 21st century. Sydney: The Institute of Chartered Accountants in Australia.

Johnson, T., J. Ainley and M. Long (1996). The 1995 Course Experience Questionnaire. Graduate Careers Council of Australia, Parkville (Victoria).

Kessell, S.R. (1992). What's a tertiary education -- and am I getting one? Australian Campus Review Weekly. 18 June, p. 8.

Kessell, S.R. (1995). The undergraduate course controller: Mentor or masochist? Proceedings of the Teaching Learning Forum [A Focus on Learning]. Perth: Edith Cowan University: 135-138.

Kessell, S.R. (1996a). Addressing the flaws in intensive science degrees. Higher Education Research and Development Society of Australasia Conference, Perth: 364-370.

Kessell, S.R. (1996b). An evaluation of "literacy" and "breadth" units, and the provision of student counselling, in undergraduate science courses. Master of Education thesis, Curtin University of Technology, Perth.

Kessell, S.R. and J.A. Kessell. (1993). Improving the breadth, communication and interpersonal skills of graduates in intensive undergraduate science degrees. Proceedings of the Teaching Learning Forum [Higher Education Teaching and Learning: Sharing Quality Practice]. Perth: Curtin University of Technology: 117-120.

Latchem, C., L. Parker and J. Weir (1995). Communication-in-Context: A report on communication skills development for the Teaching and Learning Advisory Committee. Perth: Curtin University of Technology.

Lee, A. (1991). Language and literacy in undergraduate education. In: F. Christie, P. Freebody, A. Devlin, A. Luke, J. Martin, T. Threadgold and C. Walton (eds.), Teaching English literacy: Project of national significance on the preservice preparation of teachers to teach English literacy (3 volumes). Darwin: Northern Territory University.

McInnis, C., R. James and C. McNaught (1995). First year on campus: Diversity in the initial experiences of Australian undergraduates. Melbourne: Centre for the Study of Higher Education, University of Melbourne.

Reid, I. (1996). Higher education or education for hire? Rockhampton (Queensland), Central Queensland University Press.

Sanders, J.R. (chair) (1994). The program evaluation standards: How to assess evaluations of educational programs (2nd edition). Thousand Oaks (California): Sage.

Smith, D.L. and B.J. Fraser (1980). Towards a confluence of quantitative and qualitative approaches to curriculum evaluation. Journal of Curriculum Studies, 12: 367-370.

Williamson, J., B. Fraser, K. Tobin, H. Canute, J. Lake and O. Watts (1987). A summary of The senior colleges in Western Australia: An evaluation. Perth: Ministry for Education.

Please cite as: Kessell, S. R. (1997). Teaching literacy in intensive science degrees: A formal evaluation. In Pospisil, R. and Willcoxson, L. (Eds), Learning Through Teaching, p154-162. Proceedings of the 6th Annual Teaching Learning Forum, Murdoch University, February 1997. Perth: Murdoch University. http://lsn.curtin.edu.au/tlf/tlf1997/kessell.html


[ TL Forum 1997 Proceedings Contents ] [ TL Forums Index ]
HTML: Roger Atkinson, Teaching and Learning Centre, Murdoch University [rjatkinson@bigpond.com]
This URL: http://lsn.curtin.edu.au/tlf/tlf1997/kessell.html
Last revision: 2 Apr 2002. Murdoch University
Previous URL 11 Jan 1997 to 2 Apr 2002 http://cleo.murdoch.edu.au/asu/pubs/tlf/tlf97/kess154.html