Teaching and Learning Forum 95 [ Contents ]

Evaluating outcomes in physiotherapy students

Colleen Liston
School of Physiotherapy
Curtin University
Evaluating outcomes in physiotherapy students focuses on their competency to enter the profession on graduation. Evaluation is equal to the whole constellation of values for any program, including the outcome of its graduates, divided by the complexities in criteria and expectations of the stakeholders. That is, an observed value is compared with some standard.

In terms of function, standards may provide a benchmark or basic goal for training providers, industry and training bodies, or educators, as well as state and federal authorities. They may be applied in considering registration, certification, accreditation and other forms of recognition of skills, or for setting curricula and in assessment of various kinds. Standards can be expressed in different ways. It may be suggested that consideration be given to a number of factors when setting any level. These include:

  1. their eventual use;
  2. the audiences;
  3. operational and cultural requirements;
  4. the range of workplace and occupational situations.
This paper discusses the development of competency standards for physiotherapists in Australia (Liston Bullock & Cole, 1994), and their application to evaluating outcomes in physiotherapy students: - their competence to practise.

Introduction

In a discussion about evaluation, Stufflebeam & Shinkfield (1985) see "three main uses; improvement, accountability, and enlightenment" (p7). By providing information about a current program through self-study reporting, with external confirmation and clarification, there may be improvement (or maintenance) in the quality of the program. Accountability may be achieved through providing written evidence as feedback to interested parties about the program and its outcomes/products, that is, the graduates. Enlightenment is a comfortable and comforting term where judgement and other subjective approaches are used instead of the research/investigative model (Glass, 1980; Cronbach, 1963). Furthermore, Stake (1967) proposes the equation that evaluation is equal to the whole constellation of values for any program, divided by the complexities in criteria and expectations of the stakeholders. That is, an observed value is compared with some standard.

Standards

Standards may be applied to provide criteria for evaluation purposes. They may be norm-referenced (as in relation to a passing standard set previously by a group of examiners), task-referenced (as in objectives stated for observable tasks such as matching, naming, selecting, recognising) or criterion-referenced (as when a performance is compared with stages on a scale of increasing competence).

Standards may be deemed to be necessary by governments - for example to enable industrial groups, unions, registration boards and the like to ensure that all workers, employees, professionals, or practitioners meet a minimum standard of performance in a range of skills/abilities (Locke and Latham 1990). Educationalists and professional bodies may set standards for the education or training of personnel (Jarvis 1983). These should match the minimum standard required by a newly qualified or graduated trainee or professional. Where quality assurance is required or desired, standard setting means that there is a reference point against which to judge the quality and ensure that it is maintained.

Although most issues perceived to be quantitative have a qualitative basis, the notion of standard setting is more difficult in qualitative issues than in quantitative ones. Ambiguity may exist where competence can be expressed only in higher order cognitive skill terms. The holistic nature of performance cannot be ignored. Consideration of some means of expressing the need for performance to be based on the combination of knowledge, skills, abilities and attitudes poses a problem for those setting standards.

Glass (1978) questions the use of the term minimal competence which suggests the essential, least permissible level. This appears on a continuum from absence to something especially excellent, and is extremely difficult to determine. In fact, judges disagree when seeking to determine these levels of minimal competence, and psychologically there is no basis for the concept. Placing a focus upon desired competencies for a particular profession may, however, offer a means by which a new or existing curriculum can be evaluated, particularly if the competencies have been defined in relation to the current needs in professional practice (Myers, 1990). While educators may have specified their objectives carefully, the defining of needed competencies may present an opportunity for a more integrated approach to teaching/learning and to student assessment (May, 1977; Beenhaker, 1987; Liston, 1993).

Whilst acknowledging the arbitrary nature of standard setting, Scriven (1978) suggests that this is better than nothing at all! Glass (1978), in contrast, feels that "arbitrariness is no bogeyman", yet that "nothing is safer" (p242)! So the argument is about the absolute nature of standards. In order to make a judgement about the value of anything or of anyone's performance, then some agreed level (with safety margins) may need to be determined or set.

Outcomes evaluation

Evaluation of outcomes in a higher education context, focuses on students' ability to apply acquired knowledge and reasoning skills competently. The use of educational strategies which encourage students to explore knowledge, generate new knowledge, plan, learn, and reason, will promote metacognition and reflection on graduation. Valuing all of the facets of their educational experience will foster a desire for lifelong learning in graduates.

There are six Physiotherapy Schools in universities in Australia, and each of them has a unique curriculum, differing teaching strategies, differing expectations and standards. For the physiotherapy profession, there are no benchmarks, no National or State examinations to determine eligibility for registration to practise, and no professional accreditation of the six undergraduate programs. Evaluation of outcomes in each of these programs relies on each School specifying its own criteria, its own standards, and its own requirements in terms of sources of evidence for making judgements and decisions about who graduates and who does not.

The students are assessed against some predetermined and ill-defined standards by assessors who gather evidence from a range of activities; such as essays, multiple-choice and short-answer written assessments, practical tests, viva-voce examinations, clinical observations and the like; and make judgements about the levels of competency of individuals. Those assessing or making judgements need to have a means to know what the expectations are - that is, an understanding of the competencies and knowledge base being assessed. Furthermore, the assessors require the ability to make observations and judgements, keep records, obey decision-making rules, and be open to independent monitoring.

Competencies in professional education

Competency-standards for the professions may provide an accountability mechanism for evaluating the outcomes (the graduates), whilst not compromising the unique nature of individual programs. In addition, although higher education is often purported to be about excellence, not competence per se, in reality a pass mark/minimum standard or cut off point is invariably set!

The process of education is not demeaned through defining competencies. Indeed the process is one of the outcomes, because the fostering, nurturing and cultivating aspects of it, which were identified by John Dewey in 1916, are reflected by competent graduates.

Competencies are any attributes of an individual which contribute to performance. Surely knowledge is the foundation for competent performance. Striving for excellence is more likely to be fostered by nurturing learners and cultivating competencies than by preparing for success in norm-referenced methods of assessment. The reason for developing and reviewing course objectives and the performance criteria which are to be assessed is not to stultify educational programs. Rather, it is because of the understanding that there are continuously evolving contemporary needs which require competencies to be expressed with ever greater precision. Competencies include knowledge, thinking processes, manual skills, attitudes, motivations and beliefs. Confusion and stultification that result from arguing over semantics is unproductive.

Applying competency standards in physiotherapy education

The physiotherapy undergraduate program conducted at the Curtin University School of Physiotherapy, like the majority of professional programs in higher education, provides a linkage to the graduates' ultimate workplace settings by incorporating clinical education. In the final year of the program, the major component is clinical education across a range of settings and circumstances which provide experiences in occupational situations. What better way to introduce an interface between the didactic and artificial university education setting and the real world/workplace competencies that are required by the profession? Honing skills and improving social literacy are important for physiotherapy students. Whether the new graduate meets the competency level necessary in a particular workplace setting is only the concern of the university when achievement in clinical settings is assessed. This aspect of assessment of the competency of the student should incorporate any professional standards that have been developed.

Conclusion

Perhaps standard is an alternative word used to set a level of performance, where formal examination or evaluation cannot ensure that an objective has been met. In any event, both objectives and standards require agreed conventions and professional judgement in which there may be disagreement about decisions when it comes to scoring, and passing or failing.

In Australia, Competency Standards have recently been developed by the physiotherapy profession. These standards are meant to apply to new graduates, are open to review and change, and have been accepted by Schools, Registration Boards, the Australian Examining Council for Overseas Physiotherapists and the Australian Physiotherapy Association (Liston, Bullock & Cole, 1994). The influence of competency standards in curriculum development and assessment of students, as well as in a process of accreditation of physiotherapy education programs is worthy of informed debate prior to considering any implementation. This is especially in view of the focus of the Federal Government on auditing quality procedures in higher education, which requires educational institutions and programs to have quality assurance processes in place.

Professional accreditation - assessing whether programs produce graduates who meet entry-level competency standards - is more closely linked to eligibility for registration. It may be that the only reliable means of assessing whether all physiotherapists seeking to register initially in Australia meet the entry-level competency standards, is for them to undertake an examination after graduation.

References

Beenhaker, J (1987). Determinants in physiotherapy education. Medical Teacher, 9(2), 161-165.

Cronbach, L (1963). Course improvement through evaluation. Teachers College Record, 64, 672-683.

Glass, G (1978). Standards and criteria. Journal of Educational Measurement, 25(1), 57-65.

Glass, G (1980). Evaluation research. Annual Review of Psychology, 31, 211-228.

Jarvis, P (1983). Professional Education. London: Croom Helm Limited.

Locke, E and Latham, G (1990). A Theory of Goal Setting and Task Performance. Englewood Cliffs, NJ: Prentice Hall.

Liston, C. B. (1993). The Relevance of Competency-Based Standards to Higher Education Workbook I. Competency-Based Standards Project. Perth: Curtin University Publication.

Liston C., Bullock M. and Cole J. (1994). Australian Physiotherapy Competency Standards. Brisbane, Body Control Systems.

May, B. (1977). Evaluation in a competency-based education system. Physical Therapy, 57, 28-33.

Myers, S. (1990). Program Evaluation for Accreditation. The American Journal of Occupational Therapy, 44(9), 823-826.

Scriven, M. (1978). How to anchor standards. Journal of Educational Measurement, 15(4), 273-275.

Stake, R. E. (1967). The countenance of educational evaluation. Teachers College Record, 68, 523-540.

Stufflebeam, D. L. and Shinkfield, A. J. (1985). Systematic Evaluation. Boston: Kluwer-Nijhoff.

Please cite as: Liston, C. (1995). Evaluating outcomes in physiotherapy students. In Summers, L. (Ed), A Focus on Learning, p161-164. Proceedings of the 4th Annual Teaching Learning Forum, Edith Cowan University, February 1995. Perth: Edith Cowan University. http://lsn.curtin.edu.au/tlf/tlf1995/liston.html


[ TL Forum 1995 Proceedings Contents ] [ TL Forums Index ]
HTML: Roger Atkinson, Teaching and Learning Centre, Murdoch University [rjatkinson@bigpond.com]
This URL: http://lsn.curtin.edu.au/tlf/tlf1995/liston.html
Last revision: 16 Apr 2002. Edith Cowan University
Previous URL 26 Feb 1997 to 15 Apr 2002 http://cleo.murdoch.edu.au/asu/pubs/tlf/tlf95/list161.html