Teaching and Learning Forum 2002 Home Page
Teaching and Learning Forum 2002 [ Proceedings Contents ]

Assessing professional skill development at a third year level

Werner Soontiens
School of Management, Curtin University

Barbara de la Harpe
Faculty of Education, Languages and Community Services, RMIT University

Carmela Briguglio
CBS Communications Skills Centre, Curtin University

Employers, professional bodies, instructors and students themselves recognise the need for graduates to leave university with a good understanding of both disciplinary content as well as a range of highly developed professional skills. Many universities have responded to this need by encouraging the development of such skills in their programs of study. Curtin Business School has implemented the Professional Skills Project that aims to integrate the teaching and assessing of professional skills into the units of the Bachelor of Commerce degree program.

As part of this initiative, the first author successfully won a grant to implement a project that focussed on supporting students' development of their presentation and written communication skills in his third year management unit. These skills were specifically selected based on the lecturer's previous experiences of teaching the unit which showed that while students in CBS are given opportunities to develop their of professional skills throughout the course, there seemed to be a significant variation in students' skill levels. The variation in skill development was particularly apparent in the assessment of group presentations and written assignments in semester one 2001. Thus, opportunities for students to develop these skills were integrated into the unit and data on students' perceptions of their skill development were obtained by them completing a questionnaire at the beginning and end of the unit.

In this paper, we outline how the skills were taught and assessed, present the data on the changes in students' perceptions of their skill levels, and discuss the implications for teaching and assessing presentation and written communication skills in the context of the discipline.


Background

Employers, professional bodies, instructors and students themselves recognise the need for graduates to leave university with a good understanding of both disciplinary content as well as a range of highly developed professional skills. Australian higher education has been coaxed into preparing young people for employment. The simultaneous pressures of expanding student numbers, innovations in teaching and learning and more intense competition between universities have changed the landscape of higher education (McInnes, James and Hartley, 2000).

Universities are increasingly focussing on the need for developing core skills or competencies in their graduates as articulated by professional associations and industry (Rosenman, 1996). This is done against the backdrop of rapid change and high levels of competition in the labour market. By developing and integrating transferable skills or competencies universities hope to increase the employability of their graduates (Murphy, 1997). The dominant skill deficiencies of graduates as perceived by employers in the context of the Australian labour market include inter alia communication skills, interpersonal skills and problem solving and the understanding of business practice (ACNeilson, 2000).

Research to identify skills sought in new graduates by Australian employers lists academic achievement, time management, written business communication, oral communication, interpersonal, teamworking, problem solving and comprehension of business processes as the skills most often sought by employers (ACNeilson, 2000). These skills are sought in all graduates entering industry and Government Departments.

Many universities have responded to the need to produce graduates with the skills sought by employers by encouraging the development of such skills in their programs of study. As part of Curtin's goals and strategic plan for teaching and learning, which is to produce graduates who are well equipped for careers in their chosen fields (CBS(a), 2001), Curtin Business School (CBS) has implemented the Professional Skills Project (PSP) that aims to integrate the teaching and assessing of professional skills into the units of the Bachelor of Commerce degree program. The PSP aims to enhance both the professional skills and employability of CBS graduates (CBS(b), 2001). In the assessment of appropriate skills for CBS graduates by a cross discipline Task Force, the following five key professional skills were identified; communication (written, presenting and speaking out), teamwork, decision making, and computer and information literacy (CBS(a), 2000).

The approach towards teaching professional skills is based on a developmental process over the 3 years of students' undergraduate study (CBS(b), 2000). In line with the literature on skill development, it is deemed more effective to teach, apply and assess skills in the context of the relevant disciplines (Rosenman, 1996). According to Murphy (1997), while some skills are common to most disciplines, the focus and mode of development and the context for transfer might differ across the disciplines. In addition, Rosenman (1996) points out that although core skills should be agreed upon across the university or faculty, the implementation thereof should be done within individual subjects within departments. In line with this suggestion, CBS invited staff during March 2000 to express interest in undertaking a Teaching and Learning Mini Project to foster discipline based skill development and assessment strategies (CBS(c), 2001). The aim of these projects is to establish models and case studies for other staff and at the same time to build a core group of staff who will be 'champions' and 'ambassadors' for skill development within CBS (CBS(a), 2001).

The project

As part of the Professional Skills Mini Project initiative, the first author successfully won a grant to implement a project in his third year International Management 375 unit that focussed on supporting students' development of their presentation and written communication skills. Opportunities for students to develop these skills were integrated into the unit and staff in the CBS Communications Skills Centre provided support. The skills were specifically selected since as ACNeilson (2000) point out that: "if there is a dissatisfaction with graduate skills, it probably lies in the area of written communication. ... students are not taught to write in a manner appropriate to business communications" (p.9). In addition, student literacy (including speaking, listening, reading and writing) is increasingly becoming a curriculum issue as the numbers of English as second language students increase in Australian universities (Ingleton, 1996).

Against this background and the lecturer's previous experiences of teaching the unit, these skills were specifically selected. The experience inter alia showed that while students in CBS are given opportunities to develop their professional skills throughout the course, there seemed to be a significant variation in students' skill levels. The variation in skill development was particularly apparent in the assessment of group presentations and written assignments in semester one 2001. The variation in skill levels of students can possibly be attributed to the fact that a large number of students are not primarily English speaking and the limited exposure to communicating in English despite them being third year students in Australia. Students are able to do a number of units in their undergraduate studies with partners off shore (Malaysia, Singapore, Hong Kong) and transfer to Australia to finish the outstanding number of units.

During the course of the 13-week unit, students, working in self selected groups are required to submit a written assignment of about 3000 words on a topic relevant to the unit. In addition, groups are expected to make two presentations on the assignment during the semester. Part of the CBS Professionals Skill project is that skills should be assessed (CBS(c), 2000) and in this unit the presentations and written assignment represent 40% of the final mark for the unit. Of this, about 60% is for the skills and 40% is for academic content.

Although students were also involved in developing their teamwork skills through working in groups, no support was given in their activities and interaction in teams. The teamwork activity can, therefore, be interpreted as a comparison activity in terms of the impact of providing or not providing support on skill development.

Methodology

In order to assess the improvement of students' perceptions of their presentation, writing and teamwork skills, a questionnaire was developed to gather data on students' perceptions of their skill levels. The questionnaire included 11 questions on presentation skills, 12 questions on writing skills and 10 questions on teamwork skills using a five-point Likert scale (see Table 2). A number of questions on the demographics of the students and their perceptions of the frequency, value and methods used to develop skills across all the units that they were studying were also included. (see Table 1). The latter questions were only included in the post questionnaire in order to get feedback from students about their skill development given that the approach adopted across CBS and within specific units is at this stage bottom-up and not coordinated centrally.

The questionnaire was developed in such a way that it served three purposes. Firstly, to collect data on the perceptions of students at the beginning of the semester (week one). Secondly, to collect data on the perceptions of students after the presentations (between week 7 and 13) and at the end of the semester (week 13). Thirdly, to indicate to students the actual criteria that would be used to assess the skills throughout the course.

All students attending lectures in International Management 375 on the Bentley campus were requested to complete the self-assessment. In week one there were 131 attendants while in week 13 there were 92 attendants. Presentation assessments were administered immediately after individuals had made presentations and a total of 116 assessments were collected.

After the data were collected in week one, a consultant from the Communication Skills Centre visited all classes and presented a 90 minute workshop on making presentations. The first part consisted of class discussion and input of ideas into what makes a good presentation. Discussion included ideas relating to principles of good communication as well as technical aspects such as appropriate use of overhead transparencies or PowerPoint. Students were then placed into groups and given a topic on which to prepare a brief 5 to 10 minute team presentation. Each group then presented using overhead transparencies developed in class. After the presentation, class members and the Consultant made friendly suggestions for improvement where appropriate.

Between weeks three and seven all groups had to make their initial presentation. Feedback based on the skills as identified in the questionnaire was given to each group of presenters. This implied that during these weeks students were exposed to at least two presentations per session. In addition, students were asked to assess the final presentations of their peers, thus giving them the opportunity to assess the skills of others.

The written assignment was to be submitted to the CBS Communications Skills Centre before being assessed by the lecturer. Students made appointments to meet with consultants at the Centre and submitted their draft assignment the day before the scheduled meeting. Consultants read assignments, underlined errors and unclear expression and wrote comments and suggestions for improvement on a sheet developed for this purpose, which had to be submitted to their lecturer with the final draft of their assignment. Group members then met with a consultant and discussed improvements for their assignment. It should be noted that assignments were not edited or corrected by Centre staff. Rather it was stressed to students that they were responsible for making necessary improvements and corrections, and Centre staff were there to provide advice on how this might be done. The perceptions of teamwork were collected to serve as a control item in the sense that no support was provided to students to prepare them to address problems with team interactions. Students had a free choice of team members in week one and had to complete both presentations and the written assignment as a team.

As mentioned, the data on students' perceptions of their teamwork skills were collected to serve as comparison data in the sense that no support was provided to students to prepare them for or to address any problems that they may have had with team interactions. As stated previously, students, in week one, had free choice in the selection of team members and had to complete both presentations and the written assignment as part of that team.

Pre and post data gathered were analysed in terms of the demographics and responses to the different skills (presenting, writing and teamwork). For the data on the skills effect sizes were calculated. Effect sizes were based on the calculation of the mean of the responses to each question and the change between the pre- and post-semester perceptions. The change was also interpreted in terms of its educational significance.

Data

The analysis of the demographic data and responses to the questions on the implementation of professional skills are presented in Table 1.

Table 1: Demographical data and professional skill implementation

Demographicsn%
Gender MaleFemaleMaleFemale
55764258
Language
background
EnglishNon-EnglishEnglishNon-English
51803961
Permanent
residence
AustraliaOtherAustraliaOther
45863466
Age Under 2121 and overUnder 2121 and over
41903169
Units
completed
n%
< 1010 to 20> 20< 1010 to 20> 20
Total1172 4385533
At Bentley5937 30452823
Skill implementationn%

YesNoYesNo
Does multiple reinforcement improve professional skills911991
Professional skills are developed in different ways by different units884964
Professional skills add value to my qualification9201000
The amount of exposure to skill development isToo littleJust rightToo muchToo littleJust rightToo much
1274613807

From Table 1 it is clear that the students were predominantly from an non-English background (61%), under 21 years of age (69%) and have a non-Australian permanent residence (66%).

Most students were near the end of their course, with 55% having completed between 10 and 20 units and 33% having completed more than 20 units of the 24 required to graduate. Incidentally, 45% of the students had completed less than 10 units at the Bentley campus in Australia.

Questions on the value and the exposure to professional skills development were included to determine whether the lack of coordination and integration across units, programs and schools within CBS was experienced negatively by students. Analysis of the data showed that the response to these questions was overwhelming positive and in favour of skill development. For example, as indicated in Table 1, all students indicated that professional skills added value to their qualification, while 99% agreed that multiple reinforcement improved their skill levels. Notwithstanding the lack of coordination in the teaching of skills, 96% of students indicated that they experienced the same professional skills being taught in different ways by different units. The level of exposure to skill development in the different units was assessed as just right by 80% of students. While 13% of students indicated that they would prefer more support for skill development in their units, only 7% indicated that they would prefer less exposure. The data suggests that the future inclusion of professional skills in units should be supported.

Data analysis of the student responses to the questionnaires administered at the beginning and the end of the semester are presented in Table 2.

Table 2: Data analysis of self assessment

Item
No.
Item descriptionMeansESN
PrePostchangegsigPrePost
1.1I was comfortable presenting in English3.874.160.280.32*131116
1.2I was confident when talking in front of an audience3.183.730.560.68**131116
1.3I was able to use aids effectively to improve my presentation3.663.900.240.33*131116
1.4I was comfortable speaking in a loud and clear voice3.504.020.510.65**131116
1.5I was able to effectively make eye contact with the audience3.473.900.430.49*131116
1.6I found it easy to structure my presentation3.443.850.410.55**131116
1.7I was able to present without having to read from notes3.043.610.570.62**131116
1.8I found it easy to stick to the time allocated3.333.860.530.63**131116
1.9I could keep the audience interested2.993.500.510.64**131116
1.10I felt comfortable when asked questions by the audience3.093.760.670.72**131116
1.11Overall, I was a good presenter3.193.790.600.82***131116
Presentation Skill (average items: 1.1 to 1.11)3.343.830.480.54**
2.1I find it easy to express myself in English3.804.040.240.26*13192
2.2I have no problems with grammar and spelling3.463.620.160.16
13192
2.3I find it easy to write in an academic style3.373.650.280.34*13192
2.4I can construct an informative introduction3.443.710.260.35*13192
2.5I am comfortable in writing a comprehensive summary3.353.620.270.37*13192
2.6I am able to develop a suitable structure for the content3.403.850.450.65**13192
2.7I know how to present a report professionally3.313.760.450.53**13192
2.8I find it easy to locate relevant resources3.333.740.410.56**13192
2.9I know how to reference when using the exact words of the author3.634.000.370.44*13192
2.10I know how to reference when paraphrasing the words of the author3.613.950.330.39*13192
2.11I know exactly how to construct a reference list3.814.150.340.42*13192
2.12Overall, I believe I am a good writer3.283.660.380.55**13192
Writing Skill (average items: 2.1 to 2.12)3.483.810.330.40*
3.1I find it easy to work in a team using English as the language of communication4.094.100.010.01
13192
3.2I find it easy to stick to the preset time frames3.733.800.080.12
13192
3.3I try to make an equal contribution as a team member4.274.21-0.06-0.09
13192
3.4I am sensitive to needs of members from other cultures3.743.900.160.19
13192
3.5I realise that not all members show the same commitment and sense of responsibility3.933.990.060.06
13192
3.6I prefer being the team leader2.973.280.310.34*13192
3.7I am able to accommodate different attitudes and approaches3.713.750.040.06
13192
3.8I am willing to learn a lot from other team members4.304.12-0.18-0.27*13192
3.9I enjoy working in a team3.953.62-0.33-0.37*13192
3.10Overall, I believe I am a good team worker3.924.100.180.28*13192
Team Working Skill (average items: 3.1 to 3.10)3.863.890.030.03

Note: Large effect size = *** (g>0.8), Medium effect size = ** (g>0.5), Small effect size = * (g>0.2)

As shown in Table 2, of all the skills, the greatest improvement in students' perceptions of their skill levels was for the presentation skill (g=0.54) Data suggests that the greatest change in this skill was due to students indicating that they believed that they were better presenters at the end of the semester that at the beginning of the semester (g=0.82). Amongst presentation skills there were six other aspects that showed medium effect size improvements (g>0.5) and there was no aspect that had no effect size improvement. The aspects of the presentation skill that students rated themselves highly for in the pre-semester assessment were also rated highly in the post-semester assessment.

The improvement in writing skills was less significant than for the presentation skill, the overall effect size of the improvement of writing skills was 0.4. There were only four skills where students indicated to have improved more than 0.5. Although the aspects of the writing skill rated the highest pre-semester were still rated the highest post-semester there was a change in the skills perceived to be worst pre-semester compared to post-semester. In the pre-semester assessment the skill rated the lowest was the overall perception to be a good writer while in the post-semester assessment grammar, spelling and the writing of a comprehensive summary were the worst skills.

The experiences of students in terms of teamwork skills were significantly different. Apart from the fact that there were no significant changes in pre-semester and post-semester assessment for 7 of the 10 skills, there were two skills that have deteriorated. Students indicated a small effect size (g.0.2) deterioration in their willingness to learn a lot from other team members and their enjoyment of working in a team. Overall, the change in students' perceptions of their teamwork skills indicated no significant change.

Discussion and implications

It is difficult to obtain direct empirical evidence of a link between the existence of support services and positive academic outcomes, including professional skills, due to the many external variables that have an impact on the lives and academic performance of students. Promnitz and Germain (1996) state, however, that the experiences of students generally support the positive role which support services play. In addition, Ingleton (1996) insinuates that the promotion and support of literacy and communication skills will address the literacy levels of students.

Data in this paper suggests that the more students were exposed and supported in the development of professional skills, the more confident students became in these skills or ...the more they perceived that their skills had improved/developed. The continuous exposure of students to presentations, both of themselves and peers throughout the unit seems to have contributed to the most significant improvement in the self assessed skills. Writing skills were supported by inputs from the CBS communications skills centre and subsequently also showed an improvement. The group of skills where no support was provided, namely teamwork, showed no improvement from students and even indicated that some of these skills were worse after the semester than they were perceived at the beginning of the semester.

Data also suggested that the random inclusion of professional skill development, as currently applied in CBS units, is appreciated by students. Also, the actual development of skills seem to be positively influenced by providing support and exposure to the criteria used in the assessment of the skills.

Conclusion

The inclusion of professional skills in academic units is perceived by students as useful and adding value to the program. Similarly, the non-coordinated inclusion of the various skills in various units is experienced as a variety of spices by students. Although there is no direct proven link between the availability of support infrastructure and the amount of skill improvement gained by students, data in this study indicate that the best skill improvement was derived in the skills where ample support was provided. The challenge in including professional skills in a curriculum is to integrate the available infrastructure, services and staff in order to allow students to derive maximum benefits from professional skill development.

References

ACNeilson (2000). Employer Satisfaction with Graduate Skills: Research Report. Department of Education, Training and Youth Affairs, Canberra.

CBS (2000a). Skills Curtin Business School has chosen. [verified 20 Dec 2001] http://www.cbs.curtin.edu.au/staff/PSP/CBS_skills.cfm

CBS (2000b). Employers and students. [verified 20 Dec 2001] http://www.cbs.curtin.edu.au/staff/PSP/what_emp.cfm

CBS (2000c). Assessing professional skills. [verified 20 Dec 2001] http://www.cbs.curtin.edu.au/staff/PSP/Assess.cfm

CBS (2001a). Professional skills project: Mini Projects background. [verified 20 Dec 2001] http://www.cbs.curtin.edu.au/staff/PSP/background.cfm

CBS (2001b). CBS professional skills project. [verified 20 Dec 2001] http://www.cbs.curtin.edu.au/staff/PSP/Page_3.cfm

CBS (2001c). Professional skills project: Mini projects. [verified 20 Dec 2001] http://www.cbs.curtin.edu.au/staff/PSP/mini_projects.cfm

Ingleton, S. (1996). Changing literacy: Changing teaching and learning. Paper for the First National Conference on Tertiary Literacy, Victoria University of Technology, March 15-16.

McInnes, C., James, R & Hartley, R. 2000. Trends in the First Year Experience in Australian Universities. Department of Education, Training and Youth Affairs, Canberra.

Murphy, E. (1997). From university to work: Developing transferable skills. In Pospisil, R. and Willcoxson, L. (Eds), Learning Through Teaching, p239-243. Proceedings of the 6th Annual Teaching Learning Forum, Murdoch University, February 1997. Perth: Murdoch University. http://cleo.murdoch.edu.au/asu/pubs/tlf/tlf97/murp239.html

Promnitz, J & Germain, C. (1996). Student Support Services and Academic Outcomes: Achieving Positive Outcomes. Department of Employment, Education Training and Youth Affairs, Canberra. [verified 20 Dec 2001] http://www.dest.gov.au/archive/highered/eippubs/student/eip96-10.htm

Rosenman, L. (1996). The Broadening of University Education: An Analysis of Entry Restructuring and Curriculum Change Options. Department of Employment, Education, Training and Youth Affairs, Canberra. [verified 20 Dec 2001] http://www.dest.gov.au/archive/highered/eippubs/eip9612.htm

Contact person: Werner Soontiens, Lecturer, Management, Curtin University of Technology. soontiew@cbs.curtin.edu.au

Please cite as: Soontiens, W., de la Harpe, B. and Briguglio, C. (2002). Assessing professional skill development at a third year level. In Focusing on the Student. Proceedings of the 11th Annual Teaching Learning Forum, 5-6 February 2002. Perth: Edith Cowan University. http://lsn.curtin.edu.au/tlf/tlf2002/soontiens.html


[ Proceedings Contents ] [ TL Forum 2002 ] [ TL Forums Index ]
This URL: http://lsn.curtin.edu.au/tlf/tlf2002/soontiens.html
Created 23 Dec 2001. Last revision: 1 Feb 2002. HTML: Roger Atkinson [ rjatkinson@bigpond.com ]