Teaching and Learning Forum 2009 Home Page

Category: Research
Teaching and Learning Forum 2009 [ Refereed papers ]
Teaching, learning, and assessing a graduate skill

Doug Atkinson
School of Information Systems
Curtin University of Technology

A teaching/learning intervention was designed and implemented by a teacher to improve the graduate skill of referencing for undergraduate commerce students studying web design. Over two semesters, 120 students were studied. In each semester, a "participant" group voluntarily undertook a class exercise during a lecture and received feedback from the teacher. A "non-participant" group chose not to undertake the exercise. In addition, all students received tuition at laboratories, assignment instructions, assignment assessment, and access to a library referencing resource. Referencing skill was later measured for both groups via a written exam question. Results indicated that the participant group achieved a superior level of referencing skill. Aside from the inference that teaching and learning a skill improves performance, the study demonstrates how a graduate skill can be taught, learnt and assessed at the unit level of a tertiary curriculum. The paper covers graduate skills, referencing, the teaching/learning design, the quasi-experimental research design, and the results.


Introduction

The research described in this paper concerns the design and efficacy of the teaching, learning, and assessment of the graduate skill of referencing. The objective was to design a teaching intervention to improve student referencing skills and to research the efficacy of that design. The paper begins with an introduction of graduate skills and referencing in particular. The teaching and learning design is then described. The quasi-experimental research design to test the efficacy follows. The results, discussion and recommendations complete the paper.

The significance of teaching and learning graduate skills is that there are generic useful qualities that students develop over the course of a university degree that can be applied in the global workplace. These qualities, for example, communication skills, are generic and useful in the sense that they are required in any workplace or profession, and are useful over time. In contrast, some discipline specific knowledge and skills, for example, programming in assembly language, can have a narrow area of application and may become obsolete as technologies and contexts change.

Within Western Australia, Curtin University, Murdoch University (Murdoch University, 2004) and Edith Cowan University have developed graduate skill programs. At Curtin University the graduate attributes/graduate skills project began in the Curtin Business School (CBS) in 1998 (Guthrie, McGowan, & de la Harpe, 2001). In 2004 the University developed graduate attributes that were to be included in all Curtin courses (Curtin University, 2003). The high level goals of the Curtin Business School (CBS) program required students to graduate with skills in several categories. These categories included information literacy, information technology literacy, problem solving, decision making, teamwork, presentation, interaction, and writing (Curtin Business School, n.d.).

Within this research, the referencing skill was treated as a sub-skill of the categories of information literacy, and writing. The context was a web design unit where the author was the unit controller. An extract from the unit outline (Table 1) illustrates the referencing skill (statement 3) within the context of graduate skills outcomes.

Table 1: Graduate skills statement extracted from the unit outline of Design 150

Graduate Skills Outcomes

Employers worldwide want graduates who have developed effective graduate skills. The CBS graduate skills program includes communication (writing, interpersonal interactions and cultural awareness, and presenting), critical and creative thinking (problem solving and decision making), team work, IT literacy and information literacy.

On successful completion of this unit, students should be able to demonstrate:

  1. IT literacy through use of contemporary information technology (VB Editor, Dreamweaver, HTML) to construct user interfaces. [taught in laboratories and assessed in assignment B]
  2. Critical and creative thinking through planning, designing, developing and evaluating user interfaces. [taught in lectures and laboratories and assessed in assignments]
  3. Information literacy through the use of Harvard Referencing to acknowledge information created by others. [taught in lecture and lab topic 10 and assessed in assignments]

The rationale for pursuing referencing as a graduate skill was that a graduate who develops skills in referencing over time should make a better reader, writer, and researcher. They should be better able to judge the depth of research of a written work and make assessments about its quality. Literate graduates should in turn make better contributions to the knowledge economy.

Referencing (citing) and making a reference (citation) is where a writer acknowledges ideas as coming from other authors or sources and provides the publication details of the sources. The act of referencing includes providing short codes in-text (e.g., an author and date, or a number). These short codes then link to a longer ordered description of publication details (e.g., author, date, title, publisher and place of publication) either at the end of the text or at the bottom of the page. This enables a reader to know that an idea came from another source and allows the reader to find the source based on the publication details provided.

In the context of graduate skills, referencing can be considered a sub-skill of both written communication and information literacy as the two concepts are closely related. Good writing requires a synthesis of the existing literature with links so that the reader can ascertain the qualities of the research and have the opportunity to consult the sources. An information literate graduate needs to know how information is referenced so that they can manage their own writing, access other writer's sources, and judge the qualities of written materials.

Within the Curtin Business School, referencing was referred to under the skill categories of information literacy and writing:

Having introduced the significance of graduate skills and referencing, the teaching and learning design is now described.

Teaching and learning design

The design of the teaching of the graduate skill of referencing developed over time in concert with the graduate skills program. The program began with the first year common core 'foundation' units of the commerce degree. As the program was developed, the teaching Schools (Information Systems, Management, Marketing, Business Law, Economics and Finance, and Accounting) were encouraged to embed graduate skills into later units. In this study, the unit Design 150 (User Interface and Web Design), was a first year unit that was the next level after the common core units. It was undertaken by any students studying an Information Systems major.

Additionally, there was an increasing concern about plagiarism within the School and in the University system generally (Le Heron, 2001). Curtin University advocated an education strategy particularly for the first year units where students were often ignorant about referencing. The particular unit, involving web design, had a major assignment that required the design and development of a proposal document, a website, and a usability document. In early implementations of the unit there were numerous cases of plagiarism where students simply did not reference materials used in the website. To rectify the situation the original unit controller, and later the author who inherited the unit, instituted a more rigorous approach to the teaching and learning of referencing.

Firstly, to signal the importance of referencing and to encourage good scholarship, a mark worth 20% of the assignment was allocated. In the assignment instructions a section dealing with the consequences of plagiarism was also included. To improve the learning of referencing, a laboratory exercise (Table 2) was devoted to the task that required students to reference some material they had used in their website. Students were also directed to the referencing examples provided by the Curtin Library (Curtin University, n.d.; 2008).

Table 2: The laboratory referencing exercise (an extract)

Your Task

To ensure that you are confident with the referencing required, as a group prepare a reference (using the correct Harvard referencing format) for:

Text that may be used in your website (i.e. use a subscript or superscript number at the end of the sentence/paragraph and provide the footnote containing the reference at the bottom of that page).

An image that may be used in your website (i.e. using an Alt Tag to contain the footnote number and then the footnote containing the reference at the bottom of that page).

Ask your lecturer / unit leader to check your work.

References

Harvard Referencing, 2006,
Retrieved 2nd March 2006 from
http://library.curtin.edu.au/referencing/harvard.pdf

When the author inherited the unit, a decision was made to promote referencing to the status of a graduate skill, given that it was important throughout the degree and beyond. A statement was made in the unit outline which was then reinforced verbally at the first lecture. One issue that had been generally noted was the lack of attention to online referencing. The nature of the unit being website design represented an opportunity for dealing with this in an appropriate context. Hence there was a synergy between the discipline demands of the unit and the generic skill.

After several semesters, the author decided that a formal participative exercise held in the lecture (Table 3) would reinforce the skill development. The advantage of the lecture was that the author as lecturer had control over the teaching rather than being largely dependent on sessional staff who took the laboratories. At this stage then, the teaching and learning of referencing was embedded throughout the unit, via the unit outline, the assignment instructions and assessment, a lecture exercise, and a laboratory exercise.

Table 3: Pro-forma for the class referencing exercise

Imagine that today you have accessed the Amazon.com website and have used some material from the site.
  1. Write down the Harvard Reference below.

  2. Now based on the feedback from the teacher write down the correct reference below.

  3. Write down the points that you needed to correct.
Please hand this page back to your teacher. He/she will use it to improve learning about referencing.

Finally, a decision was made to include the assessment of referencing in the formal examination. The Curtin Business School (CBS) had been pushing to reinforce the formal teaching and assessment of the graduate skills so the author felt that if assessment was to be treated as seriously as discipline skills then it should take place not only in the continuous assessment but also in the final written examination. Thus within the examination, a question, worth 3 marks out of 50, required students to provide an in-text and an end-of-text reference for a website.

During the implementation of the teaching initiatives, the author saw an opportunity to formally analyse a link between the teaching of the skill and the performance by the students. Specifically there was an opportunity to see whether participation in the lecture exercise brought about an improved performance as measured by the examination assessment. The lecture exercise was controlled by the author so teacher variation was eliminated. Furthermore, there were naturally occurring groups of participants and non-participants because the lecture attendance was voluntary.

This background thus gave rise to the formal objective of assessing the teaching intervention to improve students' referencing skills based on a quasi-experimental design. This is elaborated in the following research method and design section.

Research method and design

A quasi-experimental design (Cook & Campbell, 1979; Mertens, 2005) was adopted to test the efficacy of the teaching intervention. This method refers to approximations to experimental design brought about by pragmatic real world considerations. In terms of research paradigms, the method is usually used within a Positivist/Post-Positivist paradigm (Mackenzie & Knipe, 2006).

Within this study, the design aspects were a measurement of performance, and treatment and control conditions. A non-experimental aspect was the self-selection of subjects into the treatment and control conditions. Furthermore, the researcher was interacting with the subjects in the experiment so there were possibilities of a Hawthorne effect and possible researcher effects on the performance measurements. Despite these limitations, the design is defended as being useful in pursuing the objective of assessing the teaching intervention to improve students' referencing skills.

The researcher was the teacher who had a vested interest in improving the students' referencing skills. The researcher/teacher was involved in conducting the activity, receiving feedback during interaction with students, and improving the activity for the future. The students had a vested interest in improving their referencing skills for both the satisfaction of learning a skill, and receiving better marks in assessment activities.

The testing of the teaching intervention was done via statistical measurement and analysis taking advantage of the natural occurrence of two groups of students - "participants" and "non-participants". Participants were those who attended a class where the main teaching of the referencing skill was carried out. Non-participants were those who chose not to attend. These groups were an approximation of a treatment group and a control group respectively. The approximation refers to the fact that the students self selected into the groups rather than being randomly assigned as in a pure experiment. This then opens the possibility that differences between the groups were causing differences in outcome measures, rather than the differences being purely due to the treatment.

The measurement of performance of the referencing skill was based on the author's assessment of a written answer to a question on referencing a web site in the end of semester examination. This activity was done some time after the official exam marking. Here it was not subject to the usual stress of a tight deadline and the fine assessment of one question out of a much larger examination paper. The students' written answers were assessed by the author via a marking guide out of a maximum total of four marks. A mark was allocated for the author, the year, the date of retrieval, and the Uniform Resource Location (URL). A comparison of this mark was made between the two groups (participants and non-participants) using a T-test of means.

It was expected that the participant group, having received the extra tuition during the class referencing exercise, would perform better than the non-participant group in the final examination.

Thus the hypotheses tested were:

Null hypothesis: The participant and non-participant groups have the same performance level of the referencing skill; and
Alternative hypothesis: The participant group achieved a higher performance level of the referencing skill than the non-participant group.
The students were enrolled in a first year unit of a commerce degree majoring in information systems. The unit was concerned with user interface and web design. The study was run over two semesters of tuition in 2006. The measurement of referencing skill performance was based on students who sat the final examination. A total of 120 students were involved. There were 58 students in the first semester cohort (Table 4) and 62 students in the second semester cohort (Table 5). In the first semester cohort, 11 students (participants) voluntarily completed a class referencing exercise. The other 47 students (non-participants) chose not to attend or did not complete the class activity. In the second semester cohort, 16 students (participants) voluntarily completed a class referencing exercise. The other 46 students (non-participants) chose not to attend or did not complete the class activity.

The participation rate of 23% was not unusual as the class referencing exercise was held during lecture time in the latter part of the semester. Lecture attendance in the School is not compulsory. In the latter part of the semester many students decide not to attend lectures and rather spend time on assignments. Additionally, the lecture and laboratory materials are provided online, and many students perceive these as an adequate substitute for attending lectures. The author's own perspective as a teacher is that extra materials are covered at lectures, and as in the class referencing exercise, non-attendees forgo the live interaction with potential negative consequences for their learning.

At the time of the study, the particular referencing style advocated by the Curtin Business School and supported by the University was the Harvard (author-date) style. This has since been modified to the Chicago (author-date) style. Students had online access to a document provided by the Library (similar to Curtin University, 2008). In addition to an overview of referencing this document provided nine pages of examples including four examples of referencing web documents (Curtin University, 2008, p. 8). Assignment and lab documents in the unit made explicit reference to this resource (Table 2), and the lecturer made numerous verbal references.

The class activity was designed and conducted as follows. Firstly, there was a ten minute interactive discussion of the importance of referencing covering issues such as scholarship, avoiding plagiarism, and allowing readers to find sources. Then each participant was handed a hardcopy one page pro-forma (Table 3) with questions about a referencing task. The lecturer projected a public screen displaying the home page of a famous web site, for example, Amazon, Yahoo!, or Google. The participants were then asked to complete the questions on the pro-forma. The lecturer then projected an answer to the questions, gave some verbal feedback, and asked the participants to correct and mark their work. Participants were then requested to hand up their completed pro-forma. The whole activity took about 30 minutes.

In the same week of semester, the student cohort received a class exercise at the laboratory where they were required to do references to text and images for their website group assignment. This exercise was more tailored to the specific assignment then the class exercise above.

Following submission of the assignment in the latter weeks of the semester, students received written feedback on the marking criteria including comments about the adequacy and quality of the referencing. The final activity for the students was the formal written examination for the unit.

The written examination was a routine assessment method for the undergraduate students. With the exception of some project units, all undergraduate units taught by the School have a written centrally scheduled examination conducted at the end of the teaching semester. An examination will generally account for at least 50% of the student's total grade in the unit - the rest being based on continuous assessment. In this particular unit the examination was 50%, the mid-semester test was 20%, and the major website assignment was 30%. The latter was group work whereas the examination and the test were individually assessed.

The examination question on referencing was similar in nature to the activity undertaken in class. Furthermore, students had been assessed on referencing in their website assignment. Thus students were being examined on a skill they had been given opportunity to practice and improve. However, students were not given specific direction that a referencing question would appear in the examination, thus there was an element of surprise.

The next section details the statistical results and the analysis of the students' written examination answers.

Results

There were two cohorts of students. The first cohort undertook the course in the first semester of 2006. The second cohort undertook the course in the second semester of 2006. The results are reported in chronological order.

Table 4: Comparison of Exam Scores (First Cohort)


ParticipantsNon-participants
Mean (out of 4 marks maximum)2.821.98
Variance1.361.89
Observations1147
Df56
t Stat1.87
p-value0.03

For the first cohort of students, the participants performed better than the non-participants in the final examination (Table 4). On the specific referencing question, the participants averaged 2.82 marks out of a maximum of 4 marks. The non-participants averaged 1.98 marks. This was statistically significant at the 0.05 significance level (p-value=0.03).

Qualitatively, the difference between the groups could be described as "good" (3 out of 4 marks) versus "fair" (2 out of 4 marks). For students this is an indicator that undertaking the class exercise and receiving feedback improves their skill level. For teachers this is an indicator that formally teaching a graduate skill improves student learning outcomes.

For the second cohort of students, again the participants performed better than the non-participants in the final examination (Table 5). On a specific referencing question, the participants averaged 2.75 marks out of a maximum of 4 marks. The non-participants averaged 2.35 marks. This was statistically significant at the 0.10 significance level (p-value=0.09).

Table 5: Comparison of Exam Scores (Second Cohort)


ParticipantsNon-participants
Mean (out of 4 marks maximum)2.752.35
Variance0.731.92
Observations1646
Df43
t Stat1.36
p-value0.09

Again, qualitatively the difference between the groups could be described as "good" (3 out of 4 marks) versus "fair" (2 out of 4 marks). Thus for both cohorts of students there was evidence of improved performance for those students who participated in the class exercise.

It is noted that the second cohort of non-participants performed better than the first cohort (2.35 versus 1.98 out of 4). This could be due to the better online support materials available by the second semester. The non-participants whilst not attending the face-to-face teaching session on referencing still had access to the online resources. This may also explain the narrowing of the gap between participants and non-participants in second semester compared to the first.

Both cohorts of students could be said to have 'passed' the written examination question on referencing, in that on average they performed at the level of two of the four reference items being correct. For a first year undergraduate unit this level may be considered adequate, but should not preclude striving to improve the student performance.

Discussion

The teaching and learning of referencing as a graduate skill is important for several reasons. It is an integral aspect of good scholarship and it promotes the acknowledgement of others' ideas and allows readers to find source material. It avoids plagiarism. It can also be used to sensitize students to the importance of ideas, intellectual capital, intellectual property, and copyright. As future workers in global knowledge-based economies it is important that graduates have well developed skills and knowledge to synthesise others' ideas and their own.

To develop better skills, referencing needs to be taught, practised, and assessed. Whilst this can be done in a specific part of the curriculum it is helpful if it is developed over the course of a student's degree as part of a graduate skills program, in much the same way as discipline skills are developed.

The Curtin Business School has set out to develop graduate skills as a strategic part of teaching and learning. This has occurred through a mixture of organisational (top-down) activity and teacher-led (bottom-up) activity. Naturally there are tensions as teachers fear the loss of discipline skills at the expense of more general graduate skills. There are however, opportunities for synergy illustrated by the example in this study.

Limitations, recommendations and future research

This study relied on a quasi-experimental design. The referencing skill was measured by the judge's assessment of the written answer to a question. There is future opportunity for a multiple measure that would include performance on continuous assessment items. The students self-selected for participation or otherwise and it is possible that better students self-selected for participation. A test of comparative overall exam performance could be used to include or exclude this possibility. There is no control over what other learning activities the non-participants engaged in so it is possible that they made good use of online resources however it is just as likely that participants took advantage of those resources.

The general recommendation is that if graduate skills are seriously considered to be distinguishing and useful qualities then they should be accorded a similar status to discipline skills. At the unit level of curriculum this means that there should be planned teaching and learning activities that provide multiple opportunities in lectures, tutorials, assignments, and examinations for students to learn and receive formative and summative feedback on their skills.

The focus in this study was on the unit level improvement in referencing skill. This could easily be extended to other graduate skills. Furthermore, the research can be extended beyond the unit level to examine improvement in graduate skills over the degree course. Here there is opportunity to use and develop tests like the Graduate Skills Assessment (GSA) (ACER, 2006). A student could use results to demonstrate their qualifications to prospective employers in the same way that grades on discipline based units are used. A university could use results to improve their graduate skills programs.

Conclusion

The study has demonstrated how a graduate skill can be taught, learnt, and assessed within a unit. In each of two semesters, one group of students undertook a class activity with feedback on their performance. The other group did not. A measurement of the skill based on a written examination revealed that the participants performed better on the skill. The study illustrates how a graduate skill can receive the same status as discipline skills in terms of teaching, learning and assessment.

More research on graduate skills programs is needed so that education of tertiary students is appropriately balanced and directed to the course objectives. The improved skills of graduates reflect the qualities of the interactions students have with teachers and resources over the three to four years of their degree. Within the framework of a graduate skills program, faculty need good examples and research to make informed decisions about how to balance and integrate discipline and graduate skills.

Acknowledgement

The author wishes to thank his colleagues Ms Daniella Driscoll (School of Information Systems) and Ms Julie Howe (Communication Skills Centre) for their support, and the students of Design 150 who participated in the research.

References

Cook, T. & Campbell, D. (1979). Quasi-experimentation: Design and analysis issues for field settings. Houghton Mifflin: Boston.

Curtin Business School (n.d.). Graduate Attributes and Professional Skills Program [viewed 20 Mar 2007] http://www.business.curtin.edu.au/index.cfm?objectid=9E965E72-0AE5-C5B3-DA175DCBBB4EF58F

Curtin University (n.d.). Referencing: Curtin University Library. Curtin Library and Information Service, [viewed 12 Jan 2009] http://library.curtin.edu.au/research_and_information_skills/referencing/

Curtin University (2008). Chicago author-date referencing 2008. Curtin Library and Information Service, [viewed 12 Jan 2009] http://library.curtin.edu.au/research_and_information_skills/referencing/chicago.pdf

Curtin University (2003). University Teaching and Learning Plan, 2003-2005, October 2003. [viewed 12 Jan 2009] http://lsn.curtin.edu.au/ssu/tlplan0305.pdf

Guthrie, R., McGowan, J. & de la Harpe, B. (2001). Professional skills? - 'When the going gets tough, the tough have a meeting'. In A. Herrmann and M. M. Kulski (Eds), Expanding Horizons in Teaching and Learning. Proceedings of the 10th Annual Teaching Learning Forum, 7-9 February 2001. Perth: Curtin University of Technology. http://lsn.curtin.edu.au/tlf/tlf2001/guthrie2.html

Le Heron, J. (2001). Plagiarism, learning dishonesty or just plain cheating: The context and countermeasures in Information Systems teaching. Australian Journal of Educational Technology, 17(3), 244-264. http://www.ascilite.org.au/ajet/ajet17/leheron.html

Mackenzie, N. & Knipe, S. (2006). Research dilemmas: Paradigms, methods and methodology. Issues In Educational Research, 16(2), 193-205. http://www.iier.org.au/iier16/mackenzie.html

Mertens, D.M. (2005). Research methods in education and psychology: Integrating diversity with quantitative and qualitative approaches. (2nd ed.) Thousand Oaks: Sage.

Murdoch University (2004). Graduate attributes at other universities. [viewed 20 Feb, 2007] http://www.tlc.murdoch.edu.au/gradatt/gaothers.html

Author: Dr Doug Atkinson, School of Information Systems, Curtin University of Technology, GPO Box U1987, Perth 6001, Western Australia. Email: d.atkinson@curtin.edu.au

Please cite as: Atkinson, D. (2009). Teaching, learning, and assessing a graduate skill. In Teaching and learning for global graduates. Proceedings of the 18th Annual Teaching Learning Forum, 29-30 January 2009. Perth: Curtin University of Technology. http://otl.curtin.edu.au/tlf/tlf2009/refereed/atkinson.html

Copyright 2009 . The authors assign to the TL Forum and not for profit educational institutions a non-exclusive licence to reproduce this article for personal use or for institutional teaching and learning purposes, in any format (including website mirrors), provided that the article is used and cited in accordance with the usual academic conventions.


[ Refereed papers ] [ Contents - All Presentations ] [ Home Page ]
This URL: http://otl.curtin.edu.au/tlf/tlf2009/refereed/atkinson.html
Created 15 Jan 2009. Last revision: 15 Jan 2009.