|Teaching and Learning Forum 2009 [ Refereed papers ]|
School of Information Systems
Curtin University of Technology
A teaching/learning intervention was designed and implemented by a teacher to improve the graduate skill of referencing for undergraduate commerce students studying web design. Over two semesters, 120 students were studied. In each semester, a "participant" group voluntarily undertook a class exercise during a lecture and received feedback from the teacher. A "non-participant" group chose not to undertake the exercise. In addition, all students received tuition at laboratories, assignment instructions, assignment assessment, and access to a library referencing resource. Referencing skill was later measured for both groups via a written exam question. Results indicated that the participant group achieved a superior level of referencing skill. Aside from the inference that teaching and learning a skill improves performance, the study demonstrates how a graduate skill can be taught, learnt and assessed at the unit level of a tertiary curriculum. The paper covers graduate skills, referencing, the teaching/learning design, the quasi-experimental research design, and the results.
The significance of teaching and learning graduate skills is that there are generic useful qualities that students develop over the course of a university degree that can be applied in the global workplace. These qualities, for example, communication skills, are generic and useful in the sense that they are required in any workplace or profession, and are useful over time. In contrast, some discipline specific knowledge and skills, for example, programming in assembly language, can have a narrow area of application and may become obsolete as technologies and contexts change.
Within Western Australia, Curtin University, Murdoch University (Murdoch University, 2004) and Edith Cowan University have developed graduate skill programs. At Curtin University the graduate attributes/graduate skills project began in the Curtin Business School (CBS) in 1998 (Guthrie, McGowan, & de la Harpe, 2001). In 2004 the University developed graduate attributes that were to be included in all Curtin courses (Curtin University, 2003). The high level goals of the Curtin Business School (CBS) program required students to graduate with skills in several categories. These categories included information literacy, information technology literacy, problem solving, decision making, teamwork, presentation, interaction, and writing (Curtin Business School, n.d.).
Within this research, the referencing skill was treated as a sub-skill of the categories of information literacy, and writing. The context was a web design unit where the author was the unit controller. An extract from the unit outline (Table 1) illustrates the referencing skill (statement 3) within the context of graduate skills outcomes.
|Graduate Skills Outcomes
Employers worldwide want graduates who have developed effective graduate skills. The CBS graduate skills program includes communication (writing, interpersonal interactions and cultural awareness, and presenting), critical and creative thinking (problem solving and decision making), team work, IT literacy and information literacy.
On successful completion of this unit, students should be able to demonstrate:
The rationale for pursuing referencing as a graduate skill was that a graduate who develops skills in referencing over time should make a better reader, writer, and researcher. They should be better able to judge the depth of research of a written work and make assessments about its quality. Literate graduates should in turn make better contributions to the knowledge economy.
Referencing (citing) and making a reference (citation) is where a writer acknowledges ideas as coming from other authors or sources and provides the publication details of the sources. The act of referencing includes providing short codes in-text (e.g., an author and date, or a number). These short codes then link to a longer ordered description of publication details (e.g., author, date, title, publisher and place of publication) either at the end of the text or at the bottom of the page. This enables a reader to know that an idea came from another source and allows the reader to find the source based on the publication details provided.
In the context of graduate skills, referencing can be considered a sub-skill of both written communication and information literacy as the two concepts are closely related. Good writing requires a synthesis of the existing literature with links so that the reader can ascertain the qualities of the research and have the opportunity to consult the sources. An information literate graduate needs to know how information is referenced so that they can manage their own writing, access other writer's sources, and judge the qualities of written materials.
Within the Curtin Business School, referencing was referred to under the skill categories of information literacy and writing:
Additionally, there was an increasing concern about plagiarism within the School and in the University system generally (Le Heron, 2001). Curtin University advocated an education strategy particularly for the first year units where students were often ignorant about referencing. The particular unit, involving web design, had a major assignment that required the design and development of a proposal document, a website, and a usability document. In early implementations of the unit there were numerous cases of plagiarism where students simply did not reference materials used in the website. To rectify the situation the original unit controller, and later the author who inherited the unit, instituted a more rigorous approach to the teaching and learning of referencing.
Firstly, to signal the importance of referencing and to encourage good scholarship, a mark worth 20% of the assignment was allocated. In the assignment instructions a section dealing with the consequences of plagiarism was also included. To improve the learning of referencing, a laboratory exercise (Table 2) was devoted to the task that required students to reference some material they had used in their website. Students were also directed to the referencing examples provided by the Curtin Library (Curtin University, n.d.; 2008).
To ensure that you are confident with the referencing required, as a group prepare a reference (using the correct Harvard referencing format) for:
Text that may be used in your website (i.e. use a subscript or superscript number at the end of the sentence/paragraph and provide the footnote containing the reference at the bottom of that page).
An image that may be used in your website (i.e. using an Alt Tag to contain the footnote number and then the footnote containing the reference at the bottom of that page).
Ask your lecturer / unit leader to check your work.
Harvard Referencing, 2006,
When the author inherited the unit, a decision was made to promote referencing to the status of a graduate skill, given that it was important throughout the degree and beyond. A statement was made in the unit outline which was then reinforced verbally at the first lecture. One issue that had been generally noted was the lack of attention to online referencing. The nature of the unit being website design represented an opportunity for dealing with this in an appropriate context. Hence there was a synergy between the discipline demands of the unit and the generic skill.
After several semesters, the author decided that a formal participative exercise held in the lecture (Table 3) would reinforce the skill development. The advantage of the lecture was that the author as lecturer had control over the teaching rather than being largely dependent on sessional staff who took the laboratories. At this stage then, the teaching and learning of referencing was embedded throughout the unit, via the unit outline, the assignment instructions and assessment, a lecture exercise, and a laboratory exercise.
|Imagine that today you have accessed the Amazon.com website and have used some material from the site.
Finally, a decision was made to include the assessment of referencing in the formal examination. The Curtin Business School (CBS) had been pushing to reinforce the formal teaching and assessment of the graduate skills so the author felt that if assessment was to be treated as seriously as discipline skills then it should take place not only in the continuous assessment but also in the final written examination. Thus within the examination, a question, worth 3 marks out of 50, required students to provide an in-text and an end-of-text reference for a website.
During the implementation of the teaching initiatives, the author saw an opportunity to formally analyse a link between the teaching of the skill and the performance by the students. Specifically there was an opportunity to see whether participation in the lecture exercise brought about an improved performance as measured by the examination assessment. The lecture exercise was controlled by the author so teacher variation was eliminated. Furthermore, there were naturally occurring groups of participants and non-participants because the lecture attendance was voluntary.
This background thus gave rise to the formal objective of assessing the teaching intervention to improve students' referencing skills based on a quasi-experimental design. This is elaborated in the following research method and design section.
Within this study, the design aspects were a measurement of performance, and treatment and control conditions. A non-experimental aspect was the self-selection of subjects into the treatment and control conditions. Furthermore, the researcher was interacting with the subjects in the experiment so there were possibilities of a Hawthorne effect and possible researcher effects on the performance measurements. Despite these limitations, the design is defended as being useful in pursuing the objective of assessing the teaching intervention to improve students' referencing skills.
The researcher was the teacher who had a vested interest in improving the students' referencing skills. The researcher/teacher was involved in conducting the activity, receiving feedback during interaction with students, and improving the activity for the future. The students had a vested interest in improving their referencing skills for both the satisfaction of learning a skill, and receiving better marks in assessment activities.
The testing of the teaching intervention was done via statistical measurement and analysis taking advantage of the natural occurrence of two groups of students - "participants" and "non-participants". Participants were those who attended a class where the main teaching of the referencing skill was carried out. Non-participants were those who chose not to attend. These groups were an approximation of a treatment group and a control group respectively. The approximation refers to the fact that the students self selected into the groups rather than being randomly assigned as in a pure experiment. This then opens the possibility that differences between the groups were causing differences in outcome measures, rather than the differences being purely due to the treatment.
The measurement of performance of the referencing skill was based on the author's assessment of a written answer to a question on referencing a web site in the end of semester examination. This activity was done some time after the official exam marking. Here it was not subject to the usual stress of a tight deadline and the fine assessment of one question out of a much larger examination paper. The students' written answers were assessed by the author via a marking guide out of a maximum total of four marks. A mark was allocated for the author, the year, the date of retrieval, and the Uniform Resource Location (URL). A comparison of this mark was made between the two groups (participants and non-participants) using a T-test of means.
It was expected that the participant group, having received the extra tuition during the class referencing exercise, would perform better than the non-participant group in the final examination.
Thus the hypotheses tested were:
Null hypothesis: The participant and non-participant groups have the same performance level of the referencing skill; andThe students were enrolled in a first year unit of a commerce degree majoring in information systems. The unit was concerned with user interface and web design. The study was run over two semesters of tuition in 2006. The measurement of referencing skill performance was based on students who sat the final examination. A total of 120 students were involved. There were 58 students in the first semester cohort (Table 4) and 62 students in the second semester cohort (Table 5). In the first semester cohort, 11 students (participants) voluntarily completed a class referencing exercise. The other 47 students (non-participants) chose not to attend or did not complete the class activity. In the second semester cohort, 16 students (participants) voluntarily completed a class referencing exercise. The other 46 students (non-participants) chose not to attend or did not complete the class activity.
Alternative hypothesis: The participant group achieved a higher performance level of the referencing skill than the non-participant group.
The participation rate of 23% was not unusual as the class referencing exercise was held during lecture time in the latter part of the semester. Lecture attendance in the School is not compulsory. In the latter part of the semester many students decide not to attend lectures and rather spend time on assignments. Additionally, the lecture and laboratory materials are provided online, and many students perceive these as an adequate substitute for attending lectures. The author's own perspective as a teacher is that extra materials are covered at lectures, and as in the class referencing exercise, non-attendees forgo the live interaction with potential negative consequences for their learning.
At the time of the study, the particular referencing style advocated by the Curtin Business School and supported by the University was the Harvard (author-date) style. This has since been modified to the Chicago (author-date) style. Students had online access to a document provided by the Library (similar to Curtin University, 2008). In addition to an overview of referencing this document provided nine pages of examples including four examples of referencing web documents (Curtin University, 2008, p. 8). Assignment and lab documents in the unit made explicit reference to this resource (Table 2), and the lecturer made numerous verbal references.
The class activity was designed and conducted as follows. Firstly, there was a ten minute interactive discussion of the importance of referencing covering issues such as scholarship, avoiding plagiarism, and allowing readers to find sources. Then each participant was handed a hardcopy one page pro-forma (Table 3) with questions about a referencing task. The lecturer projected a public screen displaying the home page of a famous web site, for example, Amazon, Yahoo!, or Google. The participants were then asked to complete the questions on the pro-forma. The lecturer then projected an answer to the questions, gave some verbal feedback, and asked the participants to correct and mark their work. Participants were then requested to hand up their completed pro-forma. The whole activity took about 30 minutes.
In the same week of semester, the student cohort received a class exercise at the laboratory where they were required to do references to text and images for their website group assignment. This exercise was more tailored to the specific assignment then the class exercise above.
Following submission of the assignment in the latter weeks of the semester, students received written feedback on the marking criteria including comments about the adequacy and quality of the referencing. The final activity for the students was the formal written examination for the unit.
The written examination was a routine assessment method for the undergraduate students. With the exception of some project units, all undergraduate units taught by the School have a written centrally scheduled examination conducted at the end of the teaching semester. An examination will generally account for at least 50% of the student's total grade in the unit - the rest being based on continuous assessment. In this particular unit the examination was 50%, the mid-semester test was 20%, and the major website assignment was 30%. The latter was group work whereas the examination and the test were individually assessed.
The examination question on referencing was similar in nature to the activity undertaken in class. Furthermore, students had been assessed on referencing in their website assignment. Thus students were being examined on a skill they had been given opportunity to practice and improve. However, students were not given specific direction that a referencing question would appear in the examination, thus there was an element of surprise.
The next section details the statistical results and the analysis of the students' written examination answers.
|Mean (out of 4 marks maximum)||2.82||1.98|
For the first cohort of students, the participants performed better than the non-participants in the final examination (Table 4). On the specific referencing question, the participants averaged 2.82 marks out of a maximum of 4 marks. The non-participants averaged 1.98 marks. This was statistically significant at the 0.05 significance level (p-value=0.03).
Qualitatively, the difference between the groups could be described as "good" (3 out of 4 marks) versus "fair" (2 out of 4 marks). For students this is an indicator that undertaking the class exercise and receiving feedback improves their skill level. For teachers this is an indicator that formally teaching a graduate skill improves student learning outcomes.
For the second cohort of students, again the participants performed better than the non-participants in the final examination (Table 5). On a specific referencing question, the participants averaged 2.75 marks out of a maximum of 4 marks. The non-participants averaged 2.35 marks. This was statistically significant at the 0.10 significance level (p-value=0.09).
|Mean (out of 4 marks maximum)||2.75||2.35|
Again, qualitatively the difference between the groups could be described as "good" (3 out of 4 marks) versus "fair" (2 out of 4 marks). Thus for both cohorts of students there was evidence of improved performance for those students who participated in the class exercise.
It is noted that the second cohort of non-participants performed better than the first cohort (2.35 versus 1.98 out of 4). This could be due to the better online support materials available by the second semester. The non-participants whilst not attending the face-to-face teaching session on referencing still had access to the online resources. This may also explain the narrowing of the gap between participants and non-participants in second semester compared to the first.
Both cohorts of students could be said to have 'passed' the written examination question on referencing, in that on average they performed at the level of two of the four reference items being correct. For a first year undergraduate unit this level may be considered adequate, but should not preclude striving to improve the student performance.
To develop better skills, referencing needs to be taught, practised, and assessed. Whilst this can be done in a specific part of the curriculum it is helpful if it is developed over the course of a student's degree as part of a graduate skills program, in much the same way as discipline skills are developed.
The Curtin Business School has set out to develop graduate skills as a strategic part of teaching and learning. This has occurred through a mixture of organisational (top-down) activity and teacher-led (bottom-up) activity. Naturally there are tensions as teachers fear the loss of discipline skills at the expense of more general graduate skills. There are however, opportunities for synergy illustrated by the example in this study.
The general recommendation is that if graduate skills are seriously considered to be distinguishing and useful qualities then they should be accorded a similar status to discipline skills. At the unit level of curriculum this means that there should be planned teaching and learning activities that provide multiple opportunities in lectures, tutorials, assignments, and examinations for students to learn and receive formative and summative feedback on their skills.
The focus in this study was on the unit level improvement in referencing skill. This could easily be extended to other graduate skills. Furthermore, the research can be extended beyond the unit level to examine improvement in graduate skills over the degree course. Here there is opportunity to use and develop tests like the Graduate Skills Assessment (GSA) (ACER, 2006). A student could use results to demonstrate their qualifications to prospective employers in the same way that grades on discipline based units are used. A university could use results to improve their graduate skills programs.
More research on graduate skills programs is needed so that education of tertiary students is appropriately balanced and directed to the course objectives. The improved skills of graduates reflect the qualities of the interactions students have with teachers and resources over the three to four years of their degree. Within the framework of a graduate skills program, faculty need good examples and research to make informed decisions about how to balance and integrate discipline and graduate skills.
Curtin Business School (n.d.). Graduate Attributes and Professional Skills Program [viewed 20 Mar 2007] http://www.business.curtin.edu.au/index.cfm?objectid=9E965E72-0AE5-C5B3-DA175DCBBB4EF58F
Curtin University (n.d.). Referencing: Curtin University Library. Curtin Library and Information Service, [viewed 12 Jan 2009] http://library.curtin.edu.au/research_and_information_skills/referencing/
Curtin University (2008). Chicago author-date referencing 2008. Curtin Library and Information Service, [viewed 12 Jan 2009] http://library.curtin.edu.au/research_and_information_skills/referencing/chicago.pdf
Curtin University (2003). University Teaching and Learning Plan, 2003-2005, October 2003. [viewed 12 Jan 2009] http://lsn.curtin.edu.au/ssu/tlplan0305.pdf
Guthrie, R., McGowan, J. & de la Harpe, B. (2001). Professional skills? - 'When the going gets tough, the tough have a meeting'. In A. Herrmann and M. M. Kulski (Eds), Expanding Horizons in Teaching and Learning. Proceedings of the 10th Annual Teaching Learning Forum, 7-9 February 2001. Perth: Curtin University of Technology. http://lsn.curtin.edu.au/tlf/tlf2001/guthrie2.html
Le Heron, J. (2001). Plagiarism, learning dishonesty or just plain cheating: The context and countermeasures in Information Systems teaching. Australian Journal of Educational Technology, 17(3), 244-264. http://www.ascilite.org.au/ajet/ajet17/leheron.html
Mackenzie, N. & Knipe, S. (2006). Research dilemmas: Paradigms, methods and methodology. Issues In Educational Research, 16(2), 193-205. http://www.iier.org.au/iier16/mackenzie.html
Mertens, D.M. (2005). Research methods in education and psychology: Integrating diversity with quantitative and qualitative approaches. (2nd ed.) Thousand Oaks: Sage.
Murdoch University (2004). Graduate attributes at other universities. [viewed 20 Feb, 2007] http://www.tlc.murdoch.edu.au/gradatt/gaothers.html
|Author: Dr Doug Atkinson, School of Information Systems, Curtin University of Technology, GPO Box U1987, Perth 6001, Western Australia. Email: firstname.lastname@example.org
Please cite as: Atkinson, D. (2009). Teaching, learning, and assessing a graduate skill. In Teaching and learning for global graduates. Proceedings of the 18th Annual Teaching Learning Forum, 29-30 January 2009. Perth: Curtin University of Technology. http://otl.curtin.edu.au/tlf/tlf2009/refereed/atkinson.html
Copyright 2009 . The authors assign to the TL Forum and not for profit educational institutions a non-exclusive licence to reproduce this article for personal use or for institutional teaching and learning purposes, in any format (including website mirrors), provided that the article is used and cited in accordance with the usual academic conventions.