Teaching and Learning Forum 2010 Home Page

Category: Research

Teaching and Learning Forum 2010 [ Refereed papers ]
The impact of diverse student backgrounds and flexible delivery modes on assessment outcomes

Jianhong (Cecilia) Xia and Shelley Yeo
Faculty of Science and Engineering
Curtin University of Technology

Managing the diversity of student backgrounds in today's complex higher education classrooms is a challenging task for lecturers. This paper reports on action research involving a complex nested group of units in Geographic Information Science (GIS) delivered to a diverse student body, including international and domestic students from different disciplines studying in on-campus, online and distance education modes. GIS is a technology-dependent discipline of relevance to a wide variety of other disciplines including, for example land management, urban development or health sciences, to facilitate land-use planning, map disease distribution, analyse spatial patterns of crime or model urban growth. The concern of the lecturer was that adaptation of the learning activities necessary for accommodating flexible modes of delivery and student backgrounds would impact on student assessment outcomes. The research findings indicate that while there were some significant differences in assessment outcomes associated with student backgrounds and discipline, no differences could be ascribed to, for example, language background of students. This paper discusses the study and seeks explanations for the results. The implications for teaching and further action research with this unit in order to sustain student numbers and teaching and learning standards in the future are also presented.


Introduction

Many higher education classrooms today scarcely resemble those of even 10 years ago. The enrolment of international students (both onshore and offshore), student expectation of greater flexibility in delivery, as well as institutional demands for improvement in efficiency, all place enormous demands on the skills and time of an academic (Dunn, Morgan, O'Reilly, & Parry, 2004). Teachers, whether based onshore or offshore, need a multitude of skills including knowledge of different teaching and learning styles, the ability to use teaching strategies that are inclusive of all students, an awareness of the cultural dimension of teaching and learning, and the ability to assess students fairly and equitably (Braddock & Loxton, 2003; Farkas-Teekens, 1997; James, McInnis, & Devlin, 2002).

This paper reports on an action research project undertaken by an early career lecturer from a non-English speaking background (NESB) who is teaching a group of units which incorporates all of the above complexities. The research focuses on determining the impact of diverse student backgrounds and flexible delivery modes on assessment outcomes with the aim of being fair to all students. Lessons and insights will be derived from this study and future action will be designed in order to sustain the number of students and standard of teaching and learning in this class.

Literature review

Some challenges of assessment in higher education

Assessment is a high stakes and increasingly more accountable function of universities. The process of assessment in a university is embodied in its policies and procedures, the design of assessment tasks and how they are administered, the process of preparing assessors and subsequently their judgement of the quality of student performance on assessment tasks, the aggregation and validation of results and grading processes and finally affirming student progression or certification. A growing emphasis on academic standards and accountability in Australia is fuelling the need for universities to be more accountable for the way they conduct and evaluate assessment of student learning (James et al., 2002).

Universities delivering programs in transnational settings are also subject to consensual or legislative requirements in relation to quality control and quality assurance assessment processes (AVCC, 2005). Higher education institutions in Australia are subject to the National Protocols for Higher Education Approval Processes (MCEETYA, 2000) that refer to the expectation of "equivalent" standards for Australian universities operating offshore. While the notion of equivalence is not fully understood, it can be taken that students' experiences, if not identical, should afford them the same opportunities to learn and to demonstrate their knowledge as students in the home university (DEST, 2005).

Flexible delivery

While there appears to be no consensus on the meaning of the term flexible delivery, it generally means offering students asynchronous access to a unit of study, learning materials, learning activities and assessment through a variety and often combination of means-face-to-face, distance and partially or fully online-and often at the students' choice (Ling et al., 2001). Online delivery options are usually mediated by a learning management system, and may include one or more of CD-ROM, online materials, mobile technologies, video or audio files (e.g. iLectures), discussion boards and video- or teleconferencing. From a university perspective, offering programs in flexible modes may be seen as a risk management strategy to ensure the sustainability of programs should there be actions or circumstances which threaten or interrupt on-campus face-to-face delivery of units.

Fair assessment

Attempts to implement fair and inclusive assessment, which accommodates different student groups and cultures, can further complicate the work of an academic. Fair assessment is not so much about ensuring equal assessment for all students but about equity, or equality of opportunity for all students to demonstrate their learning regardless of their background or other irrelevant factors (Elton & Johnston, 2002).

Gipps (1995) uses the term equity which is interpreted as the spirit of justice: "The concept of equity in assessment as we use it implies that assessment practice and interpretation of results are fair and just for all groups" (p. 273). Gipps' focus on equity in relation to assessment includes both the definition as well as the practices of assessment. Assessment opportunities should be equally available to all groups and presented in such a way that all groups feel able to participate fully. This, however, does not imply equality of outcome because other factors, for example student motivation and esteem, and teacher behaviour and expectations, may impact on student achievement. On the other hand, two of the key steps involved in ensuring fair assessment are the appropriate interpretation of students' results (in relation to the tasks and assessment standards) and the evaluation of assessment outcomes (in relation to student and teacher expectations) (Suskie, 2002).

International students and assessment

Assessment of students in a transnational context can be made more difficult by students' inability to understand or their inexperience with Western style assessment methods or instructions which can leave them insecure and anxious about their learning (Pyvis & Chapman, 2004). Offshore and/or international onshore students often face difficulties related to communication of assessment tasks, receipt of integrated formative feedback and tasks that are not sufficiently tailored to the students' needs (Dunn et al., 2004). Scarino, Crichton and Padametre (2006) also emphasise the importance of language and culture in the construction of meaning and the consequent inadequacy of assuming a direct 'translation' or easy communication of assessment procedures and ideals in the transnational setting. In online transnational learning, these cultural differences may be further compounded by the abstract nature of communication with the result that students from countries with cultures different to the culture embedded in the materials used for online learning may be disadvantaged (Goodfellow, Lea, Gonzalez, & Mason, 2001).

Background to the study

A geographic information system (GIS) is a computer system for capturing, storing, querying, analysing, and displaying geo-referenced data. Geo-referenced data describes both the locations and characteristics of spatial features, such as roads, rivers, forests and towns (Chang, 2009)

GIS technology has been applied extensively to a variety disciplines such as land management, urban development, transportation management and health sciences to facilitate land-use planning, analyse spatial patterns of crime or model urban growth or map the distribution of disease. Students enrol in GIS units to learn about the structure, function and use of geographic information systems, the input, management, analysis and presentation of geographic data and relationships, the storage and manipulation of vector and raster data, digital terrain modelling, the integration and management of spatial and attribute data, including applications and examples. Students must be competent in computing and mathematics.

Student backgrounds

The units and students in this study are described below. The students were all enrolled in one of three introductory, co-taught GIS units i.e. GIS 181/581/212. These units are all taught by the researcher. Given the diversity of students in the units, the concern of the lecturer was that some students may be disadvantaged in terms of their learning and assessment. The classes consisted of a mixture of face-to-face instruction and assessment and online content, tutorials and assessment, but each was not available to all students. The students studying offshore were unable to attend on-campus, lectures and tutorials and so undertook online lectures recorded by screen capture tools and simulated tutorials instead. They therefore did not have the same opportunities as their Bentley-based counterparts to discuss and gain assistance with problem solving, and feedback on their assessments. However, the Bentley-based students were a mixture of mature aged, international and domestic students, each with a different motivation for, an interest in studying, the units. The question that arose for the researcher was: Did the assessment results of the students indicate that any group (international, domestic, on- or off-shore, mature-aged or school leavers, or course of study) outperformed any of the other groups.

Unit assessments

To cater for the diversity in the class, the assessment pattern was as follows: The mid-semester test consisted of multiple choice and short answer questions. The final examination included short answer and discussion questions. The first assignment was designed to enable students to apply the GIS vector data analysis tool to decide the hypothetical suitability of their university to hold a Formula One car race in 2012. The aim of the second assignment was to assess students' ability to use the GIS raster tool to set aside an area for a nature reserve at the La Selva Biological Station, which is located approximately 50 kilometres northeast of the capital city of San Jose, Costa Rica, in order to preserve the flora and fauna of the region. Both assignments required students to conduct a sequence of data analyses and write a report. There were thus a variety of assessments and assessment types to enable comparisons between different students groups in the class.

Methodology

The methodology employed is action research. Action research is a cyclic process by which change and understanding can be pursued at the same time. The reflection is used to review the previous action and plan the next on. The understanding allows more informed change and at the same time is informed by that change (Dick, 1997; Kember & Kelly, 1993). The primary focus of action research is on solving real problems (O'Brien, 1998). As O'Brien points out, its methods are various and adapted to suit the situation under study as it is used in real situations, rather than in contrived, experimental studies.

In this situation, the problem under investigation is the equivalence of assessment outcomes for different sub-groups of students in the GIS nested units GIS 181, GIS 581 and GIS212 which have multiple delivery modes and locations, and adaptations of assessment that are necessary to accommodate the different delivery modes. The outcomes of this aspect of the research will be used to plan for on-going improvement in the units.

We developed four null hypotheses (H0) to test student differences on assessment results in four dimensions: location of study, student language background, student maturity and discipline of study. Each dimension represents a potential barrier to equality of learning and assessment.

Mode of studyH0There is no difference between the assessment results of on-campus and distance education students.
Student language
background
H0There is no difference between the assessment results of domestic and international students.
Student maturityH0There is no difference between the assessment results of undergraduate and postgraduate students.
Discipline of studyH0There is no difference between the assessment results of students from different disciplines (courses).

The statistical comparison method adopted were unpaired t-tests to compare two groups and one-way ANOVA with post hoc analysis to compare three unmatched groups. The null hypotheses will be rejected if the p-value is less than 0.05.

Results and discussion

In this section, students' assessment results for the GIS units are compared on the four dimensions outlined above, insights derived from the analysis are discussed and potential actions for change and improvement in teaching and learning are addressed.

Table 1 summarises the mean scores for different subgroups of students on each of the assessment items. Tables 2 and 3 present statistical data showing any significant differences that were found.

Table 1: Mean results for each sub-group of students on assessment items

Sub-groupsMean assessment results
NQuizzesAssignmentMid-semester
test
Final
examination
ModeOn-campus1095.314.65.838.5
Distance education215.514.26.145.9
LevelUndergraduate1175.114.45.738.8
Postgraduate136.815.47.147.5
BackgroundDomestic965.314.75.939.4
International345.3145.640.3
DisciplineSurveying 725.614.95.839.7
GIS 166.615.56.947.9
Geology 424.313.55.438.6

Table 2: Statistically significant results of assessment comparisons
of undergraduate and postgraduate students using t-tests



NMeanSt. DevtSig (2-tailed)
Quiz totalUndergraduate1175.11.6-3.60.000
Postgraduate136.81.6
Midsemester testUndergraduate1175.71-4.70.00
Postgraduate137.10.8
Final examinationUndergraduate11738.89.2-3.40.001
Postgraduate1347.55.4
On-campus10938.59.2-3.50.001
Distance2145.97.3

Table 3: Statistically significant results of assessment comparisons
between GIS, Surveying and Geology students using ANVOA


Sum of
squares
Mean
square
FSig.GIS - SurveyGIS - Geology
Mean
difference
Sig.Mean
difference
Sig.
Quiz total79.339.618.201.10.0112.30
Mid-semester test29.114.614.501.101.60
Final examination1242.76621.387.970.0019.50.0019.30.001

Mode of study

We found no difference between on-campus and distance education students in terms of assessment, except on the final examination. The distance education students' final examination results (mean = 45.9 out of 60) were significantly higher than the Bentley-based on-campus students (mean = 38.9, p< 0.00). While we had been concerned that the students studying by distance and online might be disadvantaged by the lack of opportunity for discussion, it appears that for this year group, there was no significant disadvantage.

The distance education students are supported with a web-based teaching tool. All the lectures are screen captured and synchronised with auto file. The students were also sent weekly emails which addressed important issues related to lectures and tutorials, and were encouraged to send emails to the lecturer if they encountered any problems regarding the online lectures and tutorials. It was apparent during the semester that distance students were more active in using the discussion forum in WebCT to communicate with and help each other. It is possible that this community-based teaching and learning tool served to encourage and motivate the distance education students to develop their self-learning skills.

While on-campus students can obtain direct help from tutors and other students when they have problems, distance education students are relatively more isolated. When they have faced difficulties in tutorials or assignments, distance students have had to be more proactive in seeking help to resolve their problems. Although they may have, at times, they faced a lot of frustration, they also appear to have developed better problem-solving skills than on-campus students.

Student language background

We found no difference between the assessment results of domestic and international students. The assignment in particular is a good indicator of students' ability to express complex ideas in English. Any potential language barriers faced by international students appear to have not been an issue for these students. Anecdotal observations from marking assessments suggest that international students tend to lose more marks due to poorer English skills (grammatical errors and unclear English expressions) but they tend to make this up by a better performance in other aspects of the assessment.

Student maturity

We found differences in assessment results of postgraduate students compared with undergraduate students on all but one assessment (the assignment). Postgraduate students' results on the quizzes (mean = 6.8), mid-semester test (mean = 7.1) and final examination (mean = 47.5) were greater than the results of the undergraduate students (means = 5.1, 5.7 and 38.8 respectively, p<0.00). The majority of postgraduate students are mature aged and tend to be very self-motivated. Seventy percent of them are working as GIS officers or cartographers. They have returned to university to extend their knowledge and skills, and to apply them to solve certain GIS problems encountered at work. For example, one student sent an email to thank us and indicated that she was able to apply new knowledge immediately to her work, and had become more expert in metadata management in her work place. The fact that there was no difference between postgraduate and undergraduate students on the assignments, which are tests of problem-solving skill, indicate that the postgraduate students do not necessarily have such skills when they start the unit.

Discipline of study

Overall, we found significant differences in assessment results of the different discipline groups. In general, the GIS students' results were significantly greater than the Surveying students' results (final exam mean difference= 9.5 p=0.000, Mid-semester test mean difference= 1.1 p=0.000, Quiz mean difference= 1.1 p=0.011), which were in turn significantly greater than the Geology students' results (final exam mean difference= 9.3 p=0.001, Mid-semester test mean difference= 1.6 p=0.000, Quiz mean difference= 2.3 p=0.000).

We can only speculate on the reasons for these differences, particularly as we do not know if there were other differences between these groups which might account for the results. Students studying GIS or Surveying may be more self-motivated because the unit content bears a closer relationship to the core business of their course, compared with the Geology students. For Geology students, the unit is offered by a Department other than their own. In addition, there is a higher requirement for mathematics and computing skills which may not be as well emphasised or needed in the Geology program.

This paper reflects the concerns of some academics who teach today's complex higher education classrooms. The academic who is the subject of this paper simultaneously teaches in flexible mode (online and face-to-face) to students who are both undergraduate and postgraduate, from both English and non English speaking backgrounds and who are in three different degree programs. A review of these students' assessment results has been used to examine whether or not there may be unfairness associated with flexible delivery and student diversity. The adaptation of delivery method and assessment for the offshore, distance education students appear to have had little effect on the students' results, thus giving some confidence that these methods are at least equivalent as experienced by the students. There is some evidence that NESB students, despite their struggles with language, have performed at least as well as their English-speaking counterparts, although this is not to say that they have not been disadvantaged by having to learn and perform assessment in English. The most pervasive difference is in the results of students enrolled in different degree programs, and we attribute this more to the motivation and past or concurrent experiences of the students concerned rather than differences in teaching or assessment methods or materials resulting from the imperative for flexible delivery. The current work practices and perhaps maturity of the postgraduate students may explain the apparent advantage that they have over the undergraduate students. Thus, we are of the view that for this cohort of students there is little evidence of unfairness associated with delivery or assessment methods. However, our observations have led us to consider future improvements.

Next actions

The lessons learnt from this study, and future actions to address them are that:

Summary

This paper reports on an action research study designed to help the lecturer understand the impact of diverse of student backgrounds and flexible delivery modes on assessment outcomes for students in a nested group of GIS units at Department of Spatial Sciences, Curtin University of Technology. Assessment results were compared for nine sub-groups of students in four different dimensions: location, language, maturity and discipline of study. We found significant differences in assessment results of undergraduate and postgraduate students, and between distance and on-campus students. Lessons and insights derived from this research will guide further actions to improve the standard of the teaching and learning of this unit

References

AVCC (2005). Provision of education to international students: Code of practice and guidelines for Australian universities. Canberra, Australia. Retrieved 7/08/07 from http://www.universitiesaustralia.edu.au/documents/publications/CodeOfPracticeAndGuidelines2005.pdf

Braddock, R., & Loxton, J. (2003). Impacts of internationalisation on teaching and learning. Paper presented at the The Learning Conference, Institute of Education, University of London, 15-18 July, 2003. http://lc03.commongroundconferences.com/ProposalSystem/Presentations/P000823.html

Chang, K. (2009). Introduction to Geographic Information Systems (5th ed.). Sydney: McGraw-Hill.

DEST (2005). A National Quality Strategy for Australian Transnational Education and Training: A Discussion Paper. Canberra: Commonwealth of Australia. http://pandora.nla.gov.au/pan/53520/20051021-0000/aei.dest.gov.au/AEI/GovernmentActivities/QAAustralianEducationAndTrainingSystem/QualStrat_pdf.pdf

Dick, B. (1997). Action learning and action research [Electronic Version]. Retrieved 13/10/09 from http://www.scu.edu.au/schools/gcm/ar/arp/actlearn.html

Dunn, L., Morgan, C., O'Reilly, M., & Parry, S. (2004). The student assessment handbook. London: Routledge Falmer.

Elton, L., & Johnston, B. (2002). Assessment in Universities: A critical review of research. [verified 24 Jan 2010] http://eprints.soton.ac.uk/59244/01/59244.pdf

Farkas-Teekens, H. (1997). A profile of the ideal lecturer for the international classroom. In H. Farkas-Teekens & M. van der Wende (Eds.), Teaching in the International Classroom. Nuffic papers 8. Amsterdam: Nuffic.

Gipps, C. (1995). What do we mean by equity in relation to assessment? Assessment in Education: Principles, Policy & Practice, 2(3), 271-281.

Goodfellow, R., Lea, M., Gonzalez, F., & Mason, R. (2001). Opportunity and e-quality: Intercultural and linguistic issues in global online learning. Distance Education, 22(1), 65-84.

James, R., McInnis, C., & Devlin, M. (2002). Assessing learning in Australian universities: Ideas, strategies and resources for quality in student assessment. Melbourne: Centre for the Study of Higher Education. http://www.cshe.unimelb.edu.au/assessinglearning/

Kember, D., & Kelly, M. (1993). Improving teaching through action research. Campbelltown, NSW: Higher Education Research and Development Society of Australasia.

Ling, P., Arger, G., Helen Smallwood, H., Ron Toomey, R., Denise Kirkpatrick, D., & Ian Barnard, I. (2001). The effectiveness of models of flexible provision of higher education. Canberra: DETYA. http://www.dest.gov.au/sectors/higher_education/publications_resources/profiles/
archives/models_of_flexible_provision_higher_education.htm

MCEETYA (2000). National protocols for higher education approval processes. Canberra: Ministerial Council on Education, Employment, Training and Youth Affairs. http://www.mceetya.edu.au/mceecdya/national_protocols_for_higher_education_mainpage,15212.html

O'Brien, R. (1998). An overview of the methodological approach of action research. Retrieved 29/9/09, from http://www.web.ca/~robrien/papers/arfinal.html

Pyvis, D., & Chapman, A. (2004). Student experiences of offshore higher education: Issues for quality. AUQA Occasional Publications Number 3. Melbourne: Australian Universities Quality Agency. http://www.auqa.edu.au/files/publications/pyvis_chapman_report.pdf

Scarino, A., Crichton, J., & Papademetre, L. (2006). A framework for quality assurance in the development and delivery of offshore programs in languages other than English. London: The Observatory on Borderless Higher Education.

Suskie, L. (2002). Fair assessment practices: Giving students equitable opportunities to demonstrate learning [Electronic Version]. Adventures in Assessment, 14 from http://sabes.org/resources/publications/adventures/vol14/14suskie.htm

Authors: Jianhong (Cecilia) Xia and Shelley Yeo, Faculty of Science and Engineering, Curtin University of Technology.
Email: c.xia@curtin.edu.au, s.yeo@curtin.edu.au

Please cite as: Xia, J. & Yeo, S. (2010). The impact of diverse student backgrounds and flexible delivery modes on assessment outcomes. In Educating for sustainability. Proceedings of the 19th Annual Teaching Learning Forum, 28-29 January 2010. Perth: Edith Cowan University. http://otl.curtin.edu.au/tlf/tlf2010/refereed/xia.html

Copyright 2010 Jianhong (Cecilia) Xia and Shelley Yeo. The authors assign to the TL Forum and not for profit educational institutions a non-exclusive licence to reproduce this article for personal use or for institutional teaching and learning purposes, in any format, provided that the article is used and cited in accordance with the usual academic conventions.


[ Refereed papers ] [ Contents - All Presentations ] [ Home Page ]
This URL: http://otl.curtin.edu.au/tlf/tlf2010/refereed/xia.html
Created 23 Jan 2010. Last revision: 24 Jan 2010.