Teaching and Learning Forum 2000 [ Proceedings Contents ]

Assessing the assessment criteria: Students' opinions

Tekle Shanka
School of Marketing
Curtin University of Technology
    This paper presents the findings of students' assessment of the oral presentation assessment criteria being used in tutorials in one of the common core units in Business School. The criterion currently in use contains 14 items of assessment grouped into three headings, namely clarity (5 items), content (6 items) and format and structure (3 items). A one page questionnaire was distributed in tutorial classes. Participants were asked to indicate their opinion of the 14 items of assessment on a 7 point Likert scale from 1 "least important" to 7 "most important". A total of 318 completed questionnaires were received and analysed. Factor analysis with significant factor loading of .30 resulted in four headings (factors) which together explain 61.2% of variances. The first factor, concepts, and the second factor, resources, each consisted of 4 items. Factor 3, named non-verbal communications, and the fourth factor named clarity, each consisted 3 items.

    ANOVA tests were run to determine significant differences based on demographics. Significant differences were noted in respect of two factors, namely resources and non-verbal communications. In respect of resources the difference was between two age groups, 17-19 years and 30+ years. The mean score for 17-19 year olds was 5.06 while the 30+ year olds scored a mean of 5.59 In respect of non-verbal communications the difference was between Australian students and other students. The Australian students' mean score for the factor was 5.02 while that of other students was 5.38. Apart from the two demographic characteristics no significant difference was shown in respect of other demographics characteristics such as gender, course of study and year of study. Detailed analysis and implications are presented.

Teaching and Learning Forum 2000 Home Page


Introduction

Assessments of students' oral presentations have widely been used. Oral presentations are intended to broaden communication skills of students. Such skills are required by students in employment and will also influence their career development and will affect employers' views on the appropriateness of the business courses (Hughes and Large, 1993). Assessment criteria have widely been used in tutorials to enhance the quality of student learning in tutorials through the process of peer assessment. Peters (1996) writes that there was a considerable support for ensuring that the criteria by which assessment is undertaken were made public thus sharing the responsibility of assessment with learners. Pond and ul-Haq (1997) reiterated that based on three years of development and study over 150 students the result of the student feedback exercise shows that students considered the exercise to be effective for learning, group work and to have been "useful".

Dochy and McDowell (1997) indicate that while assessment in the past was used as a means to determine measures, today it has provided potential benefits in all stages of the learning process. According to the authors assessment can be in a form of portfolio, self or peer assessment. Dochy, Segers, and Sluijsmans (1999) suggest that assessment should take the form of self , peer or co-assessment. Peer assessment, is when groups of individuals rate their peers which may involve rating instruments or checklists designed by others before the peer assessment exercise (Falchikov, 1995). Dochy, Segers and Sluijsmans (1999:346) identify a couple of guidelines that can be used for peer assessment. These are (a) peer assessment criteria should be presented in operational terms with which students are familiar, and (b) peer assessment can be used as a tool for summative assessment, in combination with other assessment instruments. Falchikov asserts that the overwhelming view of peer assessment is a useful, reliable and valid exercise perceived by students to be beneficial.

Freeman (1995) while cautioning in the introduction of innovative forms of summative assessment which involves elements of subjectivity suggests that by using past videos one can focus on a best and worst presentation to teach students how to mark more reliably. Although there may be some doubt about peer assessment as a summative form of assessment Orsmond, Merry and Reiling (1996) argue that as a formative assessment the process has some value and clear benefits to the student learning process may be gained from peer assessment. Searby and Ewers (1997) reiterate that peer assessment challenges the belief that the lecturer is necessarily the best person to provide feedback and that the introduction of peer assessment has had the beneficial effect of making students consider the whole learning process and their part in it.

The study

This study was conducted in October 1999 in tutorials for a common Core unit being offered in the Business School. Students in their tutorials were handed out a one page questionnaire to completed and returned to their tutors at the end of the tutorial sessions. The questionnaire consisted of 14 assessment items as shown in Table 1 as well as demographic characteristics including age, home country, gender, year of study, and course of study.

Table 1: Assessment items

Clarity
1Confidence, interest and enthusiasm of presenters.
2Presenters speak clearly and can easily be heard.
3Presentation is well rehearsed, not read from notes.
4Presenters maintain eye contact with audience.
5Presenters have positive body language.
Content
6Presentation includes a concise introduction and summary of topic
7Presenters provide comprehensive answers to questions and support these with relevant theoretical concepts
8Presentation includes a clear conclusion.
9Presenters able to answer questions raised during and after presentation
10Degree of creativity shown in presentation.
11Use of resources other than textbook.
Format and structure
12OHP and other visual aids are clear and neat (eg. large font, WP, etc).
13Time management of presenters.
14Overall organisation, consistency, flow and effectiveness of presentation.

Results and discussion

A total of 318 completed questionnaires were received and analysed using the SPSS package (version 8) for Windows. Of the 318 respondents 56% were female, 53% were less than 20 years of age (mean = 21.02 years). 59% were Australian students, 95% were commerce students, and 79% were in their first year of study. Respondent profiles are shown in Table 2.

Table 2: Respondents' profiles

Demographic variable n%
Gender
Male
Female
Total

137
175
312

44
56
100
Age group
17-19 years old
20-24 years old
25-29 years old
30+ years old
Total

166
108
18
20
312

53
35
6
6
100
Home country
Australia
Asia
Other
Total

176
103
21
300

59
34
7
100
Course of study
Commerce
Other
Total

296
15
311

95
5
100
Year of study
First year
Second year
Third year
Total

245
46
18
309

79
15
6
100

All items with the exception of item 11 ("use of resources other than textbook", mean 4.83) were rated above 5 ranging from 5.04 to 5.94. The lower mean score of this item implies that students may not be keen to go out and do some additional research prior to the class presentation on their particular topic other than using the textbook only. Item 2, "presenters speak clearly and can easily be heard" scored the highest mean (5.94) suggesting that the level and tone of voice is a very important part of class presentation. Table 3 shows the means and standard deviations of each item.

Table 3: Mean scores of assessment items


MeanStd
dev
1Presenters speak clearly and can easily be heard. 5.941.02
2Overall organisation, consistency, flow and effectiveness of presentation. 5.741.07
3Presenters provide comprehensive answers to questions and support these with relevant theoretical concepts. 5.651.15
4Confidence, interest and enthusiasm of presenters. 5.581.16
5OHP and other visual aids are clear and neat (eg. large font, WP, etc). 5.531.11
6Presenters able to answer questions raised during and after presentation. 5.431.12
7Presentation includes a concise introduction and summary of topic. 5.381.15
8Presenters maintain eye contact with audience. 5.241.23
9Presentation includes a clear conclusion. 5.241.64
10Degree of creativity shown in presentation. 5.161.22
11Presentation is well rehearsed, not read from notes. 5.141.11
12Presenters have positive body language. 5.131.17
13Time management of presenters. 5.041.29
14Use of resources other than textbook. 4.831.34

Kaiser-Myer-Olkin (KMO) statistic with factor loadings of .3 and Cronbach's alpha were referenced to determine the factorability of the items. An initial analysis yielded four factors (image dimensions) with a measure of sampling adequacy (MSA) of .83 with Cronbach's alpha .78. This is deemed more than adequate for factor analysis (Hair, et. al. 1998; Coakes and Steed, 1999). However, when Item 1 " confidence, interest and enthusiasm of presenters" with the least communality (.166) was dropped from the set a second analysis resulted in an MSA .84 and Cronbach's alpha .85 with three factors only. As the item is one of those items with high mean scores it was not necessary to exclude it from further analysis.

Factor analysis using Principal Axis Factoring and Varimax rotation resulted in four headings (factors). The first factor, concepts, accounted for 33.5% of the variance e. Factor 2, resources, accounted for 11.8% of the variances. Factor 3, named non-verbal communications, accounted for 8.3% of variances and Factor 4, clarity, accounted for 7.7% of variances (Table 4).

Table 4: Factors (image dimensions)

Assessment items Factor
1
Factor
2
Factor
3
Factor
4
1Presenters provide comprehensive answers to questions and support these with relevant theoretical concepts. .690


2Presentation includes a clear conclusion. .635


3Presentation includes a concise introduction and summary of topic. .606


4Presenters are able to answer questions raised during and after presentation. .528


5Use of resources other than textbook.
.616

6OHP and other visual aids are clear and neat (eg. large font, WP, etc).
.608

7Time management of presenters.
.602

8Degree of creativity shown in presentation.
.404

9Presenters maintain eye contact with audience.

.822
10Presenters have positive body language.

.691
11Presentation is well rehearsed, not read from notes.

.461
12Presenters speak clearly and can easily be heard.


.508
13Overall organisation, consistency, flow and effectiveness of presentation.


.487
14Confidence, interest and enthusiasm of presenters


.376
Eigenvalue 4.691.651.161.07
Variance explained (percentage) 33.511.88.37.7
Cumulative variance explained (percentage) 33.545.353.661.3
Cronbach's alpha .75.71.77.51

Factors and demographic variables

ANOVA tests were run to determine if there were statistically significant differences between groups in each of the four factors. Significant differences were shown in resources and non-verbal communications. First, while students in the 30+ year age group would more likely agree that resources other than textbooks are important those in the 17-19 year age group do not see the importance of these resources for the presentation. Secondly, Students from other countries would more likely agree with the importance of non-verbal communications compared with the Australian students (Table 5).

Table 5: Overall scores on factors by demographic variables
(means and standard deviation)

Demographic
variables
Factor 1
(Concepts)
Factor 2
(Resources)
Factor 3
(Non-verbal comm)
Factor 4
(Clarity)
Gender
Male
Female
F-value

5.41 (.91)
5.45 (.85)
.21

5.08 (.91)
5.20 (.92)
1.26

5.06 (1.03)
5.25 (.93)
2.96

5.78 (1.73)
5.82 (.83)
.09
Age group
17-19 years old
20-24 years old
25-29 years old
30+ years old
F-value

5.43 (.89)
5.40 (.84)
5.20 (1.03)
5.74 (.79)
1.28

5.06 (.90)
5.14 (.95)
5.47 (.80)
5.59 (.73)
2.91*

5.05 (.94)
5.25 (.98)
5.22 (1.10)
5.52 (1.11)
1.96

5.69 (.92)
5.96 (1.83)
5.89 (.78)
5.78 (.92)
.98
Home country
Australia
Asia
Other
F-value

5.41 (.82)
5.42 (.97)
5.54 (.85)
.20

5.07 (.86)
5.25 (.92)
4.95 (1.12)
1.68

5.02 (.96)
5.30 (.95)
5.38 (.97)
3.51*

5.73 (.86)
5.87 (1.92)
5.86 (.76)
.39
Course of study
Commerce
Other
F-value
5.42 (.88)
5.64 (.65)
.96
5.14 (.91)
5.32 (.64)
.53
5.17 (.99)
5.04 (.73)
.23
5.79 (1.33)
6.02 (.64)
.45
Year of study
First year
Second year
Third year
F-value

5.44 (.87)
5.33 (.96)
5.49 (.78)
.33

5.13 (.91)
5.19 (1.00)
5.36 (.64)
.61

5.14 (.97)
5.34 (1.06)
4.91 (.88)
1.45

5.84 (1.40)
5.59 (.88)
5.81 (.60)
.70
Standard deviations are in parentheses. Factor scores were coded on a 7 point Likert scale ranging from 1 "least important" to 7 "most important".
* Significant at .05 level.

While Table 5 shows statistically significant differences between groups on the four factors, six of the 14 items listed in Table 1 also have shown significant differences in such demographics as gender, age group and home country (Table 6).

Table 6: Overall scores on individual items by demographic variables
(means and standard deviation)

Demographic
variables
Item
4
Item
5
Item
10
Item
11
Item
12
Item
13
Item
14
Gender
Male

Female

F

5.03
(1.33)
5.40
(1.13)
6.97**

5.07
(1.27)
5.18
(1.11)
.74

5.20
(1.17)
5.13
(1.27)
.27

4.77
(1.39)
4.91
(1.29)
.78

5.51
(1.19)
5.57
(1.04)
.20

4.89
(1.32)
5.18
(1.26)
3.98*

5.60
(1.18)
5.87
(.98)
5.00*
Age group
17-19 years old

20-24 years old

25-29 years old

30+ years old

F

5.10
(1.18)
5.39
(1.23)
5.28
(1.41)
5.40
(1.50)
1.39

4.96
(1.18)
5.24
(1.13)
5.50
(1.29)
5.50
(1.23)
2.70*

5.07
(1.25)
5.10
(1.18)
5.50
(1.38)
5.95
(.83)
3.65*

4.71
(1.28)
4.90
(1.37)
4.89
(1.74)
5.40
(.99)
1.76

5.50
(1.07)
5.44
(1.21)
6.11
(.83)
5.95
(.94)
2.91*

4.93
(1.34)
5.19
(1.24)
5.39
(.92)
5.05
(1.39)
1.31

5.73
(1.11)
5.82
(1.03)
5.61
(1.20)
5.65
(.99)
.34
Home country
Australia

Asia

Other

F

5.10
(1.21)
5.34
(1.29)
5.48
(1.25)
1.70

4.90
(1.20)
5.38
(1.38)
5.60
(.82)
7.49***

5.17
(1.23)
5.07
(1.26)
5.24
(1.14)
.29

4.68
(1.22)
5.14
(1.30)
4.48
(1.94)
4.63**

5.54
(1.07)
5.55
(1.13)
5.43
(1.43)
.11

4.94
(1.36)
5.23
(1.07)
4.67
(1.59)
2.57

5.77
(1.04)
5.67
(1.14)
5.81
(1.12)
.31
Course of study
Commerce

Other

F

5.24
(1.24)
5.20
(1.01)
.01

5.14
(1.18)
4.80
(1.21)
1.21

5.16
(1.24)
5.40
(.74)
.56

4.85
(1.34)
4.87
(.99)
.00

5.53
(1.12)
5.80
(.86)
.81

5.05
(1.28)
5.20
(1.65)
.19

5.73
(1.08)
6.07
(.80)
1.37
Year of study
First year

Second year

Third year

F

5.23
(1.22)
5.46
(1.20)
4.78
(1.44)
1.97

5.08
(1.19)
5.37
(1.25)
5.05
(.80)
1.18

5.14
(1.19)
5.07
(1.48)
5.67
(.77)
1.70

4.84
(1.30)
4.80
(1.56)
4.94
(1.16)
.07

5.53
(1.12)
5.63
(1.10)
5.61
(1.09)
.18

5.01
(1.31)
5.26
(1.24)
5.22
(.88)
.91

5.77
(1.10)
5.61
(.88)
6.00
(.77)
.94
Standard deviations are in parentheses. Item scores were coded on a 7 point Likert scale ranging from 1 "least important" to 7 "most important".
* Significant at .05 level, **Significant at .01 level, ***Significant at .001 level.

According to Table 6 it is shown that while female respondents compared with their male respondents tend to rate Items 4 ("presenters maintain eye contact with audience"), 13 ("time management of presenters"), and 14 ("overall organisation, consistency, flow and effectiveness of presentation") highly. On the other hand, the 30+ year group tend to rate Items 5 ("positive body language") and 10 ("degree of creativity shown in presentation") more than the 17-19 year olds. Item 12 ("OHP and other visual aids are clear and neat...") was rated higher by the 25-29 year compared with the 20-24 year group. In the case of home country, students from countries other than Australia and Asia rated Item 5 higher compared with Australian students while Asian students compared with students from other parts of the world rated Item 11 ("use of resources other than textbook") more.

Implications and conclusion

The literature supports the use of various assessment mechanisms including peer assessment of student presentations. The results of this study also indicate that students find the assessment criteria in use important. All the 14 items were scored favourably, the least being "use of resources other than textbook". The challenge is teach students the value of research into more sources rather than sticking to the recommended text.

While the result shows students' agreement with the assessment criteria, caution should be exercised in the interpretation of results. In conclusion similar surveys over a period of time with larger number of students would be a logical extension.

References

Coakes, S. J. and Steed, L. G. (1999). SPSS: Analysis without Anguish. Versions 7.0-8.0 for Windows, Brisbane, John Wiley & Sons.

Dochy, F., Segers, M. and Sluijsmans, D. (1999). The use of self, peer and co-assessment in higher education: a review. Studies in Higher Education, 24(3), 331-350.

Dochy, F.R.C. and McDowell, L. (1997). Assessment as a tool for learning. Studies in Educational Evaluation, 23(4), 279-298.

Falchikov, N. (1995). Peer feedback marking - developing peer assessment. Innovations in Education & Training International, 32(2, May), 175-187.

Freeman, M. (1995). Peer assessment by groups of group work. Assessment & Evaluation in Higher Education, 20(3), 289-300.

Hair, J. F., Anderson, R. E., Tatham, R. L. and Black, W. C. (1998). Multivariate Data Analysis. Fifth edition, New Jersey, Prentice Hall.

Hughes, I.E. and Large, B. J. (1993). Staff and peer group assessment of oral communication skills. Studies in Higher Education, 18(3), 379-385.

Orsmond, P., Merry, S. and Reiling, K. (1996). The importance of marking criteria in the use of peer assessment. Assessment & Evaluation in Higher Education, 21(3), 239-250.

Peters, M. (1996). Student attitudes to alternative forms of assessment and to openness. Open Learning, November, 48-50.

Pond, K. and ul-Haq, R. (1997). Learning to assess students using peer review. Studies in Educational Evaluation, 23(4), 331-348.

Searby, M. and Ewers, T. (1997). An evaluation of the use of peer assessment in higher education: A case study in the School of Music, Kingston University. Assessment & Evaluation in Higher Education, 22(4), 371-383.

Please cite as: Shanka, T. (2000). Assessing the assessment criteria: Students' opinions. In A. Herrmann and M.M. Kulski (Eds), Flexible Futures in Tertiary Teaching. Proceedings of the 9th Annual Teaching Learning Forum, 2-4 February 2000. Perth: Curtin University of Technology. http://lsn.curtin.edu.au/tlf/tlf2000/shanka.html


[ TL Forum 2000 Proceedings Contents ] [ TL Forums Index ]
HTML: Roger Atkinson, Teaching and Learning Centre, Murdoch University [rjatkinson@bigpond.com]
This URL: http://lsn.curtin.edu.au/tlf/tlf2000/shanka.html
Last revision: 20 Feb 2002. Curtin University of Technology
Previous URL 27 Dec 1999 to 20 Feb 2002 http://cleo.murdoch.edu.au/confs/tlf/tlf2000/shanka.html