Teaching and Learning Forum 95 [ Contents ]

A comparison of several survey techniques to obtain student evaluation of a unit in Land Management

Colin D. Walker and Richard W. Bell
Biological and Environmental Sciences
Murdoch University
Student feedback techniques appear ad hoc, to be serving purposes related to particular interests of the course coordinator. While this may be the prerogative of the course coordinator, we were aware of a diverse range of approaches used in feedback questionnaires. We have considered whether there might be more effective approaches than single item responses on individual questions that tend to dominate the more usual feedback questionnaires. This paper summarises results from two questionnaires, examining issues of custom design of questionnaires and interpreting correlations among groups of item responses that arose in undertaking several different feedback techniques on a student group. Feedback from a context-designed conjoint survey was considered the most valuable for course implementation.

Introduction

Most attempts to improve the nature of a tertiary course through student evaluation and feedback simply identify what is believed to be an appropriate questionnaire and then run this on the class of students (or a representative sub-group). This approach assumes that the staff member(s) associated with the course have a fairly good understanding of the nature of problems in their course, and how the student group will respond to such issues in their survey. It further assumes that those administering the survey know enough about the appropriate framing of questions, the reliability of their survey techniques and the appropriate ways to interpret the responses to obtain a relevant and actionable set of recommendations.

A program of comparisons between different surveys in the one unit would provide greater understanding of both the opinions of the respondents and the effectiveness of the feedback gathering instruments. Much as the old truism says: "To get the right answer, first ask the right question!" Literature resources on survey techniques are quite abundant. However, close comparison between different survey techniques are rare. It appears quite likely that the perception of course-work and content implied in a questionnaire drive the development of the respondents' opinions. In other words, our specific questions may be leading our witnesses.

As part of a program of renovation of laboratory education within the school of BES, an initiative was planned involving the identification of effective and valid survey instruments. This was carried out through a program of several questionnaires. The outcomes of this exercise were framed in terms of trying to identify design features of effective surveys and selection of an optimum approach to improvement through these. However, to make some simple recommendation identifying a single best approach is beyond reasonable expectations: not only does the intended use of survey instruments vary, but the interpretive needs of those who use them also varies. The most clearly distinguished cases of the latter are those situations where a survey is required for needs relating to staff promotion as distinct from student learning. While some may argue that these should be the same, there are many cases where two separate evaluation purposes are accepted within a profession. In our current context, this paper will highlight features of interest in terms of validly tapping the students' needs.

Questionnaires

Murdoch Questionnaire
The Murdoch questionnaire was run on two occasions, in slightly different forms. In the context of unit reviews, a full questionnaire to students had been devised by the Murdoch Academic Services Unit (ASU) personnel to explore many aspects of a unit under topics of organisation, content, assessment, resources, staff, lectures, laboratories, and overall issues. The questionnaire allows responses over a scale of four responses, from strongly disagree to strongly agree, as well as being unable to judge. Cases where 30% or more of respondents disagree or strongly disagree are considered a problem area in the delivery of the unit, where course coordinators should make improvements.

The full questionnaire, some 58 separate items, was given to the entire land management class during a normal lecture period. A second group of 24 questions that specifically related to laboratory teaching was undertaken by a group of tutorial students on a separate occasion. Some 11 questions were common to both surveys. The intention of this exercise was firstly to examine the repeatability of the questionnaire in terms of the responses on the 11 common questions, secondly to examine (or demonstrate) the continual problem that a structured questionnaire might overlook further issue(s), and thirdly, to examine the questionnaire in the fundamental role of diagnosis of unit defects. The problem in scope mentioned as the second interest is a hindrance moving from the interpretation of the results of a questionnaire study to the decision to invest resources into dealing with the issues identified.

The repeatability of the questionaire was examined by comparing the distribution of responses on the 11 replicated questions using the Kolmogoronov-Smirnov two-sample tests, and comparing the percentage distributions (Table 1). The results found that cumulative distributions generally did not differ between the two groups, but that the interpretative benchmark of 30% respondents disagreeing did vary frequently.

Table 1: Kolmogoronov-Smirnov (K-S) two-sample test for significantly differing distributions
in the cumulated responses to 11 questions common to the two Murdoch studies.

The general absence of low probabilities (no P<0.05) indicates acceptance of a null hypothesis that the distribution of responses were basically the same on both occasions. The percentage distribution does not show 'unable to judge' response levels. The 30% rule refers to the ASU recommendation that educators make improvements where more than 30% of respondents give either Disagree or Strongly Disagree as their responses. Variations in this between the two studies are marked **

No. Respondents

% Distribution -
ASU survey

% Distribution -
Tutorial

No. of QuestionASU
survey
TutorialK - S
Z
2-tailed
P
SAADSDSAADSDVariation in
30% Rule

47. Explanations
48. Encourage
49. Interested
50. Organised
51. Responses
52. Comments
53. Reasonable time
54. Ratio
55. Stimulating
56. Relevant
57. Writeup OK
63
64
63
63
59
57
60
63
64
60
63
18
19
17
19
18
19
18
18
18
17
17
1.336
0.554
0.536
0.243
0.643
0.397
1.137
0.802
1.249
0.371
0.939
0.056
0.919
0.936
1.000
0.802
0.997
0.151
0.541
0.088
0.999
0.371
3
9
11
6
8
1
24
1
9
9
26
42
64
65
62
58
38
44
77
39
70
23
38
20
17
23
24
38
14
15
39
11
18
12
4
3
4
0
9
9
1
9
1
29
10
10
16
0
5
10
0
6
11
6
29
68
79
42
68
47
42
42
56
72
71
47
16
10
26
21
42
26
31
33
6
18
6
0
0
5
10
0
21
21
6
11
6
18
**

**
**
**

**
**
**

**

Spearman rank correlation analysis identified many responses in the Murdoch questionnaire that were clearly linked issues in the respondents perceptions (data not shown). These indicated some clear behavioural traits and expectations of the students, and provided considerable guidance to unit improvement. An outline of some of these follows.

If the students were in agreement that the aims of the unit were clearly stated at the beginning (Qn 1), they also tended to consider that these aims were achieved (Qn 2). In the context of this unit, this may simply identify those students who have effectively read the introduction of the Study Guide.

Quality is a diffuse, but functional, term. When the students tended to feel that the unit was well organised (Qn 3), that the unit's teaching sessions were appropriate to the unit (Qn 4), or that the unit did not attempt to cover too much material (Qn 7), they also reported being satisfied with the quality of the unit (Qn 25). In fact, responses on Qn 25 on the quality of the unit was positively and significantly correlated with 22 of the 58 questions used overall, including all of the first 11 questions, and the two questions examining the quality of the lectures (Qn 46) and the quality of the laboratories (Qn 58). In a similar fashion, Qn 46 was correlated with 33 other questions, including 16 of the 21 examining aspects of lecturing. Question 58 was correlated with 16 questions, including 9 of the last 12 which were on the laboratory sessions. The interpretations made on the basis of questions using the term quality may not be very specific in guiding improvements.

Several cases of negative correlations between responses on different questions were observed and deserve note. The existence of these cases suggests that the optimisation process may never be perfect, but can only satisfy a majority of individuals needs. In the ASU survey, there were seven cases of significant (P<0.05) negative correlations. The workload being reasonable (Qn 6) was negatively associated with the lecturer indicating enough suitable references (Qn 35). The lectures aiding understanding (Qn 8) was negatively associated with the lecturer encouraging student questions (Qn 33). The staff being accessible outside of classes (Qn 22) was negatively associated with lab classes having a satisfactory staff:student ratio (Qn 54). The lecturer being well organised (Qn 26) was negatively associated with enough examples being given during lectures (Qn 44). The study guide setting out the unit requirements clearly was negatively associated with the demonstrator encouraging the student to think (Qn 48). The response that the lecturer writes clearly (Qn 29) was negatively associated with the demonstrator encouraging the student to think (Qn 48). The response that the lecturer presents difficult concepts clearly (Qn 31) was negatively correlated with the being satisfied with the quality of the laboratory sessions(Qn 58).

Further negative correlations were observed when the expanded laboratory questionnaire was used. There were five statistically significant (P<0.05) examples of these, as follows: A response that the demonstrator was interested in students (Qn 4) was negatively correlated with both responses on there being a satisfactory staff:student ratio (Qn 9) and there being clear links with theory (Qn 22). A response that there was adequate equipment (Qn11) was negatively correlated with a response that the aims of the laboratories were clearly stated (Qn 21). The latter response was also negatively correlated with responses on the presence of enough space in the laboratories (Qn 15). The responses on space in the laboratories (Qn 15) were negatively correlated with responses on there being an opportunity to discuss the practical work in class time (Qn 23). The last three of these correlations suggest that the learning in laboratories is being influenced by queueing. We suggest that where students were obliged to queue for equipment, they will tend to read the study guide further and discuss the laboratory exercises with colleagues and demonstrators.

Conjoint Analysis
Conjoint analysis seeks to obtain an insight into the relative importance of different aspects of a service or product that might be modified in an effort to improve customer acceptance or 'utility'. It does this by asking the respondent to choose between different combinations of the differing aspects or 'attributes' (Green and Wind, 1975). While this technique has only been used occasionally on education services (Walker, Winzar and Johnson, 1991; Carter, 1993), it has been widely used in the commercial services field (Greene and Nkong, 1989).

In the current study, as in Carter (1993), the relevance (or Salience in the terminology of Carter) of the analysis to the individual students was enhanced by focussing a discussion with the tutorial group on those aspects of a unit that they most liked. In the context of the land management unit, this basically identified five main areas of concern (Content Workload, Flexibility, Student Organisation, Assessment, and Feedback). A set of different levels of unit attributes in these areas was devised (Table 2). A set of hypothetical units with a wide range of combinations of each of the attributes was then devised from these (not shown). At a subsequent tutorial, the students were presented with a set of the hypothetical units, with each unit outlined just in terms of the five attributes, and each printed on an individual slip of paper. The students were each asked to consider these as possible formats of the land management unit, to assemble the slips in their order of preference, and to rate each of them.

The results of this study are summarised in Table 3, wherein the term 'Importance' indicates the overall effect of the utility of an attribute on the model, relative to that observed over all attributes. The overall Utility (U) of a hypothetical unit may be estimated using a model as follows:

Mathematical equation for the utility of a hypothetical unit
over all five attributes, with a Pearson R value of 0.974. Some bias in the estimates of B may be expected from the assumption of linearity of utility over the three levels of attributes.

The results suggest that flexibility in the unit delivery was more important than its workload (Table 3). This is an unexpected outcome in that workload was often the major focus of discussions on the unit. The options that were responsible for the improvement in utility due to course flexibility appear to be the prospect of lecture repeats and videotaping, and frequent laboratory timeslots. This appears to be more suitable in modern student lifestyles.

Table 2: Unit Attributes and three Factor levels of these used for Conjoint Study.

Content Workload
  1. Substantial theory workload requiring intensive reading and considerable understanding of concepts from prerequisite courses, many new concepts in each lecture.
  2. Moderate theory workload with around three hours of reading from two prescribed texts each week, with several new concepts involved each week.
  3. Light theory workload focussing on simple, popular and broad concepts and exploring these in some depth, but no reading necessary outside of a 120 page text.
Flexibility
  1. Lectures, labs and tutorials at one time only, never repeated and no library tapes of lectures. Sometimes a penalty for missing scheduled activities.
  2. Lecture at one time only, but taped and always available in closed reserve. Lab classes scheduled only on two teaching weekday afternoons.
  3. A repeat of all lectures in a convenient evening time slot, with video tapes of lectures and copies of overheads available. Labs and tutorials both in mornings and afternoons, with five scheduled times.
Student Organisation
  1. Laboratory projects are due within three days of the lab but require around 5 hours of group work (your groups are of 4 members) to prepare a group report.
  2. Laboratory projects prepared in varying ways, some by groups and some individually, always due within seven days of the lab.
  3. Laboratory projects all prepared individually, and are required within two weeks of the lab.
Assessment
  1. Assessment during semester requires extensive reasoning and deduction, in-depth knowledge is expected in substantial (>1500 words/four pages) assignments.
  2. Assessment during semester requires searches through reference materials and reader, but only single paragraph answers required.
  3. Assessment during semester based on sets of 12 short answers: not essays, workings or explanations required.
Feedback
  1. No results on any assignments during semester (worth 50% of total grade) until after the final exam.
  2. Some assignments (25% of final grade) during semester are returned, but around 25% of final grade in unreturned assignments at exam time.
  3. All assignments (50% of total grade) marked within 2 weeks of submission and returned by week 11, usually with over 20 words of comments.

Respondents attached similar levels of importance to feedback and workload. Feedback was also raised as an issue in other surveys. While feedback is not assessed as a time-related issue (as here) in the Murdoch questionnaire, there are clear indications that the nature of comments on the returned assignments is not sufficiently helpful. The form of assessment appears to be a non-issue, however the respondents did indicate some loss of utility if a unit demanded that they be involved in organised group work.

Table 3: Conjoint Analysis of the matrices of relative utility respondents attached to hypothetical units drawn up from permutations of Attributes given in Table 2. (SPSS Release 4.0 for Macintosh)


Importance*Utility (s.e.)Factor Level

WORKLOAD
22.161.7563 ( .4922)
3.5127 ( .9843)
5.2690 (1.4765)
1.00
2.00
3.00
B1 =1.7563 ( .4922)

FLEXIBILITY
33.632.6650 ( .5073)
5.3300 (1.0146)
7.9950 (1.5219)
1.00
2.00
3.00
B2 =2.6650 ( .5073)

STUDENT ORGANSTN
17.101.3548 ( .4721)
2.7096 ( .9443)
4.0644 (1.4164)
1.00
2.00
3.00
B3 =1.3548 ( .4721)

ASSESSMENT
5.21-.4127 ( .4552)
-.8255 ( .9104)
-1.238 (1.3656)
1.00
2.00
3.00
B4 =-.4127 ( .4552)

FEEDBACK
21.901.7358 ( .5004)
3.4717 (1.0007)
5.2075 (1.5011)
1.00
2.00
3.00
B5 =1.7358 ( .5004)


CONSTANT
-5.6906(2.4553)

Pearson's R = .974
Kendall's tau = .833
Significance = .0000
Significance = .0009
* Note that Importance sums to 100.0 over all attributes. It indicates a fraction of the dependent variation in the model that the attribute is contributing.

Discussion

The Murdoch Questionnaire was the most explorative and wide-ranging survey instrument used in this study. Within the responses, significant correlations (P<.05) were found in around 30% of question-question comparisons. Over the 58 separate items, the questionnaire was therefore repetitively sampling common areas of respondents' perceptions quite frequently. This suggests that it may be possible to develop constructed scores from the response data, where the responses on several closely correlating questions are summed to obtain an overall response that is more accurately reporting the respondent's opinion of an area of interest. Such constructed scores can take on a wider range of values and are therefore more functional in futher statistical analyses. This is the intention of a SERVQUAL type of questionnaire, which was also undertaken, but has not been discussed here because of space limitations.

The conjoint analysis feedback is strongly 'customised' through a sequence of two interaction sessions with the study group. The initial session obtained their opinions as to what would improve the unit for them as individuals. This exercise generated a set of attributes for which a series of steps or service levels were devised by the investigator. These steps were assembled into different unit formats, trading off a more attractive level in one attribute with a worse level in another. In the second session, the respondents examined a set of hypothetical units and decided which would be most preferable to them. The hypothetical units were to be in Land Management, and it would not be appropriate to interpretate the data as applying to any University unit.

To illustrate this process, the most attractive of all the units offered was one set taking in levels 2 of Workload, 3 of Flexibility, 2 of Student Organisation, 1 of Assessment, and 3 of Feedback (Table 2). It was the top choice of over 50% of respondents, and to summarise the levels from Table 2, it had the following overall set of attributes:

Unit R
  • Moderate theory workload with around three hours of reading from two prescribed texts each week, with several new concepts involved each week.
  • A repeat of all lectures in a convenient evening time slot, with video tapes of lectures and copies of overheads available. Labs and tutorials available both in mornings and afternoons, with five scheduled times.
  • Laboratory projects prepared in varying ways, some by groups and some individually, always due within seven days of the lab.
  • Assessment during semester requires extensive reasoning and deduction, in-depth knowledge is expected in substantial (> 1500 words/four pages) assignments.
  • All assignments (50% of total grade) marked within 2 weeks of submission and returned by week 11, usually with over 20 words of comments.

Inspection of these attributes recognises a service level that is a step above that currently enjoyed by Land Management students, and it must be admitted that options such as morning laboratories, evening lectures, and all assignments returned by week 11 might be beyond available human resources. In our own response to this, we have increased flexibility in another (less expensive) way, in allowing internal students to undertake the laboratories in the laboratory week that external students undertake on-campus. Video-taped material related to the unit has been expanded.

One major strength of conjoint analysis lies in the importance it attachs to different attributes, where importance directly reflects the relative utility that each contributes in the students preferences. We know that a dollar invested in the flexibility area is worth more than one invested in other areas, and how much more in proportion. A second strength lies in the prescriptive nature of the set of attribute steps generated by the initial session's feedback. This allows the investigator to be more certain about the nature of changes mandated by the second session's feedback.

The major defect of conjoint analysis is that it may not allow access to a widely applied mechanistic model of the education service. Such a model would be an ideal tool if it was universally applicable to all units, and responses to questions in a standard questionnaire could be accumulated to give scores for appropriate components of the model. This would also serve needs in staff development and promotion, and we suggest that diverse feedback methods will remain for such varied agendas.

References

Carter, M. G. (1993). SMART - Customer satisfaction: From theory to practice. In Taking Focus on Consumers. Proc. Conf. MRSA, Perth. pp.1-16.

Greene, C. S. and Nkonge, J. (1989). Gaining a competitive edge through Conjoint Analysis. Business, 39(Apr-Jun), 14-18.

Siegel, S. and Castellan, N. J. Jr. (1988). Nonparametric statistics for the behavioural sciences. 2 ed. McGraw Hill, NY.

Walker, M., Winzar, H. and Johnson, L. (1991). Modelling perceptions of business degrees. Working Paper Ser. No. 20, Key Centre Strategic Mgt., Queensland University of Technology, Brisbane.

Please cite as: Walker, C. D. and Bell, R. W. (1995). A comparison of several survey techniques to obtain student evaluation of a unit in Land Management. In Summers, L. (Ed), A Focus on Learning, p264-270. Proceedings of the 4th Annual Teaching Learning Forum, Edith Cowan University, February 1995. Perth: Edith Cowan University. http://lsn.curtin.edu.au/tlf/tlf1995/walker.html


[ TL Forum 1995 Proceedings Contents ] [ TL Forums Index ]
HTML: Roger Atkinson, Teaching and Learning Centre, Murdoch University [rjatkinson@bigpond.com]
This URL: http://lsn.curtin.edu.au/tlf/tlf1995/walker.html
Last revision: 19 Apr 2002. Edith Cowan University
Previous URL 15 Mar 1997 to 19 Apr 2002 http://cleo.murdoch.edu.au/asu/pubs/tlf/tlf95/walk264.html