Teaching and Learning Forum 2004 Home Page
Teaching and Learning Forum 2004 [ Proceedings Contents ]
Systematic approach to triple feedback systems for teaching enhancement

R. Morien, E. Chang, K.F. Chin and C. Cheah
Curtin University of Technology

This paper identifies issues that arise from traditional university feedback systems. Traditional university feedback systems are undertaken as annual student surveys in areas including curriculum and teaching which may be conducted by the academic development unit, student union or at faculty or school level which generate statistical results. All universities around the world have such feedback systems. Some universities take the results seriously at senior management level, some only at academic teaching staff level and some only at a student level. A common problem is that these teaching survey results may only be seen by teachers, it doesn't matter whether the results are good or bad. The questions arise: How much value can be gained from seeking feedback annually rather than routinely? How to measure the significance by those who utilise the survey results to enhance teaching? How can these anonymous results be utilized to enhance service not just at subject level or from the teachers' point of view but for the entire curriculum? How much effort has to be put in by teachers to use it for teaching improvement? How much effort has to be put in by senior management to use it for monitoring, supervising and controlling the standard of teaching performance, curriculum standards and incremental educational improvement?

We note that there is another kind of feedback which has not been addressed systematically nor dealt with efficiently, that is the feedback from academics who deliver the teaching materials. This kind of feedback has been dealt with in an ad-hoc fashion among many universities. Some universities use a sub-group of staff or elected teaching staff, others rely on senior management, or carry out periodic meetings that may be unproductive because matters are not followed up or disagreements during the meetings result in wasted time and in little change. This is caused by ad-hoc management of the specialist academic expert. Normally, an academic's teaching area is the same as or is related to their research area, or at least their area of competence, experience and expertise. Naturally, the academics in a particular teaching area are recognised as being more advanced in that area than others. Academics understand that teaching is informed by research. Therefore, any feedback from specialist academics should not be dealt with in an ad-hoc fashion and their feedback on the subject or curriculum is as vital as student feedback.

We also note that many universities utilise industry advisory panels. In many universities, they only meet once a year. However, we found only ad-hoc management of such feedback exists and no measurement has taken place on how the industry panel and their input has been applied. What are the effective ways of utilising the connection between course development and education improvement? Yet at the same time curriculum must be proposed that does more than merely meet current, possibly transient, industry skills demands (Morien & Schmidenberg, 1994). In this paper, we illustrate a dynamic curriculum development architecture, which systematically collects triple input or feedback from learners (students), teachers (academics) and industry panelists. We provide an incremental management approach to use these as a basis for new course development and strategic management of the improvement process of course development, as well as a matrix on the measurement of how one utilises the triple feedback for teaching and learning improvement and the value output from the triple feedback system.


Introduction

Teaching and learning is one of the core activities of a university. All universities set a goal to be awarded World Class University status in teaching and learning and in doing so they set up a university teaching and learning community or forum and publish major university objectives and priorities that enable the university and academics to develop its teaching and learning programs to achieve this vision (Curtin University of Technology, 2002). Some universities develop teaching and learning plans based on a "learner centered approach"; that is they use learning outcomes as a measure to improve teaching and learning programs and teaching and learning quality. Some universities focus on both teaching quality and learning quality. They identify the quality of teaching by the quality of course content, quality of course delivery, quality of teaching standard, quality of systematic evaluation and feedback and they also focus on the learning outcomes by measurement of focus on knowledge, skills, professionalism and strategy to make the learner a strategic and creative thinker. In regard to one of these two approaches, a university would have set up a framework or some sort of systematic mechanism or system that allows the measurement of the teaching and learning effectiveness or bench marking systems that allow the assessment of the quality of teaching and learning. There are three major activities involved in teaching and learning management (Glatthorn, Harris, & Carr, 2003; March & Willis 2003; DEST, 2003).
  1. Measurement of teaching quality
  2. Measurement of learning outcomes
  3. Measurement of programs and curricula
Through these three core activities, the university tries to achieve its quality management. However one of the major measurements of quality is through evaluation and the feedback systems. All universities have their systematic approach to evaluation of their programs, and the achievement of teaching and learning and feedbacks. Some of the these systems are more effective than the others and some of the feedback systems only focus at lower levels, such as a particular subject or unit level evaluation, but not at the curriculum or degree program level. Systematic feedback at both high level (curriculum level) and lower level (subject level) is crucial to the university planning, monitoring and quality improvement process.

Teaching and learning objectives and measurement frameworks

As mentioned in the introduction there are three major focuses in the measurement of teaching and learning quality: teaching quality, learning outcomes and the quality of curricula and programs. In order to achieve world class education, the university set up systematic systems that allow continuing improvement and development of these three educational dimensions.

Teaching quality

Normally, the performance of quality teaching is measured by the criteria listed below.

If there is a feedback from academic staff themselves, it only focuses on the following.

However, as we can see from the above objectives and measurement, there is no mention about staff feedback on curricula (education programs). Curricula (education programs) are the framework of the entire course teaching program. However, we note that the feedback from academics in regard to curriculum is either weak or not done at all.

Learning outcomes

For the measurement of quality learning outcomes, a number of activities and measures have been developed within all universities and these include

There are well developed questionnaires for the subject level and teachers' performance and student satisfaction levels for each particular subject or units.

The drawback of the above feedback system is that it cannot obtain feedback about the entire educational programs and they cannot obtain information such as student feedback about the existing survey systems; as well feedback efforts stop at the subject or unit level.

There is no evaluation or feedback data collection regarding the course curriculum overall and there is no approach to collecting student feedback about how their perceptions and feelings in regard to the entire curriculum program, such as overlapping subject curricula, prerequisite appropriateness, and outdated and even wrong and factually incorrect materials defined in the curriculum.

Quality of curricula

In regard to the quality of curricula, the measurement of the quality programs or the quality of the curriculum, student subject level feedback is normally used to justify the quality of the program which is inadequate, because the student feedback is only at subject level and not at entire course curriculum level.

The only measure the universities have been using is to obtain the information about

The decision about the curriculum quality based on the above criteria and one of two outcomes may occur.
Option 1: Shut down the entire course because student numbers are low.
Option 2: Modify the degree programs
(There are non-curriculum focused options as well, including marketing of the courses, offering advanced standing to incoming students, often based on inappropriate prior knowledge or experience).

In both of the above curriculum-oriented decisions, there is no assessment of the inner factors of the curriculum; whether the entire curriculum has met the student needs by simply measuring the external factors such as student numbers. There is no feedback collection from academics who are the implementers of the entire curriculum framework.

Teaching and learning development

Traditional approaches to teaching and learning development have three major components.

Component 1

Education provisioning that includes curriculum planning and curriculum design (Figure 1). Curriculum planning and curriculum design should match the universities' visions, missions and goals. It takes input from senior academics and industry partners to provide both internal and external knowledge to set up strategic directions of entire teaching programs. They form a blueprint for the subject development. They normally include curriculum planning and design across the university, faculty and schools. At each school level they form degree programs or curriculum that may include the total number of subjects across number of years.

Figure 1

Figure 1: Curriculum design process

Component 2

Educational program development that includes each subject's or unit's development. This is normally carried out by the academics or teachers. The subject development normally includes plans and content from lecture 1 to lecture N and teaching materials for each of the lectures, tutorials, lab, exercises, exams and assignment as well as assessments. The subject developers need to follow the objectives set out in the curriculum design and to make sure their development correspond to the overall curriculum goals.

Figure 2

Figure 2: Curriculum implementation

Component 3

Educational delivery that includes classroom practice, distant or e-learning practice. This shows in Figure 3. In this component teachers together working with the tutors, lab demonstrators, assessment markers to deliver a particular content to the student. The entire delivery follows the subject plans in a step by step manner throughout the semester and continually improves the quality of delivery annually to achieve better learning outcomes.

Figure 3

Figure 3: Teaching and learning process

Traditional approach to teaching and learning development

Figure 4

Figure 4: Traditional approach to teaching and learning development

The above diagram (Figure 4) shows the general approach to teaching and learning development among universities, illustrating the curriculum development process. This is a general approach in many universities. Senior academics usually design the entire curriculum, the teaching academics implement this design, and deliver the education, the students develop their knowledge and skills directly from the teaching academics, and there is a systematic feedback procedure on the academic delivery of the planned material. Industry partners usually give the comments at high level curriculum, and sometimes, their feedback and comments are taken into account in the curriculum planning and design.

Traditional approach to feedback is shown in Figure 4 where the feedback comes from industry representatives to the curriculum level and student feedback to subject level. Normally, those feedback cycles are well structured, systematic and periodical. The feedback is normally handled by the academic development unit or central teaching and learning committee. The data is collected, analysed and as a basis for the next improvement of the curriculum.

Triple feedback systems proposal

The traditional feedback systems have many weaknesses. The traditional feedback systems may be good at the subject or unit level but not at the curriculum level. Figure 5 shows the entire feedback cycle that should be undertaken in regard to the overall course curriculum. The feedback for the entire course curriculum and subject curricula should have three concurrent data collection mechanisms; from students, from academics and from industry. Already feedback cycles exist from students to the subject level as well as course curriculum level from industry partners, and is well planned in general.

However, we also need to collect feedback from academics who are the implementers of the curriculum, and their comments and input on the curriculum is important and crucial. We note that the following factors are very important to the curriculum development (see figure 5). The following comments and feedback would not be obtained by just subject level evaluation.

Feedback from academics

Issue 1  The teaching content is flexible to enable subject lecturers to make changes to its original content, and sometimes these changes then overlap with other content of other subjects, creating duplication of subject matter.
Issue 2Keep subject content in line with curriculum design is managed in ad hoc fashion.
Issue 3Teaching difficulties may cause inconsistent definition of pre-requisites requirement. For example, pre-requisites may be suitable to one degree program students, but not the others, however, they will all have to be part of an integrating unit at later year.
Issue 4Ad hoc management of optional subjects/units may cause miss link of knowledge acquisition and delivery. This often happens when students in a class are from different degree programs.
Issue 5Some staff find it difficult to deliver some subjects, because the students in a class have too many differences in their knowledge, such as local students and overseas students.
Issue 6Some students may misunderstand that some of the pre-defined subjects sound the same, so they decide to just take one of these subjects, but in the curriculum design, they should do all subjects.
Issue 7Some subject is better by using a problem based learning approach, rather than general concepts study or strictly follow what is defined in the curriculum design.
Issue 8Some academics want to update the subject to tailor to the advanced technologies and methodologies, however, sometimes the changes are not permitted, and out of date materials reused.

The above inputs from academics are important to the quality of teaching. However, this is normally managed in an ad hoc fashion, and no follow up is carried out. We believe that the academics are the implementers and their input should be captured and systematic analysis of their input should take place in the quality teaching process.

Feedback from students

The systematic capture of students' feedback on the curriculum is missing. There are many issues that may arise from students' concerns regarding the entire degree program, such as the following.

Issue 1  Many subjects are too abstract or too general. The knowledge gained from the teaching is minimal.
Issue 2Subjects may be too heavily overloaded with content, and students have difficulty in assimilating and understanding the over-loaded curriculum content, notwithstanding that the subject matter may be excellent.
Issue 3Subjects may in fact be content sparse and students, especially advanced students, may find them uninteresting and unuseful.
Issue 4Students may comment on which subject or subjects are more suitable as prerequisites.
Issue 5Students get different advice from different administrators on choosing subjects. Sometimes this causes difficulties in learning, because they do not have any background to support the current subject study.
Issue 6The student could make accurate comments on what subject is overlapping with another subject, which subjects is a total waste of time. There are horizontal overlapping (such as number of subjects taught in one year) and vertical overlapping (which is the subject overlapping between different years).
Issue 7The concerns about the mixture of local and international students, and the differences in knowledge and skills. Such as computer skills or English skills.
Issue 8Significant concerns may arise due to the learning styles of students from different cultural, and especially educational backgrounds - especially where teaching styles may have actually been quite authoritarian and learning by rote the norm.
Issue 9They often give suggestions on the upgrading of subject content or materials, or suggestions of a number of advanced topics. Some student may be well advanced and may be able to suggest the proper subjects to be taught to address social and economic needs.

Conclusion

In summary, each of the triple feedback mechanisms, from academics and students as well as industry are equally important to achieve world class teaching and learning. A systematic approach to capture the two missing feedback mechanisms to curriculum design should be developed. Without triple feedback system, one cannot make conclusions about the quality of curricula, teaching and learning.

The Curriculum design is a blue print for the subject development. If the framework of the curriculum is not well designed, and feedback from academics and students is not considered, one cannot sure that the curriculum planning and design are of good quality. In this paper, we point out that feedback from academics and students to the curriculum design is equally important and a systematic approach to capture the feedback should be developed. It is important that each stage (planning, design, development, delivery) have feedbacks, not just at implementation level. With advance in IT, the automated system feedback system, document management and automatic classification of knowledge would enhance the feedback collections.

Figure 5

Figure 5: The proposed triple feedback loop to curriculum improvement

References

Australian Government Department of Education, Science and Training (DEST) (2003). 2003/04 Budget paper: Outcomes and Output Group Information, page 15.

Curtin University of Technology (2002). Curtin Strategic Plan on Teaching and Learning, 2002-2005.

Glatthorn, A., Harris, D.E. & Carr, J.F. (2003). Curriculum Handbook - Chapter: Planning and Organizing for Curriculum Renewal. Association for Supervision and Curriculum Development (ACSD). [viewed Oct 2003, verified 7 Jun 2004] http://www.ascd.org/handbook/demo/planning2.html

March, C.J. & Willis, G. (2003). Curriculum alternative approaches; Ongoing issues. 3rd Edition. Merrill Prentice-Hall. Chapter 1, Section 1.2 Defining Curriculum, pages 7 to 14.

Morien, R. & Schmidenberg (1994). Educating Information Systems Professionals: The Tertiary Education Challenge. APITITE '94 (Asia Pacific Information Technology in Training and Education), Brisbane, June.

Authors: R. Morien, E. Chang, K.F. Chin and C. Cheah
School of Information Systems, Curtin University of Technology
Email: cheahc@cbs.curtin.edu.au

Please cite as: Morien, R. Chang, E. Chin, K.F. and Cheah C. (2004). Systematic approach to triple feedback systems for teaching enhancement . In Seeking Educational Excellence. Proceedings of the 13th Annual Teaching Learning Forum, 9-10 February 2004. Perth: Murdoch University. http://lsn.curtin.edu.au/tlf/tlf2004/morien.html


[ Proceedings Contents ] [ Teaching and Learning Forum 2004 ]
This URL: http://lsn.curtin.edu.au/tlf/tlf2004/morien.html
Created 19 June 2004. Last revision: 19 June 2004.