Teaching and Learning Forum 2001 Home Page
[ Teaching and Learning Forum 2001 ] [ Proceedings Contents ]

End user computing: Learning style differences as a predictor of training outcomes

Raj Gururajan
School of Computer and Information Science
Edith Cowan University

Introduction

Studies in EUC training have focused on traits, especially the impact of experience and cognitive style variables in their design and use (Bohlen & Ferrat, 1997; Davis & Bostrom, 1993; Sein, Olfman, Bostrom, & Davis, 1993). However, studies that have considered cognitive and learning style variables have failed to produce the theoretical foundation on which training outcomes were explained. This has resulted in studies failing to defend the importance of these variables and resulted in contradictory outcomes.

Despite few studies, a number of studies in EUC training studies have ignored these variables in a training environment. Suggestions by scholars such as Robey (1982) have invoked little response in EUC community to investigate these variables. The Bostrom's (1990) framework was an attempt to take the suggestions and despite such attempts only few studies have investigated this aspect of individual differences. Since 1997 only Bohlen (1997) has considered this aspect in his study. Even he has failed to explain the theoretical background in reporting his outcomes. This study examines user individual differences of learning in a training environment.

Individual differences

Prior studies in EUC have focused on individual difference variables associated with descriptive traits such as anxiety, attitudes, reasoning etc. Barring learning styles, these individual difference variables haven't been able to serve the role of consistent predictor of learning outcomes. The exploratory nature of prior studies has clearly indicated that learning style is a consistent predictor and an important variable in the context of learning software applications.

Studies in EUC training have established that learning style variable is an important predictor of performance, both by itself and in interaction with training methods (Bostrom, Olfman, & Sein, 1990, p. 106). The learning style is defined as the knowledge of skills and cognitive factors that individuals possess during a learning sequence (Presland, 1994). It has been an accepted fact that individuals change their learning styles during a course of a learning sequence (Honey & Mumford, 1992). The dimensions along which such changes occur have been studied in learning theory. In training domain, to study such changes, one should first identify learner characteristics and then plan their training settings to accommodate changes. Studies in instructional psychology have demonstrated that it is necessary to adapt instructional methods and teaching strategies to accommodate key individual differences (Bostrom, Olfman, & Sein, 1990). EUC studies (Bohlen & Ferrat, 1997; Davis & Bostrom, 1993) have recommended that individual learning styles be determined before training is provided in order to measure outcomes.

There are multiple competing learning style theories available. The single learning style continuum argues that each individual can be placed somewhere on a bipolar scale. Examples of such fields are field independent/dependent scales (Witkin & Goodenough, 1988). The definite learning style model proposes that each person has one of finite number of learning styles. Examples of this are serialist/holist classification (Pask & Scott, 1972). The situational learning style model postulates that individuals are able to select from a number of possible learning styles, depending on the learning task at hand. Examples are surface/deep processing tasks (Marton, 1976). The multidimensional learning style model specifies that each person has a different degree of combination of styles. Examples are analytic/intuitive dichotomy (Pask & Scott, 1972). The current state of this theoretical development suggests that there is no clear agreement on a universal learning style theory or measurement.

In order to measure learning styles in EUC training studies, two instruments have been widely used. The first instrument is Kolb's Learning Style Inventory (KLSI). The instrument is based on experiential learning. The theory views learning as a discovery process that incorporates the characteristics of problem solving and learning. Ruble and Stout (1993) criticised Kolb's instrument for its validity in EUC training studies. The criticism was laid on the poor psychometric properties of KLSI. In answering to the criticisms, Bostrom et al. (1990) have accepted this fact. Further, it appears that many studies that have used KLSI were conducted in a tertiary setting where there is time for change in learning styles. However, in short training studies, such changes may not happen. So, it is possible to assume that learning style is stable for the duration of study in a short training program.

The other instrument used is Honey and Mumford's Learning Cycle. Honey & Mumford (1986) modified Kolb's approach and produced a model called the learning cycle. In this model, the learners are classified according to their strengths and weaknesses compared to their preferences. The model suggests four contrasting stages of a learning cycle.

Honey and Mumford (1986; 1992) modified Kolb's approach into learning cycle and classified learners in terms of their strengths and weaknesses for each stage of the cycle. They suggest four contrasting stages of a learning cycle. Activist are people who involve themselves in new experiences, tackling problems by brainstorming, and moving from one task to the next as the excitement fades. Reflectors are cautious and thoughtful people who like to consider all the possible angles before making any decisions and whose actions are based on observations and reflections. Theorists are people who integrate their observations into logical models based on analysis and objectivity. Pragmatists are practical people who like to apply new ideas immediately, and get impatient with an over emphasis on reflection. A wholly effective learner has the abilities characteristic of all four stages. However, such ideal learners are rare. It is mentioned that no one particular style is better than the other.

Training approaches and outcomes

Previous studies in EUC reveal that learning is facilitated through training (Borgman, 1986; Bostrom, Olfman, & Sein, 1990; Carnevale & Carnevale, 1994). In EUC, training approaches have been used to integrate existing knowledge with previous knowledge to derive new knowledge. However, few studies in the early 1990s attempted to rectify this problem by using a research framework where a link to training is provided by mapping via training (Bostrom, Olfman, & Sein, 1990; Davis & Bostrom, 1993; Sein, Olfman, Bostrom, & Davis, 1993). These studies suggested that learners learn by either exploring the features of application software or learn by following the instruction given in a step by step manner. These two approaches are classified as the exploration oriented training approach and the instruction oriented training approach.

These two training approaches represent radically different users. While the exploration oriented training approach facilitates users to trial and error features, the instruction oriented approach provides little user control. The two approaches also feature deductive and inductive orientations respectively. Previous studies clearly indicated that the issue of suitability of training approaches for EUC training is yet to be resolved. While certain studies advocate the supremacy of the exploration oriented approach (Black, Carrol, & McGuigan, 1987; Carrol, Smith-Kerker, Ford, & Mazur-Rimetz, 1987; Kamouri, Kamouri, & Smith, 1986), other studies have established that the instruction oriented approach is effective in EUC training (Davies, Bagozzi, & Warshaw, 1989; Gomez, Egan, & Bowers, 1986). These studies reveal that there is no agreement regarding training outcomes.

Despite the disagreements in EUC training outcomes agreement, some studies have agreed that the primary role of training approaches in EUC training should be to provide meaningful learning through the integration or assimilation of new information in short term memory and knowledge from long term memory (Davis & Bostrom, 1993). However, in order for this process to occur, learners must actively work with both prior knowledge and new information. So, training materials, which support this process should be considered in training approaches. Studies have emphasised the importance of training materials in defining and deciding upon which training approaches to use.

To assist training approaches, preparation of training materials should be considered in terms of three components: concepts, procedures and usage of a given software application. The preparation of training materials should focus on the features of application software in EUC training (Gentry, 1994). The training material features considered in the previous studies can be classified under two categories: process features and structural features. The above discussion highlights the need for proper construction of training materials. The process and structural features will elaborate three further components: concepts, procedures and usage. In this study, the usage component refers to both the functional elements of software packages as well as the interfaces for measurement purposes. Hence, these features will be provided in terms of instruction orientation and exploration orientation. These two orientations will constitute the training approaches variable. Studies conducted by Bostrom (1990), Davis (1993), Bohlen (1997), Davies (1985) and Olfman (1994) have clearly indicated that there is a link between training approaches, user differences and training outcomes.

Research methodology

The previous studies have indicated that learning preferences have an influence on training outcomes. Studies have insisted and recommended that EUC studies extract learning style preferences prior to the commencement of the experiment as these styles serve as the predictor of training outcomes (Bohlen & Ferrat, 1997; Davis & Bostrom, 1993). Therefore, it is safe to assume that the learning style preferences are a determinant to EUC training outcomes. This leads to the following research question:

Research Question 1:

Are end user training outcomes affected by different user learning styles when users learn software (using short training programs)?
The literature review indicates that the learning style preferences affect outcomes. However, there is very little evidence available as to which learning style is superior in a given situation. Very little experimental research has been done to verify that end user training outcomes are related to method of instruction and learning styles. If training outcome are affected by method of instruction and learning style as suggested by Bostrom (1990); Davis (1993) and Sein (1999), then these factors should be considered while training end users. Therefore, to address the first research question, the following null form of hypotheses are generated and stated in the following hypotheses.
  1. There will be no difference in effectiveness due to learning style preferences.
  2. There will be no difference in efficiency due to learning style preferences.
Previous studies have established that training approaches determine training outcomes (Davis & Bostrom, 1993; Sein, Bostrom, & Olfman, 1999). While certain studies advocate instruction oriented approach, others advocate exploration oriented approach. Despite the disagreement in the approaches, EUC studies agree on the fact that training approaches play a crucial role in determining training outcomes. This leads to the following research questions:

Research Question 2:

Are end user training outcomes affected by training approaches when users learn software (using short training programs)?
As indicated earlier, the two training approaches - instruction and exploration - accommodate radically different styles. The instruction approach supports inductive approach and the exploration supports deductive approach. The learners will be expected to possess different styles for these two approaches. While the instruction approach supports learners who depend upon complete set of instructions, exploration supports learners who would like to experiment with the available functions in a software application. Therefore, to address the second research question, the following hypotheses are stated:
  1. There will be no difference in effectiveness due to training type.
  2. There will be no difference in efficiency due to training type.
Previous studies have also indicated that training approaches interact with user styles in determining training outcomes. This is especially true for information processing needs. In other words, the interaction between learning styles and training approaches has an influence in determining training outcomes. This leads to the following two hypotheses:
  1. There will be no difference in effectiveness due to the interaction of training type and learning style preference.
  2. There will be no difference in efficiency due to the interaction of training type and learning style preference.
To test the hypotheses, an experiment was conducted. To categorise users into learning styles, Honey & Mumford's instrument was used. The following paragraphs explain the experimental procedure.

Experimental procedure

The research was conducted in a classroom setting. 180 subjects participated in the study. The subjects were administered with Honey & Mumford's questionnaire to assess their learning preferences. The research design was a two factor experimental design. Learning style is one factor, while the training approach is the other. Subjects were classified into one of the four categories of Honey and Mumford's learning style preferences. The subjects in each of the category were asked to choose their preferred training approach based on their experience.

Participants

The participants of the research were tertiary end user computing students enrolled in a computer science program. The participants possess limited IT knowledge. They range from 18 years to 40 years in age. Participants were drawn from Computer Science, Information Technology, Mathematics, Food Science, Aviation, Software Engineering and Sports Science courses. Participants have sufficient knowledge of PC operations. The participants were administered with Honey and Mumford's Learning Style Questionnaire (LSQ) to categorise them into learning style groups. The grouping is to establish a relationship between types of learning preferences. The participants filled in a set of questionnaires to determine their level of knowledge and experience prior to the LSQ.

Training outcomes

Effectiveness

Based on previous studies, effectiveness is defined for this experiment in terms of "score" gained by the number of steps used to conduct a task, number of errors committed and the number of backtracks in completing a step (Davis, 1993, p34; Bohlen, 1997 , p17; Olfman, 1995 p.344). To be effective, participants would use minimum number of steps with precision. It is difficult to predefine the minimum and maximum scores for given tasks as participants may opt to conduct a step in a task in any arbitrary manner leading to a varying combination of keystrokes. Therefore, the measure is mentioned as a function of various types of strokes. The effectiveness could be mathematically defined as:

Effectiveness = function (correct strokes, icon access, menu access, dialogue box interaction, errors, backtracks)

This could be mathematically shown as

Effectiveness = f(CS, IA, MA, DB, BTRK, ERR)

Efficiency

Based on previous studies, efficiency is defined for this experiment in terms of "time" taken to a complete a task (Bohlen and Ferrat, 1997, p17; Bostrom, 1990, p19; Sein, 1993, p343). The factor time is directly proportional to the number of keystrokes. It could be mathematically defined as:

Efficiency = function (keystrokes, time)

This could be shown mathematically as Efficiency = f(KS, T)

Experiment

As mentioned previously, 180 undergraduate students from an introductory computer science course volunteered to participate in the study. The students were selected on the basis of having had little or no previous experience with a project management software application. Also, all the students reported that they are conversant with PC operations.

The experiment was organised into 4 sessions of about 30 minutes each. The first session was a briefing session and the Learning style preference questionnaire was filled in by users. The second session was used for training. The third session was used for a 12 task hands on exercise. The fourth session was used for filling in the satisfaction questionnaire.

The students were provided with training manuals. The training manuals were prepared based on Wood's (1990, p164) task complexity model. The training manual was examined by two independent judges for suitability and approved for the purpose of this research. The training manual consisted of actions for both icon and menu operations. So, students were able to choose either one of the styles. To guide students to follow steps either with icons or with menus, a number of verbal and imagery type of clues were provided.

In addition to various guiding instructions, the training manuals provided a number of information cues to students. Whenever students committed an error, a provision to recover from the error was given in the manuals.

During the training phase, subjects were allowed to work on the training manual for 45 minutes. The time restriction was to comply with various administrative procedures. In addition to this (45 minutes), subjects were given with another 45 minutes to work with various examples. These two sessions were held on different days in order to meet administrative procedures in booking computer laboratories.

Once the training and the example exercises were completed, subjects were administered with a hands on exercise. The hands on exercise consisted of 12 tasks of a project management schedule. Solutions to the tasks were recorded using Lotus ScreenCam program for playback and recording. The hands on task was recorded using a Lotus ScreenCam software. The entire hands on task was recorded and the average size of the file was about 4 MB. Replaying the file collected the responses. This operation took about 45 minutes per participant.

Subjects were asked to playback their solutions and record the number of accesses to menus, icons, number of keystrokes activated, correct keystrokes, backtracks, erroneous strokes and any interaction with dialogue boxes. These were used to compute the effectiveness. The time was recorded using the computer clock and was used to compute efficiency.

Data analysis

The Learning Cycle questionnaire was derived from Honey and Mumford (1992, p265). The questionnaire consisted of 80 items. Participants have to answer either "yes" or "no" by placing a tick or a cross for each item. Honey and Mumford (1992, p265) have provided a scoring mechanism to determine the learning style of participants and this was applied to derive individual learning style preferences. Prior to determination of learning styles, the Learning Cycle questionnaire was tested for reliability (for overall questionnaire) and an alpha of over 0.90 was obtained, indicating that the questionnaire was reliable.

When the data was initially analysed, it was found that some students had failed to complete the tasks or failed to save the files properly. This has resulted in an elimination of 20 students from the data analysis.

Initially the data was tested for normal distribution and was found to be normal. When the regression analysis was performed, it was found that he variables also correlated well. This was established by performing a regression analysis with training type and learning style preferences as two variables (this is not shown in this paper). The data was then analysed to examine various trends.

Efficiency

It can be seen from the box plot in Figure 1 (given below) that the mean value of exploration group is higher for every learning style. It was mentioned that the efficiency is calculated in terms of time and the higher means show that the groups consumed more time for exploration orientation that instruction orientation. The theorists style and pragmatists have scored almost equal means. Activists and reflectors have scored considerably different means. This indicates that instruction group is efficient in terms completing the given task in terms of time components.

Figure 1

Figure 1: Box plot for efficiency

Effectiveness

From the box plot in Figure 2, it can be noticed that exploration group has scored a higher mean for the three learning styles - activist, reflector and theorist. The pragmatists have scored lower means for exploration orientation. This indicates that the exploration orientation is effective in terms of obtaining higher scores.

Figure 2

Figure 1: Box plot for effectiveness

Table 1 shows the mean and standard deviation values for the training outcome efficiency and effectiveness respectively. The mean values for the outcomes efficiency and effectiveness are comparable for training orientations and learning styles. In addition to these, the variances are also comparable. When, a regression analysis was performed for both effectiveness and efficiency, a normal curve was yielded asserting the data is normal (this is not shown in the paper). This collective trend was interpreted as the existence of strong evidence for a univariate analysis.

Table 1: Mean and SD for training outcome efficiency and effectiveness Table 1

Table 1

The results of analysis of variance (shown in Tables 2 and 3) indicate that none of the effects (training type, learning style preference and the interaction) were significant at 0.10 level. The analysis performed reveals the nature of the main effects and the interaction effects. The analysis shows relatively small R-square values for efficiency and effectiveness. These small R-square values indicate that the models did not account for a good deal of variation in these dependent variables.

Table 2: Efficiency

Source dfFSig.
Corrected Model 7.969.456
Intercept 11304.738.000
TRGTYPE 1.691.407
LSTYL 3.671.571
TRGTYPE * LSTYL 3.764.516
Error 152

Total 160

Corrected Total 159


Table 3: Effectiveness

Source dfFSig.
Corrected Model 7.516.821
Intercept 11046.018.000
TRGTYPE 1.888.348
LSTYL 3.425.736
TRGTYPE * LSTYL 3.434.729
Error 152

Total 160

Corrected Total 159

a. R squared = .023 (adjusted R squared = -.022)

The F test performed indicates that the main effects and interaction effects are not significant. For efficiency outcome, the F-values are F(Training approach, 1) = 0.691; F(Learning style, 3) = 0.671. For effectiveness outcome, the F-values are F(Training approach, 1) = 0.888; F(Learning style, 3) = 0.425. All the values are well over the significant levels and this is an indication that the null hypotheses cannot be rejected. This is confirmed by the p-values over the level of significance for every hypothesis.

Discussion

Despite that there is no statistical evidence to reject the hypotheses, there is clear evidence that instruction training is superior in terms of efficiency. The learning style preference groups - activists and reflector - have scored significantly lower mean values than the theorists and pragmatists. This is translated as the time taken to complete the hands on tasks is much lower for the two groups mentioned previously.

In terms of effectiveness, the exploration groups have scored significantly higher means for activists, reflectors and theorists groups. Pragmatists have scored lower average. This can be translated as the exploration training treatment yielding significantly better results compared to the instruction group in terms of scores.

This also supports the proposition of Assimilation Theory. Subjects used their previous knowledge to derive new knowledge in order to achieve meaningful learning. Theorists who have undergone training can recall their conceptual knowledge to reduce their time in performing given tasks. On the other hand, people who have explored the application, found it difficult to complete the tasks in short time duration because of the lack of previous knowledge. The exploration group found it difficult to arrive at a meaningful learning which was essential to conduct the tasks quickly. This is shown in the outcome efficiency. Subjects have taken considerable time to absorb the new knowledge when it comes to exploration. The instruction approach has provided conceptual models to subjects. The conceptual models provided a context in which thinking is facilitated for reasoning purposes. In the case of instruction training type, subjects are provided with assimilative contexts with set of instructions and step by step procedures reflecting the functions of the application. Previous studies have confirmed this trend.

The exploration, on the other hand, allowed users to carry out a task based on the semantic distance. In other words, the semantic distance, which is relationship between a user's conceptualisation of an operation and the mechanisms that the training type provides to carry it out, is facilitated through the deduction process. In this study, the instruction training closely represents the user's conceptual model and hence semantically direct. Exploration training based subject were not able to do this because of the number of steps involved and the complex conceptual model provided by the deductive process.

It should be noted that the study supports the concept of using instructions to train end users when the application software is difficult to learn. A number of previous studies have supported this concept. However, there are studies, which have shown that this is not the case. The differences could be attributed to the lack of classification followed in this study. Further, the study did not categorise the tasks into simple and complex as defined by Mayer and hence this could have an impact in the disparity of the result.

Another aspect that is worth noting is the task itself. Despite the fact that the tasks are evaluated for appropriateness, and the model followed to create the tasks was Wood and Campbell's model, the tasks were not evaluated for their complexity. It appears that there are no universal guidelines available for such a purpose. This could have influenced the outcome of training to some extent.

Limitations and conclusion

It was mentioned that the subjects were drawn from an entry level course in tertiary settings. However, this study did not make an attempt to extract the background information regarding the computing experience and usage to a greater depth. It was assumed that the subjects were accustomed to a structured environment. However, the validity of such assumption can be questioned in end user computing studies, as most of the end users were self taught. Another aspect that could be debated was the duration of training. This study allocated a 45-minute duration for training and hands on tasks testing. Literature provides evidence that this ranges from 30 minutes to 8 hours. This aspect could be one influencing factor in assimilation of knowledge. Further research needed in this area.

References

Black, J. B., Carrol, J. M., & McGuigan, S. M. (1987). What kind of minimal instruction manual is the most effective? Paper presented at the Human factors in computing systems, New York.

Bohlen, G., & Ferrat, T. (1997). End user training: An experimental comparison of lecture versus computer based training. Journal of End User Computing, 9(3), 4-27.

Borgman, C. L. (1986). The user's mental model of an information retrieval system. International Journal of Man Machine Studies, 24(1), 47-64.

Bostrom, R. P., Olfman, L., & Sein, M. K. (1990). The importance of learning style in End User Training. MIS Quarterly, 14(1), 101-119.

Carnevale, A., & Carnevale, E. (1994). Growth pattern in workplace training. Training and development, 48, 22-28.

Carrol, J. M., Smith-Kerker, P. L., Ford, J. R., & Mazur-Rimetz, S. A. (1987). The minimal manual. Human Computer Interaction, 3(2), 123-153.

Davies, F. D., Bagozzi, R. P., & Warshaw, P. R. (1989). User acceptance of computer technology: A comparison of two theoretical models. Communications of the ACM, 35(8), 982-1003.

Davis, S., & Bostrom, R. (1993). Training end users: An experimental investigation of the roles of the computer interface and training methods. MIS Quarterly, (March), 61-79.

Gentry, C. G. (1994). Introduction to instructional development: Process and technique. Belmont, CA: Wadsworth Publishing Company.

Gomez, L. M., Egan, D. E., & Bowers, C. (1986). Learning to use a text editor: Some learner characteristics that predict success. Human Computer Interaction, 2(1), 1-23.

Honey, P., & Mumford, A. (1986). Using your learning styles. Maidenhead-Berkshire: Peter Honey.

Honey, P., & Mumford, A. (1992). The manual of learning styles. Maidenhead-Berkshire: Peter Honey.

Kamouri, A. L., Kamouri, J., & Smith, K. (1986). Training by exploration: Facilitating the transfer of procedural knowledge through analogical reasoning. International Journal of Man-Machine Studies, 24(2), 171-192.

Marton, F. (1976). What does it take to learn? Some implications on an alternative view of learning. In N. J. Entwistle (Ed), Strategies for Research and Development in Higher Education, 200-222.

Pask, G., & Scott, B. C. E. (1972). Learning strategies and individual competence. International Journal of Man-Machine Studies, 4, 217-253.

Presland, J. (1994). Learning Styles and Continuous Professional Development. Educational Psychology in Practice, 10, 179-184.

Robey, D., & Taggart, W. (1982). Human information processing in information and decision support systems. MIS Quarterly, 6(1), 61-73.

Ruble, T. L., & Stout, D. E. (1993). Factors of success for end user computing: An unwanted leap in faith. MIS Quarterly, 17(1), 115-118.

Sein, M. K., Bostrom, R. P., & Olfman, L. (1999). Rethinking end user training strategy: applying a hierarchical knowledge level model. Journal of End User Computing, 11(1), 32-39.

Sein, M. K., Olfman, L., Bostrom, R., & Davis, S. (1993). Visualization ability as a predictor of user learning success. International Journal of Man-Machine Studies, 39, 599-620.

Witkin, H. A., & Goodenough, D. (1988). Cognitive Styles, Essence and Origins: Field Dependence and Field Independence. New York: International University Press.

Author: Raj Gururajan, School of Computer and Information Science, Edith Cowan University. r.gururajan@cowan.edu.au

Please cite as: Gururajan, R. (2001). End user computing: Learning style differences as a predictor of training outcomes. In A. Herrmann and M. M. Kulski (Eds), Expanding Horizons in Teaching and Learning. Proceedings of the 10th Annual Teaching Learning Forum, 7-9 February 2001. Perth: Curtin University of Technology. http://lsn.curtin.edu.au/tlf/tlf2001/gururajan.html


[ Abstract for this article ] [ TL Forum 2001 Proceedings Contents ] [ All Abstracts ] [ TL Forums Index ]
HTML: Roger Atkinson, Teaching and Learning Centre, Murdoch University [rjatkinson@bigpond.com]
This URL: http://lsn.curtin.edu.au/tlf/tlf2001/gururajan.html
Last revision: 8 Feb 2002. Curtin University of Technology
Previous URL 22 Dec 2000 to 8 Feb 2002 http://cleo.murdoch.edu.au/confs/tlf/tlf2001/gururajan.html