Investigating the effectiveness of an online course : development of the comparative learning environment questionnaire

Abstract

This study was undertaken with the purpose of evaluating a newly-developed online course. The study involved, firstly, designing, developing and validating two questionnaires that could be used to assess the relative effectiveness of the online course in terms of students’ perceptions of the learning environment and attitudes. The development of the learning environment instrument drew on and extended the wide array of already validated instruments in the field of learning environments. The development used a multi-stage approach that included a review of pertinent literature, identifying salient scales and developing pertinent survey items. Both surveys were field tested and refinements were made on the basis of the findings. Its unique format allowed the comparison of the newly-developed online course with a parallel face-to-face course. The learning environment survey assesses the six scales of Task Orientation, Responsibility and Independence, Access, Computer Usage, Authentic Learning, Information Design and Appeal and the attitude instrument is comprised of three scales, namely, Student Enjoyment, Academic Efficacy and Anxiety.The learning environment survey and attitude survey were administered to 1000 students, of whom 991 provided complete and usable responses. Analyses of the data obtained from 991 students were conducted to support the factorial validity and internal consistency reliability of both the learning environment and attitude instruments. For the learning environment instrument, principal axis factoring with varimax rotation (conducted separately for the online and face-to-face course) confirmed that, without exception, all items had a factor loading of at least 0.40 on their a priori scale and no other scale. The scale reliability estimates were high for both versions of the learning environment instrument, ranging from 0.85 to 0.92 for the online course and from 0.85 to 0.91 for the face-to-face environment. The attitude instrument also was found to be reliable. Factor analysis confirmed the a priori structure of the student attitudes instrument for the online and face-to-face courses comprising 18 items in three scales. All items had a factor loading of at least 0.40 on their a priori scale and no other scale. The Cronbach alpha coefficient for the online and face-to-face course for each of the three attitude scales ranged from 0.91 to 0.95 for online course and from 0.91 to 0.96 for face-to-face course.Qualitative information from focus-group interviews with 90 students were used to supplement and support the quantitative results. Interview data collected from 90 students who took part in the semi-structured interviews and focus-group discussions helped to confirm and provide reasons for the quantitative findings that students generally had more positive views of their online course than the face-to-face course.Data collected using the learning environment and attitude surveys were used to compare students’ scores for the online and face-to-face courses using MANOVA. The results indicated statistically significant (p<0.05) differences for all six learning environment and three attitudes scales, with students’ scoring higher for their online learning environment. For the six learning environment scales, had enjoyment and academic efficacy lower for anxiety when compared to the face-to-face course. In addition, the effect sizes (calculated to determine the magnitude of the differences in standard deviations) were found to be large (ranging from 0.39 to 1.00) for all of the learning environment and attitude scales. The qualitative information supported these findings in all cases, and helped to explain the reasons why.Associations were found between students’ perceptions of the online learning environment and their attitudes (Enjoyment, Academic Efficacy and Anxiety). The multiple correlation were statistically significant for all three outcomes. The regression weights (ß) indicated that five of the six learning environment scales were positively, significantly (p<0.05) and independently related to Enjoyment and Academic Efficacy, and two of the six learning environment scales were negatively, significantly (p<0.05) and independently related to Anxiety. These result suggest that, to increase the likelihood of students’ enjoyment of the online course, the instructional design should incorporate material that is presented in manageable amounts with clear guidelines and instructions (Task Orientation), provide opportunities for students to proceed through the course at their own pace and to repeat components that have not been understood (Access), use authentic tasks to which students can relate (Authentic Learning) and use audio and visual materials, tables and graphics to assist students to understand the content (Information Design and Appeal). Two environment scales, Access and Information Design and Appeal, were statistically significantly (p<0.05) related to Anxiety. with students studying the online course being less anxious if there was more Access and Information Design and Appeal.