Which form of assessment provides the best information about student performance in chemistry examinations?

Abstract

Background: This study developed from observations of apparent achievement differences between male and female chemistry performances in a state university entrance examination. Male students performed more strongly than female students, especially in higher scores. Apart from the gender of the students, two other important factors that might influence student performance were format of questions (short-answer or multiple-choice) and type of questions (recall or application). Purpose: The research question addressed in this study was: Is there a relationship between performance in state university entrance examinations in chemistry and school chemistry examinations and student gender, format of questions - multiple-choice or short-answer, and conceptual level - recall or application? Sample: The two sources of data were: (1) secondary analyses of five consecutive years' data published by the examining authority of chemistry examinations, and (2) tests conducted with 192 students which provided information about all aspects of the three variables (question format, question type and gender) under consideration. Design and methods: Both sources of data were analysed using ANOVA to compare means for the variables under consideration and the statistical significance of any differences. The data from the tests were also analysed using Rasch analysis to determine differences in gender performance.Results: When overall mean data are considered, both male and female students performed better on multiple-choice questions and recall questions than on short-answer questions and application questions, respectively. When overall mean data are considered, male students outperformed female students in both the university entrance and school tests, particularly in the higher scores. When data were analysed with Rasch, there was no statistically significant difference in performance between males and females of equal ability. Conclusions Both male and female students generally perform better on multiple-choice questions than they do on short-answer questions. However, when the questions are matched in terms of difficulty (using Rasch analysis), the differences in performance between multiple-choice and short-answer are quite small. Rasch analysis showed that there was little difference in performance between males and females of equal ability. This study shows that a simple face-value score analysis of relative student performance - in this case, in chemistry - can be deceptive unless the actual abilities of the students concerned, as measured by a tool such as Rasch, are taken into consideration before reaching any conclusion.