An Illustrative Example of the Benefits of Using a Rasch Analysis in an Experimental Design Investigation

Abstract

The study compared the consequences of choosing to use either deterministic or probabilistic data analyses in an experimental investigation. The empirical research from which the data for these analyses was drawn applied computer assisted language learning (CALL) as the treatment in a one-group pretest-posttest design. The empirical investigation concerned the effect of CALL on Korean university students’ ability to correctly identify the meaning of loanwords - native vernacular that originated in nonnative languages and is now part of the native vernacular. The empirical investigation is explained and the design of experimental research and analysis of experimental research data are discussed. Stochastic and deterministic measurement models are then examined followed by the application of these models to analyse data from the empirical investigation. Data analyses included a paired sample t-test, one-way analysis of variance (ANOVA) and a Rasch Unidimensional Measurement Model (RUMM) analysis of differential item functioning (DIF). The analyses and their respective results are assayed in terms of capacity to inform hypothesis confirmation. Parametric tests using non-interval data (raw test scores) were shown to be less sensitive than the RUMM DIF analysis of Rasch Model transformed scores when estimating the differences between the pretest and posttest data.