Validation study of a method for assessing complex ill-structured problem solving by using causal representations

Abstract

The important but little understood problem that motivated this study was the lack of research on valid assessment methods to determine progress in higher-order learning in situations involving complex and ill-structured problems. Without a valid assessment method, little progress can occur in instructional design research with regard to designing effective learning environments to facilitate acquisition of expertise in complex, ill-structured knowledge domains. In this paper, we first present a method based on causal representations for assessing progress of learning in complex, ill-structured problem solving and discuss its theoretical framework. Then, we present an experimental study investigating its validity against adapted protocol analysis. This study explored the impact of a massively multiplayer online educational game, which was designed to support an interdisciplinary STEM education on ninth-grade students' complex, ill-structured problem solving skill acquisition. We identify conceptual similarities and differences between the two methods, present our comparative study and its results, and then discuss implications for diagnostics and applications. We conclude by determining how the two approaches could be used in conjunction for further research on complex and ill-structured problem solving.