Empirical comparison of tree ensemble variable importance measures

Abstract

Tree ensembles are becoming well-established as popular and powerful data modelling techniques. Tree ensemble models are essentially black box models, although their individual members may not be, and with their growing popularity, interest in the interpretation of tree ensemble models has also grown. This study presents variable importance measures associated with random forests, conditional inference forests and boosted trees, and employs a number of simulated data sets to compare these methods. Overall, variable importance indicators based on bagged conditional inference forests appear to strike a good balance between identification of significant variables and avoiding unnecessary flagging of correlated variables. Data preprocessing and interpretation by experts knowledgeable with a specific data set remain vital.