An evaluation framework for peer-based youth programs

Abstract

The aim of this research study was to develop a theory-based and practice-based evaluation framework and evaluation approaches for peer-based youth programs (such as drop-in spaces and after school programs) which target young people aged 12-24 years at risk of poor mental health outcomes.The study used a participatory action research design and qualitative inquiry methods. The methodology was informed by two pilot studies and required the collaboration of eight youth service provider organisations and four Curtin University research centres. Staff, volunteers and young people from 12 peer-based youth programs participated in the study. The research was conducted in three stages.The first stage of the research collected data from practitioners concerning program objectives, the features of programs they considered important for program effectiveness, and their experience of program evaluation including the barriers and enablers associated with evaluating programs for high risk youth populations. The data were used to generate keywords for an analysis of the published literature to identify sources of literature that would support or refute the data from practitioners, including relevant theories, models and evaluation tools and methods. A draft conceptual model for peer-based programs resulted from Stage 1 and provided the basis for the final evaluation framework.Stage 2 of the research was a participatory action research process resulting in the development of an evaluation framework for peer-based youth programs informed by theory and practice. A second output was a range of qualitative and quantitative evaluation approaches developed in conjunction with the participating practitioners and five young people recruited as peer research assistants. The evaluation framework and selected evaluation approaches were tested in three programs with limited support from the researcher. In Stage 3, the data were consolidated to identify implications for practice, policy and future research.The study identified 11 barriers to program evaluation mainly related to practitioners’ limited skills and knowledge in evaluation and limited capacity to implement evaluation activities. Inadequate program funding and the flexibility required within programs to be responsive to the needs of at risk youth contributed to unclear objectives and low levels of evaluation.The study contributed to the literature by identifying seven additional features of effective programs that may be particularly relevant for at risk youth populations. Previous studies have not focused on at risk youth populations. The focus of previous research has also centred on programs which promoted academic and employment outcomes rather than early intervention for mental health and wellbeing.In addition, the study articulated theory-based and practice-based evaluation parameters (the evaluation framework) which could be used to measure the effectiveness of peer-based programs and piloted 23 evaluation approaches. The evaluation approaches that were piloted included both traditional methods (such as questionnaires) and innovative methods (such as arts-based methods). Evaluation approaches which could be integrated into normal program activities and achieved high levels of youth engagement were found to be particularly effective and sustainable.The evaluation framework and evaluation approaches are now available to be tested in other types of peer-based programs and contexts. Evaluation of informal social programs is challenging and further efforts will be needed to build capacity within the youth sector to fully translate the outputs of this research into practice. Recommended strategies to achieve this goal could include advocating to funding bodies for alternative systems of measuring the effectiveness of peer-based programs, the development of standard indicators for measuring mental and social wellbeing in young people, and building evaluation capacity within the youth sector.