Comprehensive primary prevention work requires that we work beyond the individual and relationship levels in order to create deep and meaningful change in our communities. The prospect of evaluating beyond the individual and relationship levels of our work can seem daunting. However, although the potential scope of data collection is larger for community-level initiatives, the evaluation principles and practices are the same.
A strong theory of change that details how your short-, mid-, and long-term outcomes relate to each other will help you identify this synergy and point to opportunities for shared or related indicators across levels of the social-ecological model (SEM). Without a solid theory about these connections – and programming to address that theory – your work might not yet be ready to be evaluated beyond the formative level (see, e.g., Juvenile Justice Evaluation Center, 2003).
Part of what makes this work tricky is that research has not caught up with the current demands of practice to change risk factors across the ecology to meet our ultimate goal of preventing violence (Centers for Disease Control and Prevention [CDC], 2017; Degue et al., 2014). That is, while we have good theories to guide us, we do not have a large enough evidence-base from the literature to tell us that if we want to create X societal change, then we need to do Y (or, more likely, that we need to do A+B+C+Y). Thus, through evaluating our efforts, we are in a position to help build evidence around what works, guided by the current state of theoretical knowledge about community change and evaluation. Go us!
In order to demystify evaluating community- and societal-level changes, it is important to remember a few issues about the socio-ecological model. First, the ecological model does not exist in the real world. That is to say, the domains of our lives interact dynamically all the time and are difficult to separate into distinct and discrete domains at any given point in time. Relatedly, the cumulative synergy of programming (at various levels) is what works to achieve outcomes. This is the point of comprehensive initiatives. These outcomes should not necessarily be driven/associated with a given level of the SEM, especially as we move toward mid-term and longer-term outcomes.
For an example of related indicators, consider this idea, provided in the section on observational data collection:
Imagine that you are implementing programming within a middle school to increase bystander interventions. You work within individual classrooms to increase students’ motivation to intervene and give them the skills to do so. You also work with teachers to help them understand the importance of these efforts and give them bystander skills of their own. Additionally, you work with school administrators to develop policies and procedures that support bystander intervention. You’re hoping that these efforts will change the culture of the school. In this instance, observational indicators of the change for students could involve observing whether or not someone intervenes when a situation warrants intervention. An indicator of whether or not the norms have changed in support of such behavior might include observing how other students, teachers, or administrators respond to the person who intervenes or to the situation after the student intervenes. You can see the connection here between individual, relationship, and community factors with the interactions between behaviors and responses.
Community and societal level changes are often longer term than are changes for individuals, so this means that we need to think about how to measure benchmarks en route to those longer-term changes.
The Washington Coalition of Sexual Assault Programs published a brainstorming template that might be helpful as you begin to identify benchmarks.
Community Readiness
One way to think about benchmarks in your community-level work is to look at shifts in community readiness to create change around sexual violence. The Tri-Ethnic Center for Prevention Research has a model of community change that addresses nine stages of readiness over 5 dimensions. Their guidebook (2014) provides extensive guidance about how to assess readiness. Shifts in readiness would represent shifts in a community’s sense of ownership over the issue of sexual violence prevention. This has been successfully applied in the sexual violence prevention field by the North Dakota Department of Health and The Improve Group. They have created a series of tools to walk you through how to use the community readiness assessment that include a community leader survey, a series of how-to webinars, a discussion guide, and example reports.
An additional consideration is measuring principles and values as an indicator of the way our work changes our communities. The Whole Measures model developed by the Center for Whole Communities uses a series of rubrics to evaluate their community-based efforts around ecological sustainability. Among the principles they measure are social justice and community building.
Evaluating Partnerships
Much of our community building work requires developing or expanding partnerships.This is complex and time consuming work. Evaluation can help us to evaluate the effectiveness and growth of these partnerships. If you are beginning new partnerships to prevent violence at the community level, take a look at the tools on identifying partners, building partnerships, and sustaining partnerships from Veto Violence.
A lot of measures have been developed to evaluate community coalitions and partnerships. Here we highlight several that are currently being used in our field. If you are looking to establish an initial sketch of the strengths of a partnership and build a plan for growth, consider Prevention Institute's Collaboration Assessment Tool. The Center for Advancement of Collaborative Strategies in Health has developed an evaluated Partnership Self Assessment Tool that measures partnership synergy, a key indicator of a successful collaborative. As one of the few validated tools for assessing partnerships, consider reading more about resources needed and best uses for this tool. Support and Training for the Evaluation of Programs (STEPs) developed a measurement tool menu for evaluating coalition building and community mobilization efforts that is a really great place to start if you are new to evaluating this type of work.
Evaluating Environmental Approaches
While the development and evaluation of environmental approaches to sexual violence prevention is in its infancy, here are some reports and resources that might be useful as you are developing your approach.
- Shifting Boundaries Curriculum Resources
- Program profile
- Shifting Boundaries: Final Report on an Experimental Evaluation of a Youth Dating Violence Prevention Program in New York City Middle Schools
- Assessing different levels and dosages of the Shifting Boundaries intervention to prevent youth dating violence in New York City middle schools: A randomized control trial
- Resources on Place Based Tools
- Prevent Connect Webinar: A Safe Place to Call Home: Transforming the Physical/Built Environment for Sexual and Domestic Violence Prevention recording and Summary
- Sidewalks to Sexual Violence Prevention: a Guide to Exploring Social Inclusion with Adults with Developmental & Intellectual Disabilities includes evaluation tools and guides for participatory social and environmental mapping approaches. Also see, Mapping Evaluation Podcast with Cierra Olivia Thomas-Williams of the Indiana Coalition Against Domestic Violence talks about incorporating evaluation approaches into sexual violence prevention work through PhotoVoice.
- Crime Prevention Through Environmental Design (CPTED) NSVRC podcast explores how a coalition in Kansas City, Kansas is designing environments with the goal of preventing sexual assault through a CPTED model and how evaluation has assisted with their program development. Check out their Community CPTED Toolkit for community members to do an assessment of their neighborhood.
Policy Evaluation
There are a number of policies with evidence of impact on gender inequality and related risk factors for sexual violence. Whether is be workplace policies addressing sexual and domestic violence, pay equity, or community alcohol density policies, comprehensive work to prevent sexual violence requires policy work. The CDC provides some general guidance on policy evaluation through a collection of briefs listed in the resource section below.
Resources
A Guide to Measuring Policy and Advocacy (PDF, 48 pages): This guide from the Annie E. Casey Foundation provides information about and methods for evaluating policy and advocacy efforts that seek to change social conditions. They offer several case illustrations to help keep the information grounded in real, community-based work.
A Handbook of Data Collection Tools: Companion to “A Guide to Measuring Advocacy and Policy” (PDF, 45 pages) This collection of tools helps operationalize how advocates can measure core outcome areas identified in “A Guide to Measuring Advocacy and Policy.” This resource provides 27 sample tools and methodologies.
Policy Evaluation Briefs (PDF, 8 separate briefs) NCIPC developed this series of briefs to increase the use of policy evaluation methods in the field of injury prevention and control.The briefs and related appendices are intended to provide an increased understanding of the concepts and methodologies of policy evaluation. They are not intended to provide a “how-to” but rather to provide a solid foundation for exploring the utility of policy evaluation as a methodology and an overview of the critical steps and considerations throughout the process. Each of the briefs focuses on one specific aspect of policy evaluation; however, reading them all will provide a comprehensive overview of policy evaluation concepts and methodology.
Keeping the Collaboration Healthy (PDF, 3 pages) This tool developed by the Substance Abuse and Mental Health Services Administration presents some considerations for evaluating collaboration, common functions to evaluate, and examples of instruments that measure these functions.
Community Readiness for Community Change (PDF, 71 pages) This guide from the Tri-Ethnic Center for Prevention Research provides a comprehensive overview of the issue of community readiness to create change and a detailed plan for how to assess readiness in your own community.
North Dakota Department of Health Community Readiness Assessment (Online Resources, links to PDF files and YouTube videos) In 2017 the North Dakota Department of Health (NDDoH) Division of Injury and Violence Prevention conducted a statewide community readiness assessment for sexual and intimate partner violence primary prevention work. This toolkit provide assess to their community assessment survey instruments, instructional webinars, discussion guides and example reports.
Evaluating Comprehensive Community Change (PDF, 37 pages): This report from Annie E. Casey Foundation focuses on struggles and solutions related to evaluating comprehensive community initiatives.
Evaluating Comprehensive Community Initiatives (Online Resource) This section of the Community Tool Box, an online community development resource from the University of Kansas, focuses on evaluating comprehensive community efforts in a participatory manner. It offers a robust discussion of the challenges of this work alongside a model and series of recommendations for how to do this work.
Whole Measures: Transforming Our Vision of Success (PDF, 68 pages) This guide from the Center for Whole Communities introduces the model of Whole Measures, a rubric-based assessment tool that examines ten groups of practice related to doing comprehensive community work. Although the guide is designed for organizations working on ecological issues, most of the domains are directly relevant to sexual violence.
References
Centers for Disease Control and Prevention. (2017). Sexual violence: Prevention strategies. Retrieved from https://www.cdc.gov/violenceprevention/sexualviolence/prevention.html
DeGue, S., Valle, L. A., Holt, M. K., Massetti, G. M., Matjasko, L., & Tharp A. T. (2104). A systematic review of primary prevention strategies for sexual violence perpetration. Aggression and Violent Behavior, 19, 346-362. doi:10.1016/j.avb.2014.05.004
Juvenile Justice Evaluation Center. (2003). Evaluability assessment: Examining the readiness of a program for evaluation. Retrieved from the Justice Research and Statistics Association: http://www.jrsa.org/pubs/juv-justice/evaluability-assessment.pdf
Tri-Ethnic Center for Prevention Research. (2014). Community readiness for community change (2nd ed.) Retrieved from http://triethniccenter.colostate.edu/docs/CR_Handbook_8-3-15.pdf
| Back | Index | Next |