It is easy to get caught up in the idea of using measures that have been validated through research studies (e.g., the Illinois Rape Myth Acceptance Scale) as your tools of choice. While there are definite benefits to this (see below), there are also significant drawbacks. As with all things in evaluation, using existing measures should be done very deliberately and should not be considered the default option when planning your evaluation.
- Existing measures have often undergone rigorous testing to ensure that they are actually measuring what they claim to be measuring (validity) and that they can do so repeatedly (reliability). As a result, using them strengthens your case for saying you are measuring the change you seek to measure with them.
- The hard work of drafting, re-drafting, revising, and re-revising questions is already done for you! Writing good questions is an art and a science and is far more difficult than most people imagine. Previously validated measures that have been developed by experts are likely not to fall prey to common pitfalls in survey development and also save you a lot of time in development.
- The use of existing measures can sometimes increase the buy-in from other partners, especially funders.
Drawbacks and Cautions
- Not all measures that are widely used or found in the research are actually valid and reliable.
- Even if they are valid and reliable, they may not be appropriate for the community or population with whom you are working. For example, many instruments are validated through research conducted with college populations. In contrast, many (if not most) prevention workers are working with younger populations. More than just being at a different educational level, the developmental level also influences how concepts and constructs are understood. The measures we use need to be appropriate to our given populations.
- In addition to being appropriate to our given population, our measures need to be specific to the change we are interested in creating (i.e., our outcomes) and related to the intervention(s) we are employing to make that change. Sometimes existing measures are chosen for the above benefits without sufficient consideration to their direct applicability to the work being measured; if they don’t speak to your outcomes and programming, then the benefits are irrelevant because the data won’t be useful to you.
- Measures can become outdated, especially measures that seek to assess particular social constructs like gender-role socialization. These constructs change over time, as do the indicators of them. For example, the Attitudes Toward Women Scale (Spence, Helmreich, & Stapp, 1973) includes the following item: “It is ridiculous for a woman to run a locomotive and for a man to darn socks.” The short version of this measure is from 1978. Nearly 40 years later, most Americans probably assume that it’s ridiculous for anyone to darn socks, but that’s not necessarily because there’s been a drastic shift in our cultural perceptions about gender.
- Starting with validated measures and editing the questions to fit your audience or issue is not necessarily a bad idea and might very well be better than starting from scratch. However, it is important to remember that changing anything about the instrument means it is no longer valid and reliable and is then ultimately an untested instrument
Want more guidance on selecting, adapting, and evaluating prevention approaches? Check out this guide from the Division of Violence Prevention. This guide support good decision making that balances delivering prevention approaches as intended with considering unique community contexts.