What does it take to do good evaluation?
When you read the question above, do you immediately think about data analysis skills? Or survey design skills? While an evaluator needs skills in data collection and analysis, these are only two of the many areas of knowledge and skill that help an evaluator succeed. An organization’s capacity to conduct evaluation stretches well beyond any one individual’s knowledge-level or skill sets.
Organizational Level
Recent research on evaluation capacity building highlights the importance of organizational factors in building and maintaining evaluation capacity (Taylor-Ritzler, Suarez-Balcazar, Garcia-Iriarte, Henry, & Balcazar, 2013).
“Even when individual staff members have the knowledge and motivation to engage in evaluation activities such as mainstreaming and use, these activities are less likely to occur if their organization does not provide the leadership, support, resources, and necessary learning climate” (p. 200).
Capacity building efforts often focus on building the skills of a few particular staff members. Agencies may also create a dedicated evaluation position as an attempt to increase both capacity to do evaluation and the actual practice of evaluation. Sometimes these efforts are not successful because other critical steps have not been taken to make sure the culture of the organization is one that supports and encourages each aspect of evaluation. For example, if the organization is resistant to change, will data that supports change be ignored?
Members of organizational leadership may not need robust evaluation skills themselves, but leaders need to be evaluation champions to help integrate an evaluative mindset into the organization. Moreover, they need to know enough to understand the importance of providing adequate resources for evaluation efforts. Learn from preventionists around the country in this bulletin about ways that organizations can nurture a culture of evaluation.
Practitioner Level
“Like medicine, evaluation is an art and a craft as well as a science. Becoming a good evaluator involves developing the pattern-spotting skills of a methodical and insightful detective, the critical thinking instincts of a top-notch political reporter, and the bedside manner and holistic perspective of an excellent doctor, among many other skills” (Davidson, 2005, p. 28).
What does it take to be a good evaluator? It depends on a lot of factors, but most importantly it depends on what type of evaluation will be conducted. Certain evaluation methods involve higher levels of statistical analysis than others, and certain approaches to evaluation require more process facilitation than others. Evaluators often specialize in particular methods and approaches and work collaboratively with evaluators with different skill-sets when needed. The trick is to understand the limits of your own scope of skills and knowledge, and to know when to bring in help. (In fact, this is one of the American Evaluation Association’s guiding principles.)
Generally, many skills that evaluators often need overlap nicely with the skills preventionists use in their daily work (The Canadian Evaluation Society, 2010). See the following list for a few examples:
- Meeting and group facilitation
- Conflict resolution
- Self-reflexivity
- Critical thinking
- Pattern identification
- Interpersonal communication
Preventionists who have already developed many of the skills above can supplement them with formal or on-the-job training in evaluation-specific skills like data collection and analysis and thereby possess a strong evaluation skill set.
Building Capacity
Many people and organizations try to approach evaluation capacity building as a one-time activity. Since evaluation is a broad field-of-practice and doing good evaluation requires particular individual skills and motivation in addition to organizational buy-in (Taylor-Ritzler, Suarez-Balcazar, Garcia-Iriarte, Henry, & Balcazar, 2013), evaluation capacity building is much more likely to be effective when engaged in as an ongoing process.
When embarking on a process of capacity building, consider having your organization complete an evaluation capacity assessment to identify the most critical evaluation-related needs and developing a plan for how to meet them. Some resources and ideas for doing so are outlined below.
Low or No Cost
A variety of high quality resources are available to meet skill and knowledge-building needs, including many free or low cost online resources. This toolkit provides links to many of them that might be especially relevant to your primary prevention efforts. You can also visit our collection of Self-Study Guides for inspiration on how to build knowledge related to specific sub-areas of evaluation. Websites such as Coursera and Edx provide access to free online courses from universities around the world. These websites often have courses related to general data analysis as well as courses on specific tools like social network analysis.
Investments
Hiring an evaluator to provide capacity-building services is a great investment if it is one you can make. If you are unable to hire an evaluator, you might be able to find university professors or graduate students who will provide pro-bono or reduced-rate capacity building services. Working with an outside evaluator, as opposed to an evaluator you have on staff within your organization, might be the best way to address organization-level issues related to evaluation capacity. Remember that not all researchers or evaluators are skilled capacity-builders, so you’ll want to make sure that you seek out a person who identifies this as one of their areas of expertise.
Consider joining the American Evaluation Association (AEA) to take advantage of ongoing learning opportunities, including Coffee Break webinars, eStudy courses, evaluation journals, and more.
Several evaluation conferences are held throughout the United States each year, many of which include learning opportunities that are tailored for everyone from beginners to experts. Check out the following options:
- Evaluators’ Institute at Claremont Graduate University
- AEA Annual Conference or Summer Evaluation Institute
You might also have access to similar events through regional evaluation groups.
General Resources
Cultivating Evaluation Capacity: A Guide for Programs Addressing Sexual and Domestic Violence (PDF, 58 pages): This guide from Vera Institute of Justice explores evaluation capacity building specifically for antiviolence organizations. It includes an evaluation capacity assessment tool, tips for building evaluation capacity, and a variety of useful resources.
Competencies for Canadian Evaluation Practice (PDF, 15 pages) This document from the Canadian Evaluation Society discusses competencies for evaluators along 5 different practice domains – Reflective, Technical, Situational, Management, and Interpersonal.
Evaluation Capacity Assessments
Evaluation Capacity Assessment Tool (PDF, 7 pages) This assessment, developed by Informing Change, examines aspects of both organizational- and staff-level evaluation capacity and can be self-administered.
Capacity and Organizational Readiness for Evaluation (CORE) Tool (PDF, 1 page) This short tool developed by the Innovation Network specifically focuses on organizational issues related to evaluation capacity.
Primary Prevention Capacity Assessment Tool (PPCA) (PDF, 10 pages) This tool, developed by the Ohio Rape Prevention and Education Team was developed to identify and prioritize training and technical assistance needed to build prevention and evaluation capacity within the RPE program.
Videos
Watch the NSVRC Mapping Evaluation Podcast Series.
Activities
The Training and Capacity Building slide deck and activity guide have several training and meeting activities focused on building capacity around a variety of evaluation skills.
Practice Tip
Even small changes can shift an organizational culture toward increased support for evaluation. For example, members of organizational leadership can look for opportunities to transparently and consistently collect, analyze, and respond to data in their day-to-day roles. This could include efforts as informal as taking real-time polls in staff meetings about relatively low stakes issues to more formal and involved processes.
The important piece of this is that staff members see leadership taking data seriously by
- discussing points of learning and actually making changes as a result of what was learned or
- encouraging staff members at all levels to incorporate evaluation into their work in formal and informal ways.
Staff members who are not part of formal leadership can also model evaluation driven decision making and highlight the ways that evaluative processes improve their work.
References
The Canadian Evaluation Society. (2010). Competencies for Canadian evaluation practice. Retrieved from https://evaluationcanada.ca/txt/2_competencies_cdn_evaluation_practice.pdf
Davidson, E. J. (2005). Evaluation methodology basics: The nuts and bolts of sound evaluation. Thousand Oaks, CA: Sage.
Taylor-Ritzler, T., Suarez-Balcazar, Y., Garcia-Iriarte, E., Henry, D. B., & Balcazar, F. B. (2013). Understanding and measuring evaluation capacity: A model and instrument validation study. American Journal of Evaluation, 34, 190-206. doi:10.1177/1098214012471421
| Back | Index | Next |