Evaluation During a Crisis or Disaster | El Centro Nacional de Recursos sobre Violencia Sexual (NSVRC)
Get Help Escape
English Spanish

Evaluation During a Crisis or Disaster

Evaluation During a Crisis or Disaster

When disaster strikes, it tears the curtain away from the festering problems that we have beneath them.

- Barack Obama

So why are we even talking about evaluation during a crisis or disaster? Crisis, by definition, is an unpredictable period of immense struggle, threat, and difficulty. As we grapple with disasters, those of us who are working to implement and evaluate prevention programs are often left with hard questions and uncertainty. What we do know is that disasters disrupt the physical and social environments that shape individual and community health and well-being. Evaluation strategies can offer vital tools to find out what communities in crisis need, and can help us center social justice during disasters. Making quick adaptations to programs and evaluation plans in the context of rapidly changing norms and environments can pose significant challenges. This section of the Toolkit doesn’t provide all the answers, but does include insights from evaluators and some guideposts for how evaluation work during crisis situations can be a tool for prevention. 

As people who work to prevent sexual assault, abuse and harassment try to meet existing and emerging needs amidst crisis, evaluating newly introduced methods can be daunting. We are generally already overburdened with accommodating rapid changes in our work individually and organizationally, so streamlining evaluation often requires significant thought and consideration. This can take many shapes, including pausing evaluation efforts or shifting their focus altogether. 

Ethics and Trauma-Informed Evaluation

Sexual assault prevention evaluation puts the needs of community members first. This requires working with community members in order to identify needs and ensure that efforts are centered around their voices. This will help to make sure the evaluation work can be used to improve people’s lives and not cause more harm. So, what might this look like in the context of a disaster or crisis?  

When evaluating sexual violence prevention work, it is helpful to think about trauma-informed approaches to evaluation.  A first step is to consider the 6 Guiding Principles to a Trauma-Informed Approach (Center for Preparedness and Response, 2020).

6 Guiding Principles to a Trauma-Informed Approach

After grounding yourself in the principles of trauma-informed approaches, you can then explore specific examples of how communities have prioritized equity and community wellbeing in the wake of catastrophic events in this brief from Prevention Institute. Using a trauma-informed approach will help you consider your evaluation questions, use different evaluation approaches, and adapt your data collection methods. 

The United Nations Development Programme Independent Evaluation office has published a useful infographic to help organizations understand important parameters when evaluating their programs. The infographic outlines how to 1) Rethink evaluation plans and teams 2) Evaluate the impact of the crisis at hand 3) Collect data remotely 4) Engage stakeholders virtually 5) Share evaluations globally, and 6) Connect with evaluation networks. 

For an example of how Rape Prevention and Education (RPE) program evaluators worked with RPE programs in Michigan to share the story of how they pivoted their work during the COVID-19 Pandemic, read this report, Community Connections: Key in the COVID-19 Pandemic. 

Skills Evaluators Bring to Crisis Situations

The COVID-19 pandemic has shown the importance of evaluation work and highlights the skills that evaluators bring to the collective good. In their blog, “How can we use evaluation in this time of community crisis?,” the Emergence Collective outlines the ways in which nontraditional evaluation methods are often helpful in times of difficult decision making. They have compiled a list of questions for evaluators to consider. This resource also includes questions related to organizational decision-making and impact evaluation and a list of evaluation activities and their intended outcomes. The Emergence Collective pulls from Michael Quinn Patton’s book Developmental Evaluation as well as his blog Evaluation Implications of the Coronavirus Global Health Pandemic Emergency, in addition to Bridgespan’s collection of COVID-19 response resources. 

When looking specifically at the role of evaluators, Miranda Yates reflects on her experience in youth and family services to highlight lessons learned during the global pandemic in an AEA Blog. Yates (AEA365 Blog, 2020) offers a list of practical tools and ideas which have served her and her organization during times of crisis:

  • Listen to people on the front lines and be open to pivoting as needed.
  • Focus on identifying and responding to the pressing needs that are emerging.  
  • Anticipate needs. While a general offer of help is excellent, concrete proposed ideas of what you could do are even better. At this moment, many people are not in a space to connect the ways that data and evaluation might help with managing the immediate crisis and with laying more solid groundwork for what is to come.  
  • Offer up your project management, communication and facilitation skills for whatever is needed to help organize efforts. 
  • Respond to the immediate while laying groundwork for identifying and supporting needs in the long term.
  • Keep a systems’ lens.  
  • Tap into available expertise and draw upon your partnerships and connections. 
  • As much as possible, share what you are learning. 

The full blog includes examples for each suggestion, as well as outside links to relevant resources. 

Adaptation: Adjusting Evaluation Strategies and Evaluation Questions

Adaptation is common in the sexual violence prevention field, and we have tools that can help us think through how to evaluate those adaptations. Evaluation is always evolving, and this often happens at a faster pace during and after disasters. Although adaptation is key to understanding emerging needs, disasters don’t happen in a vacuum - evaluators might find themselves affected by the very same barriers faced by the people they work with. Strategies must often be conceptualized on the fly, under high stress conditions. In order to reimagine strategies, pausing some work may be required. It may be natural to assume that work can be picked up again once the crisis is over. However, as Marian Urquilla of the Center for Community Investment notes, we must be mindful that 1) it’s impossible to know how long a crisis will last, and 2) assuming things will return to normal “when all this is over” underestimates the larger lasting impacts of a crisis. (Urquilla, 2020) After a crisis, the world will be a different place than it was before.  It’s vital to be aware of what assumptions might be shaping your strategies. For example, Did your affordable housing strategy assume a hot market? Did your leadership development program assume intensive in-person sessions with lots of time for informal relationship building?

The Center for Community Investment provides a useful collection of triage tools to help assess shifting priorities, including an annotated tool, a sample tool, a blank tool template, and instructions for facilitating group use of the tool. These tools can help identify not only new strategies, but also the ways in which questions need to change. Using the example of the COVID-19 pandemic, the UNFPA provides an overview of adapting evaluation questions to help organize evaluation criteria. Their questions help to pinpoint the relevancy, effectiveness, efficiency, coherence/coordination, and sustainability of country level programs during a crisis, and can serve as an example for other organizations and evaluators. In that same vein, DEVTECH shared a simple crosswalk chart of typical evaluation strategies and added in a column with possible adaptations during COVID-19. 

Tools

Virtual Adaptation Guidance for Sexual Violence Prevention Curriculum and Interventions (PDF, 1 page) This chart provides and overview of many key sexual violence prevention curricula and interventions that have provided virtual adaptation guidance resources to their implementers. 

 

 

It’s OK to Move Things Online

Disasters not only change the content of the work, but also how you do it since disruptions in normal methods of connection can occur. Transitioning to more flexible methods of communication, research, and community building is key. Creating more innovative virtual approaches and digital services can be useful. 

We talked about focus groups and interviews in the Data Collection section of the Toolkit. Here, we dig a little deeper to talk about conducting these virtually. Online focus groups can be a way to invite a group of people who are located in different places to a shared common space and connect with preventionists and/or program participants.

Research shows no significant difference in the outcomes and quality of virtual focus groups versus in person meetings (Underhill & Olmsted, 2003), and even finds that online focus groups are the ideal method when the work addresses sensitive topics of a personal nature- like health, sexuality, crime, politics, etc., as participants are more inclined to answer honestly and feel more comfortable (Forrestal, D’Angelo, & Vogel, 2015).

While there can be some technology challenges (Kite & Phongsavan, 2017), virtual spaces can help minimize certain mobility barriers for participants who may otherwise not be able to participate in in-person meetings. Hosting a web meeting can also bring together geographically diverse groups and contribute to a sense of community despite distance. 

Dr. Natalie E. Cook presents this short (13 min) overview on "Conducting Virtual Focus Groups" for Shine Lab (February, 2021). Virtual focus groups can be a valuable qualitative research and evaluation strategy during the pandemic and beyond (reducing barriers related to transportation and reading/writing).

Remote spaces aren’t only excellent avenues for data collection, but also offer opportunities to deliver community support, training, and facilitate positive interactions. Elizabeth DiLuzio and  Laura Zatlin of Good Shepherd Services outline a number of great strategies for before, during, and after a virtual gathering. For tips and best practices for facilitating engaging online events watch this web conference recording from PreventConnect.

As we move more into online spaces with support and prevention in mind, it's important to consider ethics and safety in the virtual world. The Safety Net Project has created an excellent overview on how to help survivors navigate privacy and safety risks when choosing to participate in online groups. Since we know that there will always be survivors participating in prevention work and prevention programs, these suggestions are important to consider for any evaluation activities that are occurring online. Some key suggestions are:

Wondering about the pros and cons of various online platforms? The Safety Net Project also provides an in depth review of Zoom options and privacy considerations, as well as a Comparison Chart of common tools for digital services to help understand what is the best technology to meet your needs. This checklist can help decision makers learn how to adapt policies and practices, make key decisions and train staff. TechSafety.org has a number of other great resources about digital safety available in both English and Spanish, in their Digital Services Toolkit. 

Beyond privacy and safety considerations, switching to online spaces for evaluation activities requires thoughtful guidance and support as well- especially if it’s a transition from work that was exclusively done in-person before. Great crowdsourced documents have emerged to support folks making this change. Doing Fieldwork in a Pandemic (Lupton, 2021), offers tips and references from researchers and evaluators about various digital research methods.

Do you have a story to share about how you have adapted or developed evaluation strategies in the context of a disaster? Submit your brief case example to be considered for inclusion in the Evaluation Toolkit. 

light bulb

 

For more context and information on how the COVID-19 impacted sexual violence prevention programs and how they adapted, watch the recordings of the National Sexual Violence Resource Center and PreventConnect web conference series. Links to all the recordings are below.

References

Center for Preparedness and Response. (2020). Infographic: 6 guiding principles to a trauma-informed approach. Centers for Disease Control and Prevention. https://www.cdc.gov/cpr/infographics/6_principles_trauma_info.htm

DiLuzio, E. (Ed.). (2020, March 21). Reflecting on the role of evaluator during this global pandemic by Miranda Yates [Blog post]. AEA365. American Evaluation Association. https://www.semanticscholar.org/paper/Considerations-for-and-Lessons-Learned-from-Online%2C-Forrestal-D%E2%80%99Angelo/18be6b0bd0688d51ca9a52ea1969e81027500d3b 

Forrestal, S. G., D’Angelo, A. V., & Vogel, L. K. (2015). Considerations for and lessons learned from online, synchronous focus groups. Survey practice, 8(3), 2844–2752. https://www.semanticscholar.org/paper/Considerations-for-and-Lessons-Learned-from-Online%2C-Forrestal-D%E2%80%99Angelo/18be6b0bd0688d51ca9a52ea1969e81027500d3b 

Kite, J., & Phongsavan, P. (2017). Insights for conducting real-time focus groups online using a web conferencing service. F1000Research, 6, 122. https://doi.org/10.12688/f1000research.10427.1

Lupton, D. (Ed.). (2021, July) Doing fieldwork in a pandemic (Crowd-sourced document, rev. ed.) https://docs.google.com/document/d/1clGjGABB2h2qbduTgfqribHmog9B6P0NvMgVuiHZCl8/edit

National Network to End Domestic Violence, Safety Net Project. (2020) Online support groups for survivors [Webpage]. https://www.techsafety.org/online-groups 

Underhill, C., & Olmsted, M. G. (2003). An experimental comparison of computer-mediated and face-to-face focus groups. Social Science Computer Review, 21(4), 506–512. https://doi.org/10.1177/0894439303256541

Urquilla, M. (2020, March 31). Reimagining strategy in the context of the COVID-19 crisis: A triage tool [Blog post]. Center for Community Investment. https://centerforcommunityinvestment.org/blog/reimagining-strategy-context-covid-19-crisis-triage-tool

 

Back   Index   Next