Self-Study Plan: Qualitative Evaluation (Intermediate) If you have some evaluation experience and a strong knowledge base in the foundational concepts and methods and are ready to increase your knowledge and skills around qualitative methods (like focus groups, interviews, and other non-numerical data collection and evaluation), follow the plan below. All of the resources listed in this self-study plan are free unless otherwise noted. Objectives of this Self-Study Plan: After you complete this exploration, you will be able to: • List benefits and drawbacks of collecting qualitative data for program evaluation • Describe several methods for collecting observational data • Develop basic tools for collecting and analyzing qualitative data • Apply a basic process of qualitative data analysis to identify themes in data • Identify additional areas of learning that will improve your ability to implement qualitative evaluation methods or use qualitative data STEP ONE: Orient yourself to the basics of qualitative evaluation. Time: 10 minutes STEP TWO: Learn the basics about focus groups and interviews. Title: Focus Groups and Interviews Time: 15 - 30 minutes Notes: This guide (Curtis, 2014) from the Texas Association Against Sexual Assault focuses on using focus groups and interviews for sexual violence prevention work. Although there’s a particular focus on using these methods for needs and resources assessments, the same methods are applicable to other types of evaluative efforts. STEP THREE: Learn the basics about observations. Title: Data Collection Methods for Program Evaluation: Observation Time: 15 - 30 minutes Notes: This brief from the CDC (2008) outlines the basics of observational methods for program evaluation. STEP FOUR: Learn about integrated observational methods in this online course from PreventConnect. Title: Integrated Evaluation for Program Improvement Time: 1.5 hours Notes: This course (Curtis, n.d.) focuses on integrating evaluation into your prevention efforts and is specifically geared toward sexual violence preventionists looking for improvement-oriented methodologies. The course includes both a video presentation and assignments to help you apply your learning. STEP FIVE: Learn about qualitative data analysis processes by watching this short how-to video (The Evaluation Center, 2011) in which Stephanie Evergreen teaches you the basics of qualitative analysis. Time: 30 minutes Notes: This section of a longer webinar about working with data focuses on one relatively simple approach to qualitative analysis. (The video will start just before the 30-minute mark, and the section on qualitative data analysis will finish just before the 46-minute mark. You can stop when the section on website data begins unless you want to learn about that, too!) To support your learning, follow this up by reading Listening to Our Communities: A Guide on Data Analysis, which provides a useful and accessible overview of one process for analyzing qualitative data (National Sexual Assault Coalition Resource Sharing Project [RSP], and National Sexual Violence Resource Center [NSVRC], 2014). STEP SIX: Read up on Most Significant Change and/or watch a video about it. Title: The ‘Most Significant Change’ (MSC) Technique: A Guide to Its Use (Davies & Dart, 2005) (Guidebook) and Most Significant Change (Jones, 2015) (Video) Time: One - two hours Notes: Most Significant Change is a participatory methodology that engages program participants in telling stories about the impact of an intervention. Stakeholders then select stories that seem like the “most significant change.” This process is especially useful for interventions that have multiple sites coordinated by a central agency but can be modified for smaller-scale efforts, too. STEP SEVEN: Learn about triangulation in qualitative methods. Title: What is Triangulation (Qualitative research)? Time: Five minutes Notes: This short animated video (B2B Whiteboard, 2013) covers the basics of an often-overlooked aspect of evaluation: triangulation. The video covers the primary methods of triangulation for qualitative research as well as pros and cons. STEP EIGHT: Watch the following (short) Coffee Break webinars from AEA to determine which areas of qualitative evaluation you want to learn more about next. (Coffee Break webinars are short introductions to or overviews of specific concepts, techniques, tools, etc. They provide a great opportunity to see if you need or want to learn more about the topic and usually provide resources for doing so.) Title: Understand Causes of Outcomes and Impact Time: 20 minutes Notes: In this short video, Jane Davidson (2013) gives basic information about causal attribution in evaluation and methods for assessing causality when using evaluation methods that do not include comparison or control groups. Optional (requires American Evaluation Association membership) Time: 20 minutes per module • Participatory Qualitative Analysis (Coffee Break Webinar) • Qualitative Data: Software Solution for the Evaluation Process (NViVo) • Visualizing Qualitative Data STEP NINE: Reflect on and integrate your learning. Title: Self-Study Plan Worksheet: Qualitative Evaluation Time: 30 minutes Notes: Complete this worksheet to begin thinking about how what you have learned about evaluation applies to your work. If you need help thinking through your answers or want to discuss your next steps, send an email to prevention@nsvrc.org. Handy Online Resources Activity-Based Assessment Toolkit: This toolkit, developed by the Texas Association Against Sexual Assault and the Texas Council on Family Violence (Curtis & Kukké, 2014), walks you through the process of developing micro-evaluation tools.These tools help you refine and improve your education-based efforts by evaluating participants as they apply the skills they have learned. Analyzing Qualitative Data: This guide from the University of Wisconsin-Exchange (Taylor-Powell & Renner, 2003) provides an overview of analyzing qualitative data that is brief, practical, and full of examples. Books and Print Resources Qualitative Research & Evaluation Methods: Integrating Theory and Practice by Michael Quinn Patton (2014). In-Person Training Opportunities The following conferences/institutes usually offer extensive training in qualitative methods, among many other important evaluation topics and skills. Evaluator’s Institute: Claremont Graduate University hosts one or two institutes per year to equip evaluators with cutting-edge information and foundational skills alike. Some of the greatest minds in evaluation teach full-day or multi-day workshops here, so it is a perfect opportunity to dig deep. AEA Annual Conference and Professional Development Workshops: The American Evaluation Association hosts a seven-day conference every year that begins and ends with in-depth professional development workshops ranging from a half day to two full days each. Topics and trainers often overlap with those at the Evaluator’s Institute named above, but the cost of attending the professional development workshops at AEA is substantially less. AEA Summer Evaluation Institute: In addition to their large annual conference, the American Evaluation Association hosts a summer institute that also includes both a conference format and pre- conference workshops. The entire event spans four days, including the full day of workshops prior to the conference. Since this institute is smaller than the annual conference, it also does not cover as many topics in as much depth. References B2B Whiteboard. (2013, November 11). What is triangulation (Qualitative research)? [Video file]. Retrieved from https://www.youtube.com/watch?v=DPOhrdBGsLc&feature=youtu.be Centers for Disease Control and Prevention. (2008). Data collection methods for program evaluation: Observation. Retrieved from https://www.cdc.gov/healthyyouth/evaluation/pdf/brief16.pdf Curtis, M. (n.d.). Integrated evaluation for program improvement [Online course]. Retrieved from PreventConnect: http://learn.preventconnect.org/course/view.php?id=44 Curtis, M. (2014). Focus groups and interviews. Retrieved from the Texas Association Against Sexual Assault: http://taasa.org/wp-content/uploads/2014/10/FocusGroupAndInterviewGuide.pdf Curtis, M. J., & Kukké, S. (2014). Activity-based assessments: Integrating evaluation into prevention curricula. Retrieved from the Texas Association Against Sexual Assault: http://www.taasa.org/wp-content/uploads/2014/09/Activity-Based-Assessment-Toolkit-Final.pdf Davidson, E. J. (2013, May 21). Understand causes of outcomes and impacts (CBD 141) [Webinar]. Retrieved from American Evaluation Association: http://comm.eval.org/communities/ community-home/librarydocuments/viewdocument?DocumentKey=a2b20160-c052-499d-bdb5-0ae578477d2a Davies, R., & Dart, J. (2005). The ‘Most Significant Change’ (MSC) technique: A guide to its use. Retrieved from Monitoring and Evaluation NEWS: http://www.mande.co.uk/wp-content/uploads/2005/MSCGuide.pdf The Evaluation Center. (2011, January 19). Making sense of your evaluation data [Video file]. Retrieved from https://vimeo.com/19126682#t=1786s Jones, L. (2015, March 25). Most significant change [Video file]. Retrieved from Engineers With Borders USA: https://www.youtube.com/watch?v=d6nbZN8DTyU The National Sexual Assault Coalition Resource Sharing Project, and National Sexual Violence Resource Center. (2014). Listening to our communities: Guide on data analysis. Retrieved from http://www.nsvrc.org/sites/default/files/publications_nsvrc_guides_listening-to-our-communities_guide-for-data-analysis.pdf Taylor-Powell, E., & Renner, M. (2003). Analyzing qualitative data. Retrieved from the University of Wisconsin-Extension: http://learningstore.uwex.edu/assets/pdfs/g3658-12.pdf © 2018 National Sexual Violence Resource Center. www.nsvrc.org | prevention@nsvrc.org | (877) 739-3895