0:00:01.3 Sally Laskey: Welcome to Resource On The Go. A podcast from the National Sexual Violence Resource Center on understanding, responding to, and preventing sexual abuse and assault. I'm Sally Laskey, NSVRC's Evaluation Coordinator. On today's episode, we share an excerpt from our Evaluation toolkit webinar, Applying Principles-Focused Evaluation in the Sexual Violence Prevention Context. You will hear Tatiana Masters, a research and evaluation consultant provide some basic information about what Principles-Focused Evaluation is. If you're interest is piqued, you can also access the full webinar recording and resources in our show notes for this episode.

[music]

0:01:02.9 Tatiana Masters: Okay. A little overview. We'll first introduce you to Principles-Focused Evaluation and talk a little about the principles that we evolved here in Washington State. I'll give you an overview of the process that we took both to identify and flush out those principles and also to create evaluation tools and plans that integrate them. Alright, Sally can I bring you in here for a minute and ask you, what do you know about Principles-Focused Evaluation? 

0:01:35.3 SL: Absolutely, Tatiana. So I actually first learned about Principles-Focused Evaluation at a workshop that you did at the National Sexual Assault Conference. And then I did a little bit more reading about it. And the big thing that was emerging for me, it was a way to align your evaluation approach with your core values and principles and that, it's often an approach that's used when you have a complex approach or complex intervention that you're looking to evaluate.

0:02:16.0 TM: That's great, that's such a beautiful contextualization for this next overview. So this is a pretty basic dictionary definition of what principles are. They're statements that guide you on how to think or behave, they're focused towards a result that you desire, and they're based, as Sally alluded to, on values and beliefs, and also on knowledge and really like wisdom, deep wisdom. So in Principles-Focused Evaluation, the kind of rubric or mnemonic that you can think about for what is a principle, is you want your principle to be G-U-I-D-E, which is guiding, useful, inspiring, developmental, and also, dear to my heart, evaluable. So when you're trying to figure out if something is a principle and kind of get that identified and written up so that you can share it and face evaluation on it, you can kind of test like, is this a concept of principle by... By saying like, "Is it guiding? Does it prescribe something to do? Is it directing me in a particular way? Is it worded actively that it's very important? And is it distinct? And you'll see where useful, inspiring, developmental and evaluable... Other flushing out of what makes a principle a principle.

0:03:42.8 TM: I wanna emphasize evaluable a little bit more. You can assess whether something is followed. You can think about like what behavior would you see or what changes in beliefs if the principle was being followed. You can document what happens when the principle is followed and then you can also kind of test and say, "Is this taking me in the direction I wanna go?" That's [0:04:06.6] ____ of evaluation, which I'll talk a little bit more about on the next slide. And it can also support your decision-making very implementation, like you often will find yourself in a branching point implementing a program and then a principle can kind of help you see which direction to go, the divine aspect of principles. Okay, the way PFE fits into other kinds of evaluation is really a bit of a bridge between implementation and outcome evaluation. So you might be measuring in step one, whether you're implementing the intended activities of your program with the intended audience. PFE in the middle, will ask you to look at whether you're behaving as a prevention practitioner in ways that are consistent with your values about prevention, and what you know about prevention. Are you behaving in a way that supports the outcomes you'd like to see and people who participate in that program? 

0:05:09.3 TM: And then finally, of course, outcome evaluation is whether people who participate, or communities who participate in your prevention program, are they changing their knowledge, their attitudes, their behaviors in the way that you had hoped? So PFE doesn't substitute for any elements of evaluation, it just gets in the mix with them and help them relate to it. So the Washington State RPE really needed methods to evaluate across a bunch of diverse programs, because like some other states, Washington decided that one-size-fits-all communities was not right for the state, and we're doing RPE here. So they funded seven different programs. Each of these served a different population, they delivered really quite different programming, but each of the programs had this very deep knowledge of the community that they worked in and the groups of people that they were working with, and they were very passionate about their work. Some of them had some overlap in that they delivered programming in schools, some were trying to change social norms, other ones were trying to build skills. So a lot of different stuff was happening, but they were all working towards the same ultimate goal, of course, of preventing sexual violence. Another reason PFE was a good fit for Washington, was that we had these committed prevention practitioners, and we also had people on the state level who were interested in innovative approaches to evaluation and in supporting PFE.

0:06:42.0 TM: So we thought that it was a great fit and we were drawn towards it for our evaluation because we thought it could really address some of the challenges of evaluating across all these different programs that yet we're working towards the same goal. We thought that it was compatible with some other components we had in our evaluation already, and as people... We at evaluation specialists have a lot of background in academic research on sexual violence prevention and other topics. We kind of dug how it could bring in what research says supports effective prevention in this realm, and combine that with practitioner wisdom and have those two powerful forces pulling in the same direction and not opposite from each other as sometimes sadly occurs. So we liked it for all these reasons. I'm gonna ask Sally again about relevance. Where have you seen PFE possibly being a relevant approach in communities or other places? 

0:07:41.5 SL: Yeah, I've been thinking a lot about this. There are three things, or three areas that immediately come up for me. The first is that more and more in the sexual violence prevention field, we are looking to connect the dots with other issues and looking at how we can collaborate with other movements and with other partners and in doing so, finding those key core values that we can connect around, I think is really critical as we might be looking into new partnerships. I also see this extremely relevant to the work that we're doing to center our anti-oppression efforts within our sexual violence prevention work. And then of course, what is on my mind the last year is the need for us to adapt. And I think using this approach in really focusing in on the core principles and values can help when programs need to make adaptations like we all did because of COVID-19, I'm just thinking for programs that had this in place, it gives you some good guide posts when you're having to make hard decisions about things that you wanna make sure continue to happen and things that may need to be adapted due to a variety of different circumstances.

0:09:36.5 TM: Yeah, I hadn't thought about that adaptation during times of COVID aspect, but that really sounds true to me, and it sounds right on. So let me just foreshadow the steps for you in doing Principles-Focused Evaluation. First you're gonna identify principles that are generally shared across different sexual violence prevention organizations and programs. Second, you'll define the practitioner behaviors that would demonstrate these principles are live and guiding your work. Third, you would identify the ways program participants might respond if the practitioners, the preventionists were adhering to the principles. And then fourth, you would integrate questions about the principles into your existing evaluation approaches.

0:10:27.6 TM: So I will go through each of those in turn. No worries. So first, we'll look at the steps in identifying and developing those principles. So, you remember the practice-based knowledge, what the practitioners of a program now and do in their wisdom is what informs principles. So the process I'm gonna describe is all geared towards capturing that practice-based knowledge and getting it into the GUIDE language, G-U-I-D-E, guiding, useful, inspirational, developmental and evaluable. Alrighty. So when you're identifying principles, the first thing you wanna do is some in-person group work, you get your program staff together and you do these things which I'll talk about. You write success stories to share with the group, you look at those success stories and identify what they have in common, you draft up your principles according to the GUIDE criteria, get them into a kind of language that will make them maximally helpful to people down the line. And to do all this, you commit to multiple round through sharing, revising, feedback and collaboration. We'll look a little more under the foot here.

0:11:41.6 TM: So before you get your group together, have the program representatives prepare by writing a brief prevention success story to bring and share. What we ask people to do, was write up something just like a page, maybe page and a half, that would share a time when their event or their class or their programming went really beautifully and it made a big difference, and tell that story with so much information and details so that people can really visualize and experience what the prevention person did. The next step is you get together and share these stories and compare them. The way we did it, was that we had two to four people or organizations per group, everybody had copies of the stories, swapped them and read them. They worked together to identify things in common across the stories, and then they named those themes, those commonalities, and that became the principles.

0:12:45.8 TM: I do wanna note that when I discuss group processes and numbers of people here and also timelines, this is based on our experiences, kind of like a case study in Washington. I think there's definitely room to adapt principles, identification work to smaller and larger groups. Like too many people, organizations wouldn't be feasible. But there's nothing magical about the number of people or works that we had doing here.

0:13:12.0 TM: So if you are an evaluator who's working with sexual violence prevention practitioners to identify these principles, your goal is really to help them start with examples of what their diverse work has in common and move to this briefly stated principle. People can really do this if you give them permission and reinforcement. Another thing you can do when you're refining the principles, and this happens for the most part, kind of back stage, so to speak, is you can bring in knowledge from the research base about less supports effective prevention. And this can be effective prevention, just kind of generally in any area, or it could be sexual violence specific, if that's your area.

0:13:54.8 TM: You can... If you are the content expert, you can do it. If you bring in an outside content expert that has knowledge in this field, they can look for things that might be kind of missing in the principles that have been identified, and then they can take that back to the practitioner group and bet that with them, like, "Do you think it would be right to include this concept?" So the way that you then start sharing, merging and refining the draft principles, is when you do this in person. We used a world cafe process where we used those giant post-its... So you stick up on the wall. People would put up their draft principle their group came up with, and then tour around and write input for other groups on their principles. We were there, myself and some other facilitators to help people make this happen in the moment. And we ended up with seven principles out of this process. The target from the patent book is four to eight. I feel like seven might be kind of a lot, we're sort of wishing we had stuck with maybe a few fewer, but four to eight seems about right.

0:15:00.2 TM: So what you wanna end up with are questions. So you follow, for evaluators, are pretty standard measurement development approach. First, you figure out what you expect to see if this thing is working. What will practitioners be doing if they're really adhering to this principle? And if practitioners are following the principle, what are you thinking you will see in program participants? Will they engage differently, have different behaviors, or what? Because of just time limitations, I'm gonna focus on the process we used regarding practitioner behavior, but it's the same for assessing impact on program participants. I'll walk you through what we did. It's not the one perfect way, but it worked pretty well for us, so it's just an example. We did another round of in-person group work with the same program staff. We gave each small group two of our seven principles and had them brainstorm what they would think they'd see from practitioners, when they were really upholding the principle. We wanted them to be as specific and behavioral as they could, what would the person be doing when they were facilitating our program that would show the principle was active.

0:16:18.9 TM: And we made some tailored worksheets for them to collect their ideas on, because we had noticed in the first round of principles identification, that having that kind of container, that structure for the process, helped people know what they were "supposed to do and do it" and that they just felt more comfortable with more structure and guidance in that way. So we provided that using this tailored worksheet as well as facilitation on-site. We had people then do another world cafe. Once they had worked in their small groups, they used those giant post-its and world cafe process to show each other what behaviors they thought went with each principle. And then they went around and added...

0:17:06.3 TM: To give an example, so to illustrate, one of our Washington State principles was shared power. Engage with others in ways that encourage equal participation. And the people who worked with us told us that what you want... Would be seeing when shared power was in action, they said preventionists would work to engage all participants and audiences equally. They are welcoming to everyone in their attitudes and actions, and they act as a facilitator rather than as a top-down expert. And then moving into what you would see in participants, if practitioners were exemplifying shared power, as a result, participants and audiences believe their opinions and experiences are of equal value to those of other people. They also respect the opinions and experiences. So, why? Why are we measuring all these things? This is always a very important question to address. PFE, Principles-Focused Evaluation can really, I think, help reveal the how of preventions.

0:18:13.3 TM: So when your prevention participants achieve those behavioral or those attitudinal outcomes that you had hoped for, PFE can help you assess how and why that's happening. Or conversely, if the outcomes don't look like what you had hoped, PFE can help you understand where things might be going awry. And it can also identify the conditions where participants will most benefit from your program and are most likely to show the kind of movement that you want on the program outcomes. So Principles-Focused Evaluation, you create a list of principles that try to reflect the essence of your work, and that balance the need to articulate a clear goal where you're headed with prevention work while also allowing for adaptation. You identify what you wanna see if the principles are alive and in action, and then you do data collection to see if what you would expect is happening. That's the evaluation part.

0:19:17.8 TM: I think that one of the real powerful things about Principles-Focused Evaluation is that you get this great program tool, and also you get a great evaluation measure. The grantees we worked with told us they really enjoyed the process of identifying principles itself, the sharing, the success stories, they're working to see what all their different efforts really had in common at their core. And one person said it was so poetic. She said, it was like returning to drink from a really deep well, like the well of what was conforming all their work. So this was something that we did not anticipate, but of course, we're very glad to have shared with us or other persons.

0:20:02.4 SL: Thank you for listening to this episode of Resource On The Go. For more resources and information about preventing sexual assault, visit our website at www.nsvrc.org. To learn more about Principles-Focused Evaluation and NSVRC's evaluation toolkit, visit the episode resources at nsvrc.org/podcasts. You can also get in touch with us by emailing resources@nsvrc.org.

[music]