Case Study Culturally Relevant Evaluation of Prevention Efforts © National Sexual Violence Resource Center 2016. All rights reserved. This document was supported by Cooperative Agreement #5UF2CE002359-03 from the Centers for Disease Control and Prevention (CDC). Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the CDC. The content of this publication may be reprinted with the following acknowledgement: This material was reprinted, with permission, from the National Sexual Violence Resource Center’s publication titled “Case Study: Culturally Relevant Evaluation of Prevention Efforts.” This guide is available by visiting www.nsvrc.org. Case Study Culturally Relevant Evaluation of Prevention Efforts Part I. Walking in Balance with All Our Relations: A Violence Prevention Curriculum for Indigenous People by Strong Oak Lefebvre, LICSW Visioning B.E.A.R. (Balance, Equality, And Respect) Circle Intertribal Coalition (VBCIC) developed a culturally specific violence prevention curriculum for indigenous people entitled “Walking in Balance with All Our Relations” based entirely upon indigenous wisdom and values before colonization (Visioning B.E.A.R. Circle Intertribal Coalition, in press). About our Curriculum Changes that VBCIC would like to accomplish through the use of this curriculum include a community-wide understanding among indigenous people that women and the land are one and the same and that Mother Earth has the same rights as human beings. The curriculum teaches that every member of the community is responsible for every other member’s well-being. If all things are equal and deserving of respect, then no one has a right to objectify the earth, any human being, animal, or plant. Violence against others in an interdependent relational world is unthinkable, and the community needs to be restored to wholeness whenever infractions occur. The desired changes include a deep understanding that each person has a responsibility to act on behalf of the whole, on behalf of the circle of life. Accordingly, all teachings are done through Circle Process using the Talking Stick. The Circle Process is a non-hierarchical construction that ensures that all are equal to one another and that no voice is more important than another’s. Most importantly, it is the social structure American Indians used before colonization to make all decisions to resolve all conflicts. It allows each person in the Circle to present his or her point of view. [Callout box: Indigenous people: People who are descendants from the first group of inhabitants of a country or geographical region before people from different cultures or ethnic groups arrived. The new groups later became the dominant culture through conquest, occupation, and settlement. Many indigenous people continue to practice their unique traditions and resist the influence of the dominant culture (United Nations, n.d.).] [Callout box: Colonization: “…Colonialism is a form of domination – the control by individuals or groups over the territory and/or behavior of other individuals or groups.” Colonialism is a form of exploitation, forced culture change, and power over native people (Horvath, 1972, p. 46).] Each Circle has a Circle Keeper. The position of Circle Keeper is rotated among other members in the Circle and is voluntary. The Circle Keeper’s role is to facilitate the circle, summarize, and move the conversation forward; the Circle Keeper does not control the group and is on equal footing with the participants in the Circle. The Circle Keeper is also responsible for making the Talking Stick. Each part of the Talking Stick is a reflection of the Circle Keeper’s personal medicine. Because of this, each Talking Stick is different. The Talking Stick is passed around the Circle clockwise. Only the person holding the stick may speak; no one else is allowed to interrupt or talk while one person is sharing. Individuals may pass. The Talking Stick continues being passed until all have spoken. Every person present must listen intently to the person speaking. This means Circle participants will not be thinking about what they intend to say, but rather be transformed by what they are hearing. Active listening in this way allows for changes of heart and thinking. A feather is part of the Talking Stick to reflect that the individual contributions of the whole will allow the Circle to see the big picture with the eyes of an eagle from above the clouds. Each individual would identify the details that need attention. VBCIC wants to make the incidence of sexual assault and all forms of violence unthinkable through a culture shift that incorporates indigenous traditions and values prior to colonization. The “Walking in Balance with All Our Relations” prevention curriculum features twelve modules based on indigenous values and traditions. It is designed to be integrated into the very fabric of indigenous life across the lifespan. The 12 modules are: • true democracy, • compassion, • respect, • generosity, • courage, • wisdom, • sacredness, • humility, • empathy, • balance, • gratitude, and • connection to the land. It is our intention that this curriculum will proliferate throughout the Northeast and eventually to national prominence. We offer it as an effective means of eliminating childhood sexual assault and all forms of violence, not only in indigenous communities, but in multicultural communities as well. VBCIC anticipates that the curriculum will be translated into indigenous languages. [Image: Circular illustration that shows Circle Process.] Historical Context The 12-module curriculum provides historical perspective and promotes fundamental structural change as necessary for peace, justice, and harmonious relationships to be the norm. We believe adults and communities are responsible for the well-being of children. VBCIC intends to raise awareness among elders, adults, and young people about historical trauma and how it can be healed and prevented in the future by a fundamental change. This change is based on the law of the Circle and affirms that all beings are equal to each other and that Mother Earth has the same rights as a person. VBCIC seeks to raise awareness that racism and oppression stem from relationships based on power and privilege that first began when European occupiers came to the Americas with a mandate from the pope to take away lands from any people who were not Christian. The Doctrine of Discovery is a principle of international law dating back to the late 15th century rooted in a decree issued by Pope Nicholas V in 1492. This decree specifically sanctioned and encouraged the conquest, colonization, and exploitation of non-Christian territories and peoples. The Doctrine of Discovery has not yet been rescinded, and is in fact a part of U.S. law. The newly formed United States took over the mandates of the Doctrine of Discovery from the English and adopted it into law. Further, once the new government formed, it decreed that only White, male Christians were allowed to own land (Onondaga Nation, n.d.). It then used Christian institutions to round up all indigenous children into government-owned, Christian-run residential boarding schools (Smith, n.d). VBCIC believes that an accurate historical understanding of what happened to indigenous peoples is critical to effectively addressing child abuse prevention. We used the Socio-Ecological Model in the design of this curriculum (Centers for Disease Control and Prevention [CDC], 2015). This model uses a four-level approach to violence prevention that considers the complex interplay between individual, relationship, community, and societal factors. [Image: Socio-ecological model shows four concentric circles. The smallest is labeled Individual, the next largest is Relationship, the next largest is Community, and the largest is Societal.] History of the Project The Visioning B.E.A.R. Circle Intertribal Coalition, Inc. received its 501 (c) (3) status on February 10, 2015. This represents a huge achievement for a community group that began with my work as a Prevention Coordinator through the Massachusetts Department of Public Health funding for Rape Prevention Education. The community became empowered to create its own Visioning B.E.A.R. Council to determine its own needs in preventing sexual violence. VBCIC continues to strive to prevent all forms of gender-based violence but also incorporates economic, environmental, racial, and systemic justice into its work. The change VBCIC seeks with this violence prevention curriculum is a revolutionary return to a life of loving and giving. This way of life provides the tools to care for each other in the old ways of insuring that all have what is needed to economically and spiritually survive in a world that we see as based on greed and usurpation of power to feed the few at the expense of the many. The real change will be seen when indigenous people find ways to return to our old ways of relating such as: • trading and bartering, • giving one another what we have left over and do not need, • holding people accountable for their behavior with restorative practices, • creating new forms of shelter so no one is homeless, and • educating ourselves outside what is taught in dominant cultural schooling. Current Status of the Project VBCIC has had four 30-hour “Walking in Balance with All Our Relations” curriculum trainings. These trainings involved a local indigenous group, the Nolumbeka Project; a multicultural anti-racism leadership group in Franklin County, Massachusetts; a faith-based community group; and a multicultural group in Monroe, Connecticut. VBCIC was one of 10 organizations to receive a $225,000 three-year grant from Just Beginnings Collaborative for the prevention of childhood sexual assault. We have funding to continue the evaluation of the “Walking in Balance” curriculum. We also have funding specific to undoing racism work in our community from the Haymarket People’s Fund and the Mark Nathan Foundation. The “Walking in Balance” curriculum was also brought to the Honoring the Ancient Ancestors week-long gathering in Illinois. VBCIC met elders from all over the world. We gave both a workshop and a presentation on historical trauma at this event. VBCIC is now able to bring in elder teachers from both the United States and Canada to offer teachings to the indigenous community with the funding we have received. VBCIC is writing letters, in long-hand, to every tribal and non-tribal community in the Northeast offering training in their communities that they can continue to offer themselves once they are certified as having completed the 30 hours. Once community members are trained, we can reach a far greater number of indigenous people throughout the Northeast and, indeed, the nation. Of critical importance is the availability of a sacred, 76-acre parcel of land that we can use for continuing gatherings and ongoing ceremonies. Spirit brought this our way without effort on our part. This is what is meant by “letting the ball come to you,” as an important elder taught me. “Letting the ball come to you” is not something the dominant culture teaches and is not something I understood so well at first, having received a great deal of Western education. In fact, one of the elders who will teach the “Walking in Balance” curriculum explained to other elders that “although Strong Oak has a Western education, she stills understands her traditions.” This last statement sums up, in a nutshell, the importance of bridging the gap between Western thinking about evaluation and indigenous understanding about prevention as a way of life through governing relationships with all sentient beings. Now that VBCIC has funding, it has its first staffing. The survivor-led Board of Directors will transition to a Governance Board in the coming year. The Principles of the Circle will continue to inform its work. Staff is welcome to all Board meetings, and their voices will be incorporated into the Circle Process of consensus decision-making. We have a strong “Conflict of Interest Policy” in place to make using the Circle Process possible with interested parties not able to vote. All staff will receive the same rate of pay regardless of role. VBCIC governance, mission, funding, and evaluation process is to be in complete accord with Circle Process to insure that our policies never become a source of systemic violence in themselves. We face a real challenge of finding a meaningful evaluation tool for a community of color that sees the world in a radically different way than the dominant culture does. We are working with our professional evaluators in attempt to be culturally relevant through highlighting the qualitative storytelling emphasis of our data collection. Part II. Evaluating the Curriculum Effectiveness by Holly Ramsey-Klawsnik, PhD Strong Oak Lefebvre, LICSW Patrick Lemmon, MA Evaluating culturally specific violence prevention initiatives creates both challenges and opportunities. Part II of this article will describe the work sponsored by the National Sexual Violence Resource Center (NSVRC) to adapt mainstream program evaluation procedures to measure the effectiveness of the “Walking in Balance with All Our Relations” culturally specific prevention program. The Task From October 2013 - September 2015, Holly Ramsey-Klawsnik, PhD, and Patrick Lemmon, MA, worked as evaluation consultants with the VBCIC curriculum authors on behalf of NSVRC. During this period, the curriculum was developed and portions of it were piloted. Entering work on this indigenous project, the goal was to adapt standardized program evaluation methods to measure individual changes in the behavior, attitudes, and intentions of those who would participate in the teaching modules. We also sought to measure changes in both the micro and macro level variables (as described in the Socio-Ecological Model used for this curriculum) that occurred as a result of curriculum implementation. We anticipated that the findings would (a) enable the authors and funder to determine the effectiveness of the curriculum in achieving project goals and (b) inform curriculum and teaching modifications going forward. The consultants reviewed modules as they were produced and suggested revisions. Possible evaluation techniques were considered and discussed for the curriculum as a whole as well as for each of the 12 modules. This review and discussion continued throughout the design and piloting of the curriculum. The challenge was to design and implement a culturally relevant evaluation plan adapted to the needs of the developers and users of this curriculum. Numerous cultural and practical challenges were encountered. For example, it was recognized at the beginning of this process that indigenous people rely upon oral tradition and are much more comfortable contributing information through storytelling rather than in written form. As a result, use of the standard written “pre/post test” to be completed by each participant was not an option. Additionally, it was unclear if indigenous participants would be open to having their pre/post responses audio-recorded or written by an observer. The goal of our work together in attempt to evaluate the effectiveness of this curriculum has been respectful and gentle trial, error, and revision to tailor traditional social science evaluation methods to the needs of the indigenous authors and participants. Designing effective curriculum evaluation methods requires careful consideration of the curriculum goals. As discussed earlier, VBCIC wants to accomplish the goal of promoting a community-wide understanding of respect and responsibility. Conducting evaluation research to determine if this “mega-goal” is accomplished is a daunting challenge. [Image: Close-up of hands holding a pen and paper.] Strong Oak informed the evaluation consultants early on that the biggest problem facing the indigenous community today is gender-based violence. When successful violence prevention strategies are implemented, demonstrating actual reductions in levels of violence in a community or group typically takes years. Nonetheless, it was recommended that VBCIC attempt to capture pre- and post-measures of domestic violence and sexual assault in the communities in which the curriculum was implemented. The project logic model included the goal of reducing gender-based discrimination — for example, “gender based restrictions on full participation in i ndigenous community activities will decrease by 100 percent in VBCIC-hosted events.” It was also recommended that the curriculum authors make the link between gender discrimination and gender-based violence explicit to strengthen their logic model. It is indeed challenging to address prevention program evaluation from an indigenous perspective. For indigenous people, the storytelling is the focus. This is very different from traditional scientific program evaluation methods that test hypotheses via collecting responses to standardized questions and other objective empirical data. For example, the evaluators suggested analyzing crime statistics in the communities using the curriculum to determine if arrests for incidences of interpersonal gender-based violence decreased over time. VBCIC rejected this suggestion stating that for the most part, indigenous people do not trust law enforcement and do not approach them for assistance following incidences of victimization. In their community, they wish to use a restorative justice approach to deal with perpetrators of interpersonal violence, rather than the criminal justice system. The curriculum authors insisted upon culturally relevant evaluation methods and illustrated this need with the following story: For indigenous people, it would not matter if a muskrat or a beaver or a squirrel made it down to the bottom of the lake to bring up the mud that Star Woman used to create the world. It matters only that a small creature with a big heart succeeded. Indigenous people expect the story to change over time, but the values communicated by the story to remain unchanged. Those values are what determine respectful behavior and community life based on compassion and respect. Over the course of the evaluation consultation, many recommendations were made to VBCIC. These included: • Linking each curriculum module to the overall project goals, specifically by making an explicit statement in each module as to how it relates to violence prevention. For example, how does generosity and giveaway relate to preventing gender-based violence? What is the link? How will this module help to achieve project goals? It was anticipated that these answers would also help identify the most appropriate evaluation methods to employ. • Requiring that each module have specific learning objectives and teaching goals and that all of these be linked to violence prevention. • Developing at least one pre/post evaluation component for each of the 12 modules as well as pre/post measure(s) for the entire curriculum. Both short-term and long-term impact needed to be measured using qualitative and quantitative measures. Methods had to be culturally appropriate and relevant for the “teachers” and “learners.” For the consultants to provide helpful suggestions and methods, understanding the relevant cultural and historical issues and current problems faced by the indigenous community was required. The curriculum authors needed to guide the consultants in this learning. Several evaluation methods were piloted with some of the curriculum modules. One method successfully employed was use of a simple 3”x5” inch notecard, one side labeled #1 and the other #2. A simple pre/post question is selected for each module to be read aloud to all participants. The participants are asked to write their answer to this question on side #1 prior to the Talking Stick Circle in which that module is presented. At the close of the circle, the same question is read aloud and participants are asked to write their post answer on side #2. For example, the pre/post question piloted for the “Generosity/Giveaway” module was, “When you hear the words ‘generosity’ and ‘giveaway,’ what comes to mind?” It was a significant finding that participants were willing to provide their pre/post responses in writing when it was not presented as a “test” and the activity did not require the use of complex forms, but merely a common 3”x5” card. At all Talking Stick Circles, assistance is available for participants who cannot or choose not to write their own answers. In some circles, people with vision problems or difficulty with spelling, grammar, or handwriting have allowed a “helper” to put their answers on their card. The pre/post responses collected at the “Generosity/Giveaway” pilot were tabulated and analyzed. Dramatic pre/post differences indicated the effectiveness of both the module and the Talking Stick Circle teaching modality in changing attitudes in the direction of increased focus on the welfare of the community rather than the self, healing and violence prevention, and appreciation of gifts received. These changes are all consistent with the goal of preventing violence. An additional evaluation method has been piloted. At the conclusion of the Talking Stick Circle during which a curriculum module is presented, all participants are asked to verbally state how their participation in that circle changed or impacted them. With the consensus of the group, the responses are tape-recorded. That recording is transcribed, and the responses are analyzed. Powerful responses indicative of an effective curriculum module resulted at the Generosity/Giveaway pilot. Sample responses include: “This has affected me in a very good way. I think regardless of the communities we consider ourselves a part of, we need to do this more, get together more, talk to each other more.” “I’ve been deeply touched by hearing everybody talk.” “I think the way I have been impacted is deepening my understanding of and respect for native wisdom.” Lessons Learned & Recommendations The lessons learned from working on this project have been multiple and complex. 1. It has been extremely valuable for Lemmon and Ramsey-Klawsnik to work as an “evaluation team” to promote discussion, processing, and brainstorming of evaluation methods, experiences, and outcomes. Recommendation: Program evaluators working with groups of a cultural background different from their own will do better if they work in teams. 2. The consultants attempted this evaluation project with great respect for indigenous cultures, people, and traditions and for this project of developing a culturally specific violence prevention curriculum. However, there were times when some members of the VBCIC experienced some aspects of our work as disrespectful. This, of course, took a toll on our rapport and effectiveness. Much work and time needed to be invested to understand where we went wrong and how to repair miscommunications and failed expectations. The outcome was that both the consultants and the curriculum authors learned important lessons. Recommendation: Be prepared to make mistakes and invest time and effort in addressing them when evaluating the work of a group different from your own. The work may not always be as “neutral” as traditional evaluation aims to be due to the need for the evaluators to participate in cultural events sufficiently to understand and create relevant evaluation methods. However, the results can be rich if you are prepared to work through it. 3. The consultants found that although we were brought into this project to consult, teach, and train members of VBCIC how to evaluate their own work, we believe that we have learned more than we have taught. It has been a fascinating, rewarding, and educational process. Indigenous ways of teaching, processing, and handling interpersonal differences are quite different from those of our own heritage and formal training. Recommendation: Be prepared to learn as well as teach and consult. It is critical to be open to changing and growing in one’s personal and professional attitudes, outlook, and methods when working with cultural diversity. Maintaining the position of “expert” will doom the evaluator(s), and the evaluation project, to failure. When doing cross-cultural evaluation consultation, both the consultants and those developing the program under evaluation need to be open to learning from each other. 4. The consultants gained tremendous respect for the indigenous Talking Stick Circle as a method of teaching, processing, and resolving issues. It is a much more time-intensive method than typically employed in mainstream America, and patience in using the method has been required. It is also a method that works best with small groups of about 8 to 10 people, as opposed to larger groups with more than 10. We have seen profound impact on and movement in people participating in the circles. It is a tremendously valuable and effective method for working with groups and for promoting awareness-building and social and interpersonal growth as well as resolution of differences. Recommendation: Be prepared to be patient and expect that the teaching/learning methods of other cultures may be considerably time and labor intensive yet highly effective. 5. The consultants have made much slower progress in this project than is typical. We have needed to work at the pace of VBCIC and the indigenous, gifted curriculum authors/teachers. All aspects of the project have been time-consuming: writing the curriculum modules, reviewing and revising them, testing them, designing and testing evaluation methods, and discussing all of it. We have learned that to work effectively with this group of insightful and talented people who are the creators of “Walking in Balance with All Our Relations,” we must respect not only their process, but also their timetable. Recommendation: Be prepared to move more slowly and accomplish tasks at a different pace when evaluating the pioneering efforts of a cultural group different from your own. Summary We have worked, learned, and experimented together with culturally relevant and useful violence prevention evaluation methods. We believe that the work involved in this indigenous violence prevention curriculum and culturally relevant program evaluation is unique and holds promise for the work of others. Circle Keeper Strong Oak has expressed, “Before working with the evaluators, I thought of evaluation as something not really relevant to my community. At worst, I thought of it as a way to keep resources out of communities of color. Now I am excited about developing tools to evaluate our work. I am believing that communities of color have much to learn and teach about evaluation for all communities.” Contributors • Strong Oak Lefebvre, LICSW, Visioning B.E.A.R. Circle Intertribal Coalition • Holly Ramsey-Klawsnik, PhD, Klawsnik & Klawsnik Associates • Patrick Lemmon, MA, Oregon Sexual Assault Task Force References Centers for Disease Control and Prevention, National Center for Injury Prevention and Control, Division of Violence Prevention. (2015). The social-ecological model: A framework for prevention. Retrieved from http://www.cdc.gov/violenceprevention/overview/social-ecologicalmodel.html Horvath, R. J. (1972). A definition of colonialism. Current Anthropology, 13, 45-57. doi:10.1086/201248 United Nations. (n.d.). Indigenous peoples, indigenous voices: Factsheet. Retrieved from http://www.un.org/esa/socdev/unpfii/documents/5session_factsheet1.pdf Onondaga Nation. (n.d.) The Doctrine of Discovery and its enduring impact on indigenous peoples. Retrieved from http://www.onondaganation.org/mediafiles/pdfs/un/Doctrine%20of%20Discovery.pdf Smith, A. (n.d.) Indigenous peoples and boarding schools: A comparative study. Retrieved from the United Nations: http://www.un.org/esa/socdev/unpfii/documents/IPS_Boarding_Schools.pdf Visioning B.E.A.R. Circle Intertribal Coalition. (in press). Walking in balance with all our relations: An American Indian primary prevention bystander intervention curriculum based on indigenous pre-colonization values.