In parallel with the growing awareness of the historical and systemic drivers of inequality, professionals in the field of evaluation are also reflecting on the intersection between our practices and a more equitable society. At its inception, the field of evaluation was created by academic research institutions and the federal government to “inform the allocation of public dollars and the effectiveness of the investments” (Dean-Coffey, 2018). In other words, the field was started by those who held power based on their ability to create knowledge (along with many other factors), to inform those with the power to allocate money, to make decisions about the value of investments in programs that supported people without power. This dynamic can have lasting and unintended consequences for the people and programs involved in evaluation studies. 

With this awareness, Rosov Consulting designed an evaluation that uses more equitable techniques and is in service of creating a more equitable Jewish nonprofit sector for our client, JPro. By surveying and interviewing professionals in the field, we hope to better understand what factors enable or hinder their access to professional growth opportunities so that JPro can make systemic changes to ensure those experiences are accessible to all. (If you work for an organization that serves the Jewish community and would like to share your experience, you can access the survey here.

Courtesy

Although we are penning this piece before we have findings, there are ways in which an equity lens can be applied before any data is collected. One such resource is the Equitable Evaluation Initiative (EEI). These principles — in addition to being used here as a framing device — are valuable for our work with funders and grantees in the Jewish communal sector and push us to continually reflect upon and improve our work process and product. 

Unpacking the Equity Framework 

Principle #1: Evaluation work is in service of and contributes to equity. Production, consumption and management of evaluation and evaluative work should hold at its core a responsibility to advance progress toward equity. 

Beginning with the request for proposal (RFP) for the evaluation work, JPro embedded equity into the process by offering to compensate those who responded to the RFP but were not selected to complete the work. Providing compensation lowers the barrier for entry, lessens the financial burden for applicants, and does not assume all firms can absorb the cost of proposal writing. This practice is one way to encourage a more diverse applicant pool. How organizations allocate their money is a marker for what they value. Setting aside a portion of the finite financial resources to fairly compensate people for their time reiterates the value placed on their contributions and helps to ensure there are not financial repercussions associated with doing so. 

• Learn more about offering frameworks for reflecting on collecting and sharing knowledge in a more equitable way: Chicago Beyond Why am I always being researched? (p. 1-17) and AISP Centering Racial EquityThroughout the Data Life Cycle (p. 14-33) 

Principle #2: Evaluative work should be designed and implemented in a way that is commensurate with the values underlying equity work. It should be multiculturally valid and oriented toward participant ownership. 

Data collection, even from the design phase, should include an equity approach, which pushes us to listen carefully to the community being studied and build processes of feedback that ensure a variety of voices are heard and incorporated. 

Integral to the work with JPro was engaging stakeholders from the community to provide their insights and lived experiences. In the first phase of the work — developing a conceptual framework that visually outlines the working theories about access and barriers — we convened professionals in the field to reflect on whether the framework included and clearly articulated their experiences around these issues. It was important that the working group included individuals from a variety of sectors and in different points in their careers. After drafting a survey instrument, we interviewed five additional people and asked them to reflect on the extent to which their lived experience was accurately captured through the existing survey questions. We worked to test our assumptions and hear directly from the population we hope to engage in the study. In doing so, we relinquished some power as researchers to impose our assumptions on participants — as articulated through our research, survey, and interview questions. 

• Learn more about engaging participants in evaluation: Health Nexus Participatory Evaluation Toolkit 

Principle #3: Evaluative work can and should answer critical questions about the: Effect of a strategy on different populations and on the underlying systemic drivers of inequity, and the ways in which history and cultural context are tangled up in the structural conditions and the change initiative itself. 

The questions that drive an evaluation should be reflected upon with an equity lens. This may be adding or rewording questions to uncover how subpopulations (e.g., Jews of Color, Mizrahi Jews, those with differing abilities) have benefited from an initiative in different ways. It may involve asking explicitly how historical and cultural contexts impact intended outcomes; for example, the ways in which first- and second-generation Jewish immigrants choose to participate in Jewish life. 

In framing the project, JPro acknowledged existing inequalities. Our data collection efforts seek to uncover how these inequalities impact JPro’s core services. Specifically, what personal, organizational, and sector-wide characteristics impact a person’s ability to access professional growth opportunities? We seek to uncover whether people with certain demographic characteristics (e.g., religious affiliation, age, career stage, socioeconomic status) have differing access to professional growth opportunities. What is the extent to which an organization’s systems, policies, culture, size, mission/focus area and other characteristics play a role? We also want to know whether the content, cost, format and location of the growth experience make a difference. Knowing these answers will allow JPro and the field to make changes so that offerings are more accessible to all, regardless of their personal and organizational characteristics.

• Learn more about choosing and writing evaluation questions that advance equity: UNICEF How to Design and Manage Equity-focused Evaluations (p. 33-38) and We All Count Framing Research Questions that Reflect Who is Expected to Change 

Evaluators are in a position of power when we collect and share information about the programs and people we study. It is imperative that we understand this position and use it to better understand the current and historical factors that perpetuate inequalities. This can be done by asking and answering questions that uncover differences in subpopulations, including populations of interest into the process, and giving knowledge back and empowering communities to use the information to make change.   

Allison Magagnosc, MS, is a senior project associate at Rosov Consulting.