Beyond Counting: The Importance of Depth and Scalability
by Renee Rubin Ross and Matt Grossman
Drew Kaplan is right. Funders and organizations offering Jewish experiences must pay attention to qualitative assessment. While it is challenging to measure the effects that unique Jewish experiences have on the lives of individuals, thanks to partnerships between funders and Jewish educational organizations, new approaches are emerging to do just that.
The Jim Joseph Foundation has engaged many of its grantees around the topic of measuring qualitative impact and has invested significant resources to advance the Jewish community’s efforts in this area. Different Jewish organizations think about the qualitative outcomes of their work in different ways. To cite one example, BBYO’s 2011 Impact Study, funded by the Charles and Lynn Schusterman Family Foundation, compared the results of four audience segments including an adult control group which found that a BBYO experience during one’s teen years led to a higher likelihood that one would express pride in being Jewish, pursue leadership opportunities and maintain a predominantly Jewish social group. It also found that the individual’s commitment to Jewish learning and the celebration of Jewish holidays was impacted but in a slightly less pronounced way. In response to these findings, the Jim Joseph Foundation invited and funded a grant proposal from BBYO for the hiring of three Directors of Jewish Enrichment (DJEs) to infuse greater Jewish content into BBYO’s teen programming. A collaborative effort between the Foundation, BBYO and third-party evaluators is currently using a qualitative study to explore the DJEs’ work with teens, staff and advisors.
Another qualitative model that is being explored is the impact that relationships with peers and educators have on various dimensions of Jewish growth. As described in a recent Jim Joseph Foundation posting, A Personal Reflection: Five Years at the Jim Joseph Foundation:
An evaluation of the Foundation’s grant to Hillel found that the number of interactions with either an educator or a knowledgeable peer is correlated with individual Jewish student growth. The findings were similar for Moishe House: more involvement with peers led to more Jewish growth. The centrality of these relationships on a young person’s decision to pursue a Jewish life continues to heighten the Foundation’s focus on who is placed in the path of young Jews and how often (Adene Sacks).
These examples are a few among many we have seen in the Jewish education world of organizations seeking to measure qualitative outcomes for the purpose of assessing their program models and finding ways to improve them. We know many national and local funders who see the value in this work and are ready to invest in it.
It should also be acknowledged that despite working with some of the best social scientists in the field, there are challenges in measuring qualitative impacts. This is partly because there is no definitive agreement as to what constitutes Jewish growth. As Bethamie Horowitz and others have taught, Jewish growth is not linear. To our knowledge, no universally accepted methodology of measurement exists that captures Jewish growth over many years and at many different points in one’s life cycle.
Related to this, individual funders and Jewish organizations do not necessarily track identical indicators of Jewish growth. For example, the Jim Joseph Foundation is most interested in Jewish learning and changes in Jewish behaviors. Other foundations may be more interested in patterns of social activism or an individual’s commitment to their local Jewish community.
With all of this said, one cannot overlook the importance of quantitative outcomes. If the Jewish community is going to be changed in a profound way, funders and experience providers must equally weigh scalability and depth by widely offering experiences which create deep, permanent connections to Jewish values, beliefs, behaviors and relationships. Together, we should continue to explore the area of qualitative measurement with the hope that we can foster a shared understanding of how people are changed through a myriad of compelling Jewish educational experiences.
Dr. Renee Rubin Ross is a Program Officer at the Jim Joseph Foundation. Matt Grossman is the Executive Director of BBYO.