by Rabbi Drew Kaplan

Two years ago was when the problem first surfaced for me. I was at Hillel Institute, the national staff conference for Hillel: The Foundation for Jewish Campus Life, in a session with the other rabbis at the conference. Under discussion was a study/report carried out for Hillel regarding the Senior Jewish Educators (SJEs). One of the rabbis in the room asked about measuring the impact of the SJEs: was it only with regard to the number of students they had interfaced with? What about measuring the qualitative impact upon those students? There was no answer, but that question left a strong impact on me. And not just then, but these questions continued to pop up in different conversations and most recently popped up when I attended Birthright Israel NEXT‘s Southwest NEXTwork launch three months ago, when there was a significant discussion over numbers of program attendance.
Whether it’s a program grant I am writing or discussing the desired goals/impact of a program, I am less concerned with how many people show up as a marker of programmatic success than I am with how the program actually went and its impact upon the participants/attendees. Inasmuch as how many people attend a program may reflect upon the marketing or promotion aspect of its planning, it is but one small piece.

Granted, the above may belie a bias of my programming: I am more concerned with substantial programs: whether it be a Jewish learning program (I am a rabbi, after all), a leadership development event, or some other activity meant to instill something beyond simply attending, socializing, and perhaps eating. It is these activities that I want to make sure there is a way to measure the qualitative aspects of its impact upon participants.
However, often when applying for grants or talking with other people about programs, they want to know DAM goals: making sure that goals are definable, attainable, and measurable (something I learned when I attended BBYO‘s International Leadership Training Conference (ILTC) in 1998). However, how does one define or, more importantly, measure the impact on their leadership skills?
Moreover, even if there is a way to measure it, when and how ought we go about doing it? Is it simply an entry survey and an exit survey to see if their leadership skills or Jewish identity has increased? If so, is it a binary measurement of yes/no or is it more complex, even something as seemingly simple as measuring it on a scale of 1-10. Let’s say you try this and you are able to get information such as “There was a 56% increase in Jewish identity from this event”, what does that mean? Furthermore, how does that translate?

But the biggest question mark is that this work generally is actually development work (no, not fundraising), such that we are seeing how we can provide these ideas, activities, etc. to develop in these [young (although not necessarily so)] people. Thus, even if they can tell us how they were affected, impacted, or helped through such programming immediately afterwards, what about six months down the road? One year later? What about 5, 10, 15 years later?
14 years ago, as I mentioned before, I attended BBYO’s ILTC, which was a great program, and I can still point to having learned DAM goals then. But is there any way of measuring that? What about measuring impact of a program years later after there’s been much turnover amongst staff at a particular organization?
Granted, I think anyone considering this issue must quickly recognize that acquiring, processing, and analyzing quantitative data (e.g. numbers of attendees) is significantly easier than doing the same for qualitative data (that’s actually an understatement). When meeting with the awesome Esther Kustanowitz (how’s that for a shout-out, EstherK?) the other day, she said that it took a while for the Jewish community to feel the impact of the Taglit-Birthright Israel trips because the attendees are still developing, which does not minimize the need to measure a program’s individual impact. She furthermore suggested that data collection is often driven by what funders (or grant-giving organizations) are interested in. Such that, they are probably more interested in knowing how many people showed up at an event than how they are affected. But maybe there’s a way that those of us involved in Jewish communal work can push back and offer qualitative means of measurement.
My hope of this post (in addition to feeling cathartically releaved to have gotten it out of my system and onto the Internet) was to push the conversation not only toward thinking about the importance and need of qualitative data amongst participants of Jewish communal programs, but also to see if we can develop methods of such data collection as well as to be able to get it into the minds of funders that a program’s impact is more than just number of feet who walk into a program.
Kol HaKavod, Drew! I had the same concerns when I started “evaluating” for supervisors and donors as a Hillel rabbi 15 years ago and now I’m finishing my PhD dissertation pushed by the same question! It is a real problem that numbers dominate so much here. IMO this has a lot to do with funders coming from the business world rather than the social sciences which so many of us are trained in, and I think that preference for numbers even extends to a bias towards quantitative analysis rather than qualitative analysis by funders too in my experience. The numbers just seem more reliable to them but as you point out, we can’t measure everything so succinctly in part because human beings are not so succinct. Thanks for a great opinon piece!
Scott, Thank you so much for the kind words! You are totally right, especially about the funders, and I am glad that you were/are bothered by the same question(s) with all of your experience 🙂 What is your PhD dissertation topic?
Hi Drew,
Great points, sorry I didn’t get a chance to talk to you when I was at the HIllel Institute this week briefly. We are working on tracking those qualitative metrics through the GrapeVine platform, in partnership with Hillel: FJCL and others. Here’s the article in eJP about it: http://tinyurl.com/cacusy9. If you are interested in joining the Hillel adaptation of GrapeVine, please let me know!
Best,
Sacha
Sacha,
I am glad that we were able to connect for a few moments to discuss GrapeVine while at Hillel Institute this past week and that work seems interesting and definitely sounds great!
Drew
The draft title is “Sneaking a Peek in the Black Box: A Grounded Theory of How Jewish Experiential Education Impacts the Identity Development of Jewish Emerging Adults.” The operative word is how. I’m trying to understand exactly how – and by extension why – Birthright and ASB trips work by looking at the experiences within the trips. We know they work and we know what parts work best based on outcome measurements, but we don’t really know how or why so I’m trying to shed some light on that by trying to theorize a Jewish Experiential Education model specifically for this age group. Draft due to my committee in a few weeks so we’ll see if I am making any sense soon. 🙂 I’m posting on your piece here on my blog today FYI.