Qualitative Metrics for the Jewish Community?

by Rabbi Drew Kaplan

Two years ago was when the problem first surfaced for me. I was at Hillel Institute, the national staff conference for Hillel: The Foundation for Jewish Campus Life, in a session with the other rabbis at the conference. Under discussion was a study/report carried out for Hillel regarding the Senior Jewish Educators (SJEs). One of the rabbis in the room asked about measuring the impact of the SJEs: was it only with regard to the number of students they had interfaced with? What about measuring the qualitative impact upon those students? There was no answer, but that question left a strong impact on me. And not just then, but these questions continued to pop up in different conversations and most recently popped up when I attended Birthright Israel NEXT‘s Southwest NEXTwork launch three months ago, when there was a significant discussion over numbers of program attendance.

Whether it’s a program grant I am writing or discussing the desired goals/impact of a program, I am less concerned with how many people show up as a marker of programmatic success than I am with how the program actually went and its impact upon the participants/attendees. Inasmuch as how many people attend a program may reflect upon the marketing or promotion aspect of its planning, it is but one small piece.

Granted, the above may belie a bias of my programming: I am more concerned with substantial programs: whether it be a Jewish learning program (I am a rabbi, after all), a leadership development event, or some other activity meant to instill something beyond simply attending, socializing, and perhaps eating. It is these activities that I want to make sure there is a way to measure the qualitative aspects of its impact upon participants.

However, often when applying for grants or talking with other people about programs, they want to know DAM goals: making sure that goals are definable, attainable, and measurable (something I learned when I attended BBYO‘s International Leadership Training Conference (ILTC) in 1998). However, how does one define or, more importantly, measure the impact on their leadership skills?

Moreover, even if there is a way to measure it, when and how ought we go about doing it? Is it simply an entry survey and an exit survey to see if their leadership skills or Jewish identity has increased? If so, is it a binary measurement of yes/no or is it more complex, even something as seemingly simple as measuring it on a scale of 1-10. Let’s say you try this and you are able to get information such as “There was a 56% increase in Jewish identity from this event”, what does that mean? Furthermore, how does that translate?

But the biggest question mark is that this work generally is actually development work (no, not fundraising), such that we are seeing how we can provide these ideas, activities, etc. to develop in these [young (although not necessarily so)] people. Thus, even if they can tell us how they were affected, impacted, or helped through such programming immediately afterwards, what about six months down the road? One year later? What about 5, 10, 15 years later?

14 years ago, as I mentioned before, I attended BBYO’s ILTC, which was a great program, and I can still point to having learned DAM goals then. But is there any way of measuring that? What about measuring impact of a program years later after there’s been much turnover amongst staff at a particular organization?

Granted, I think anyone considering this issue must quickly recognize that acquiring, processing, and analyzing quantitative data (e.g. numbers of attendees) is significantly easier than doing the same for qualitative data (that’s actually an understatement). When meeting with the awesome Esther Kustanowitz (how’s that for a shout-out, EstherK?) the other day, she said that it took a while for the Jewish community to feel the impact of the Taglit-Birthright Israel trips because the attendees are still developing, which does not minimize the need  to measure a program’s individual impact. She furthermore suggested that data collection is often driven by what funders (or grant-giving organizations) are interested in. Such that, they are probably more interested in knowing how many people showed up at an event than how they are affected. But maybe there’s a way that those of us involved in Jewish communal work can push back and offer qualitative means of measurement.

My hope of this post (in addition to feeling cathartically releaved to have gotten it out of my system and onto the Internet) was to push the conversation not only toward thinking about the importance and need of qualitative data amongst participants of Jewish communal programs, but also to see if we can develop methods of such data collection as well as to be able to get it into the minds of funders that a program’s impact is more than just number of feet who walk into a program.