by Jacob B. Ukeles, Steven M. Cohen and David Dutwin

In a recent article in eJewishPhilanthropy, “So You Want to do a Jewish Community Study: Five Things to Consider First” Mara Koven-Gelman, who served as the lead consultant for the 2013 Greater Buffalo Jewish Community Study, draws five key lessons from her experience.

Four out of her “five things to consider” are relatively benign, and are quite helpful to communities considering a community study. The fifth, on sampling methodology, is neither benign nor helpful. This section of her article praises the use of affiliated lists+ ethnic surnames samples, and disparages Random Digit Dialing methods as not cost-effective and out of date.

Unfortunately, there is close to zero probability that the numbers reported in the study accurately represent the entire Buffalo Jewish community, including those who are neither affiliated nor identifiably Jewish with names such as Katz, Friedman and Goldstein. The reported data probably are a reasonable approximation of the affiliated end of the Jewish spectrum, with some non-affiliated Horowitz’s thrown in for good measure.

Most Jewish community studies include a strenuous effort to reach Jews who are not identified with, nor known by, the Jewish community. “Random Digit Dial” (RDD) interviews – where interviewers call thousands of local landline and cell phone numbers randomly – are expensive and difficult to conduct. But Random Digit Dialing is crucial in order to paint an accurate picture of the full Jewish community, including those who fail to appear on community lists.

As the report clearly states, the Buffalo study is not RDD-based. It relies exclusively on organizational lists, largely composed of lists of known Jews and on lists of people with distinctive ethnic surnames (e.g., Goldberg) who have a listed telephone.

Across two dozen recent Jewish community surveys across the United States, lists of known Jews have included only about one third of Jews in a given community. Typically, not more than a fifth of Jews have distinctive Jewish names. But many Jews on community lists have distinctive Jewish names as well. So, it is highly likely that the list + distinctive name method did not have the ability to reach more than half of all the Jewish households in the community. Known Jews are older, more Jewishly engaged, more inmarried and more likely to live in Jewish neighborhoods. It is highly likely, then, that compared with those included in this study, the actual Buffalo Jewish community is younger, more highly intermarried, more geographically dispersed, and much less Jewishly engaged.

Not surprisingly, the non-random Buffalo data over-represents Jews who are connected. The study’s reported intermarriage rate – 21% of all married couples – is among the lowest rates reported in recent Jewish community studies, especially among those communities without large Orthodox populations. Even more implausibly, the study reports that 46% of the intermarried are synagogue members. This figure is the highest percent of intermarried couples who are synagogue members of any community in the United States. Among 50+ RDD-based studies, the next highest percentage is the 38% reported for Cincinnati (2008), a center of the Reform movement with a community culture of outreach to the intermarried.

In line with the widely reported low rates of Jewish engagement among intermarried families, the vast majority of community studies show significant differences between inmarried and intermarried. For example, in the Cohen Center’s RDD-based Boston study (2005), synagogue affiliation was 27% of intermarried couples and 63% of inmarried couples – a gap of 36%. In contrast, the non-RDD Buffalo study found synagogue affiliation rates of 46% among the intermarried and 69% of the inmarried – a difference of only 23 percentage points. Is it plausible that Buffalo, NY has the highest intermarried couple’s synagogue affiliation rate of any Jewish community in the US? Why do we find the smallest difference between inmarried and intermarried couples?

The study’s introduction could have alerted the reader to the likely consequences of the list-based designed. It could have stressed the undercount of under-engaged Jews, while touting the virtues of the study for understanding the engaged population. But not content with justifiable assertions, the study’s text makes a remarkable claim: “Some groups are likely undercounted in this study … we do not believe that these undercounts introduce any significant bias into our estimates.” On what basis? As a result of what comparison with dozens of other studies?

In contrast with the introduction, the Methods appendix in the same report, which follows the analysis, takes a far different tack. It spells out the undercounted groups: “[S]ome groups are particularly likely to be underrepresented in the sample. Most significant among these are unaffiliated Jews (including new residents and intermarried families); residents of counties other than Erie County; and young adult Jews.” In short, who’s missing? Answer: the newly arrived, the intermarried, the geographically remote, and young adults – the very population groups that are most often the target of community outreach.

For good reason, RDD designs have dominated social science research on American Jews for decades: they have a strong chance of getting the numbers right, and embracing the entire Jewish population – both those we know, as well as those we don’t know.

True, RDD studies are costly. But they are not nearly impossible as Koven-Gelman suggests. It is simply not true, as she writes, that RDD means one “may need to call 10,000 numbers to find one Jewish person who may not participate.” In the 2011 Cleveland RDD-based study for example, most completed RDD interviews required 200 to 300 calls, far less than the 10,000 claimed by Koven-Gelman.

In larger communities, RDD studies are the only way to generate meaningful and useful data about the entire Jewish population. In smaller communities, such as Buffalo, an RDD study may prove to be too expensive. In these situations, one can focus primarily on affiliated Jews, as was the case with the 2013 study.

Used with care, a study which includes a great deal of data about affiliated Jews and some data about less engaged Jews can be very useful in community planning. In Buffalo, there’s every indication that the local federation leadership is well aware of the strengths and limitations of the 2013 study. This type of (non-RDD) study can indeed illuminate a range of issues, including social service needs and finding ways to improve Jewish education. However, one point needs to be understood and stated clearly: Such a study will be less useful for understanding the less engaged sectors of the community or geographic areas with lower Jewish density.

Thus, we derive a sixth important lesson for communities that do a study of primarily affiliated Jews: Planners, researchers, and community leaders need to be explicit in articulating that a study focused primarily on affiliated Jews is exactly that, and is not representative of a total Jewish community.

Jacob B. Ukeles, PhD is president of Ukeles Associates, Inc. and co-president of Jewish Policy and Action Research (JPAR). JPAR has been responsible for more than 20 Jewish community studies in the United States.

Professor Steven M. Cohen, PhD is Research Professor of Jewish Social Policy, Hebrew Union College-Jewish Institute of Religion, and Director of the Berman Jewish Policy Archive @ NYU Wagner.

David Dutwin, PhD is executive vice president and chief methodologist of SSRS and principal methodologist of JPAR. He has conducted a wide range of studies on Jews and other religious groups for the Pew Research Center, the Public Religion Research Institute, Harvard University and others.