By Daniel Olson
In the Jewish community, the results of survey research often persuade funders and program developers to direct resources toward certain projects and away from others. These stakeholders have abstract goals like strengthening Jewish identity, and surveys alluringly quantify these imprecise outcomes.
While surveys can be influential, their findings can mislead because even the best survey research contains error, or gaps between the survey’s findings and the facts on the ground. Experienced, well-funded pollsters conducted surveys until Election Day predicting that Sec. Hillary Clinton would win Wisconsin, Michigan, and Pennsylvania, but she lost all three.
Honest survey researchers recognize that error is inevitable and will make transparent their attempts to account for and minimize it. This process, unfortunately, is sometimes glossed over, especially in survey research without peer review. Such surveys might contain too much error to say anything meaningful about a population, yet are published and used to persuade.
Savvy consumers of survey data know to ask the following three questions of any survey. Doing so can help you determine whether a survey’s results provide meaningful information or merely “alternative facts.”
1. Did the survey actually measure what it claims to measure?
Survey researchers interested in the Jewish community often claim to measure abstract concepts like Jewish identity or engagement. To measure these concepts, they ask about specific behaviors like Sabbath and kashrut observance, synagogue membership, and in-marriage. If the researchers do not make a convincing argument for why those questions best measure the concept, then you should be skeptical of a survey’s validity.
Furthermore, researchers can ask questions that respondents might misinterpret. Responsible researchers carefully word and test questions to avoid this “measurement error,” but even that is no guarantee that respondents will perfectly understand each question.
For example, a Pew study on Orthodox Jews from 2015 asked: “Do you personally refrain from handling or spending money on the Jewish Sabbath?” Admirably, the researchers admitted in their report that this question, which contains a double negative, could have been confusing to respondents, especially non-native English speakers. Measurement error also occurs when respondents inconsistently interpret relative terms like “very emotionally attached” or “rarely.”
To look for measurement error, you should examine the questions, determine possible misinterpretations, and check if the researcher accounted for them.
2. What group was the survey trying to understand?
Every survey has a “target population,” or the group the survey researcher wants to understand, such as every American Jewish teenager, all alumni of Birthright Israel, or the Jewish seniors living in a metropolitan area. However, it is usually impossible to obtain accurate and complete contact information for everyone in a given target population. So, researchers must rely on incomplete or inaccurate lists, resulting in “coverage error.”
For example, not every Jew who lives in a metropolitan area is in a Federation or synagogue phone directory. If researchers use those lists alone to study all Jews who live in that area, the study will not reflect the opinions of Jews not affiliated with these institutions.
To look for coverage error, pay attention to how survey researchers define their target population, ask where they got their list(s) of names, and evaluate whether these lists cover the entire target population.
3. Who didn’t take the survey?
Researchers administer their survey to either everyone on their lists, or to a selected sample. But either way, not all of the people selected to take the survey respond.
This “nonresponse error” is especially problematic if the reasons for nonresponse are also what the survey is trying to measure. For example, in a program evaluation survey, those with strong feelings about the program will likely respond at higher rates. The survey will fail to measure the full range of participants’ reactions, leading to an inaccurate portrait.
To account for this error, better researchers carefully upweight the results of groups with lower response rates to better reflect the target population (for example, if women responded at a lower rate than men).
A high response rate decreases the risk of nonresponse error, but does not fully account for it. Still, responsible researchers take steps to increase the response rate, especially among subgroups that would otherwise respond at lower rates. Look to see if the researchers used any of the following strategies to increase response rates:
- Administering the survey in person or by phone, instead of online or by mail.
- Contacting non-respondents multiple times or through different means
- Offering incentives for completing the survey.
Error is inevitable in survey research and even well-conducted studies can mislead. Answering these three questions will help you tell if the researchers sufficiently minimized error such that their results can be a meaningful basis for evaluation.
Daniel Olson is a Steinhardt Fellow in the doctoral program in Education and Jewish Studies at NYU. He is also a Wexner Graduate Fellow/Davidson Scholar. He studies disability and inclusion in Jewish education.
Top
Offering incentives for responding to a survey can be a double-edged sword. People who take a survey just for the incentive may give answers that are not well thought out and don’t reflect their true feelings.
Daniel:
Thank you for providing this article highlighting three important questions consumers of survey research should ask.
One of the missions of the Berman Jewish DataBank (jewishdatabank.org/About/index.cfm), a project of The Jewish Federations of North America, is to promote transparency about research methods that will help answer questions like the ones you pose. The DataBank is working to develop a communal research culture in which it is expected that descriptions of research methods always accompany the release of research findings. We applaud commissioning organizations and researchers who are already committed to transparency about how they conduct their research.
Research producers and consumers won’t always agree on the best methods to investigate the many important issues our community faces. But we can all agree that it is vital to provide information about research methods so that we are able to assess the quality of the research we consume.
Laurence Kotler-Berkowitz
Director, Berman Jewish DataBank
The Jewish Federations of North America
This is a good start. Here are a couple of other items:
–How was the survey administered?
There are a plethora of options (mail, telephone, on-line, to name a few) and within these options an even wider selection. Each method comes with strengths and limitations.
One example from last year’s elections–the 538 site did a great job of detailing every poll and rating each firm on a variety of methodological considerations. There were certain firms, for example, that in their telephone surveys only called land line telephones. These polls ended up skewed far more strongly for Trump, since older, white and more rural Americans are overrepresented among landline users, while younger, urban and more racially Americans who are overrepresented among cell-phone only users (and who trend Democratic) were not included in the sample.
–Who is funding this research, and what is the relationship between the researcher and the funder?
Many organizations that conduct research have a political agenda of one sort or another. This has the potential to impact every aspect of the research process, including how a topic is framed, what issues are not discussed, the ordering of questions, the terms that are used to describe the what or who that is being studied, and how findings and analyses are presented. Even a cursory glance of the questionnaires designed by the media outlets across the political spectrum reveal important differences here. Potential ethical and political minefields abound, and we in the Jewish community need to be acutely conscious of these issues as well.