• Home
  • About
    • About
    • Policies
  • Submissions
    • Op-eds
    • News / Announcements
  • Contact
  • Donate
  • Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
  • Skip to footer

eJewish Philanthropy

Your Jewish Philanthropy Resource

  • News Bits
  • Jewish Education
  • Readers Forum
  • Research
  • Show Search
Hide Search
You are here: Home / Management Tools / 3 Questions You Should Ask of Any Survey

3 Questions You Should Ask of Any Survey

February 28, 2017 By eJP

By Daniel Olson

In the Jewish community, the results of survey research often persuade funders and program developers to direct resources toward certain projects and away from others. These stakeholders have abstract goals like strengthening Jewish identity, and surveys alluringly quantify these imprecise outcomes.

While surveys can be influential, their findings can mislead because even the best survey research contains error, or gaps between the survey’s findings and the facts on the ground. Experienced, well-funded pollsters conducted surveys until Election Day predicting that Sec. Hillary Clinton would win Wisconsin, Michigan, and Pennsylvania, but she lost all three.

Honest survey researchers recognize that error is inevitable and will make transparent their attempts to account for and minimize it. This process, unfortunately, is sometimes glossed over, especially in survey research without peer review. Such surveys might contain too much error to say anything meaningful about a population, yet are published and used to persuade.

Savvy consumers of survey data know to ask the following three questions of any survey. Doing so can help you determine whether a survey’s results provide meaningful information or merely “alternative facts.”

1. Did the survey actually measure what it claims to measure?

Survey researchers interested in the Jewish community often claim to measure abstract concepts like Jewish identity or engagement. To measure these concepts, they ask about specific behaviors like Sabbath and kashrut observance, synagogue membership, and in-marriage. If the researchers do not make a convincing argument for why those questions best measure the concept, then you should be skeptical of a survey’s validity.

Furthermore, researchers can ask questions that respondents might misinterpret. Responsible researchers carefully word and test questions to avoid this “measurement error,” but even that is no guarantee that respondents will perfectly understand each question.

For example, a Pew study on Orthodox Jews from 2015 asked: “Do you personally refrain from handling or spending money on the Jewish Sabbath?” Admirably, the researchers admitted in their report that this question, which contains a double negative, could have been confusing to respondents, especially non-native English speakers. Measurement error also occurs when respondents inconsistently interpret relative terms like “very emotionally attached” or “rarely.”

To look for measurement error, you should examine the questions, determine possible misinterpretations, and check if the researcher accounted for them.

2. What group was the survey trying to understand?

Every survey has a “target population,” or the group the survey researcher wants to understand, such as every American Jewish teenager, all alumni of Birthright Israel, or the Jewish seniors living in a metropolitan area. However, it is usually impossible to obtain accurate and complete contact information for everyone in a given target population. So, researchers must rely on incomplete or inaccurate lists, resulting in “coverage error.”

For example, not every Jew who lives in a metropolitan area is in a Federation or synagogue phone directory. If researchers use those lists alone to study all Jews who live in that area, the study will not reflect the opinions of Jews not affiliated with these institutions.

To look for coverage error, pay attention to how survey researchers define their target population, ask where they got their list(s) of names, and evaluate whether these lists cover the entire target population.

3. Who didn’t take the survey?

Researchers administer their survey to either everyone on their lists, or to a selected sample. But either way, not all of the people selected to take the survey respond.

This “nonresponse error” is especially problematic if the reasons for nonresponse are also what the survey is trying to measure. For example, in a program evaluation survey, those with strong feelings about the program will likely respond at higher rates. The survey will fail to measure the full range of participants’ reactions, leading to an inaccurate portrait.

To account for this error, better researchers carefully upweight the results of groups with lower response rates to better reflect the target population (for example, if women responded at a lower rate than men).

A high response rate decreases the risk of nonresponse error, but does not fully account for it. Still, responsible researchers take steps to increase the response rate, especially among subgroups that would otherwise respond at lower rates. Look to see if the researchers used any of the following strategies to increase response rates:

  • Administering the survey in person or by phone, instead of online or by mail.
  • Contacting non-respondents multiple times or through different means
  • Offering incentives for completing the survey.

Error is inevitable in survey research and even well-conducted studies can mislead. Answering these three questions will help you tell if the researchers sufficiently minimized error such that their results can be a meaningful basis for evaluation.

Daniel Olson is a Steinhardt Fellow in the doctoral program in Education and Jewish Studies at NYU. He is also a Wexner Graduate Fellow/Davidson Scholar. He studies disability and inclusion in Jewish education.

Print Friendly, PDF & Email

Share this:

  • Click to share on Twitter (Opens in new window)
  • Click to share on Facebook (Opens in new window)
  • Click to share on LinkedIn (Opens in new window)

Filed Under: Management Tools

Click here to Email This Post Email This Post to friends or colleagues!

Reader Interactions

Comments

  1. Ron Wegsman says

    February 28, 2017 at 3:08 pm

    Offering incentives for responding to a survey can be a double-edged sword. People who take a survey just for the incentive may give answers that are not well thought out and don’t reflect their true feelings.

  2. Laurence Kotler-Berkowitz says

    February 28, 2017 at 10:17 pm

    Daniel:

    Thank you for providing this article highlighting three important questions consumers of survey research should ask.

    One of the missions of the Berman Jewish DataBank (jewishdatabank.org/About/index.cfm), a project of The Jewish Federations of North America, is to promote transparency about research methods that will help answer questions like the ones you pose. The DataBank is working to develop a communal research culture in which it is expected that descriptions of research methods always accompany the release of research findings. We applaud commissioning organizations and researchers who are already committed to transparency about how they conduct their research.

    Research producers and consumers won’t always agree on the best methods to investigate the many important issues our community faces. But we can all agree that it is vital to provide information about research methods so that we are able to assess the quality of the research we consume.

    Laurence Kotler-Berkowitz
    Director, Berman Jewish DataBank
    The Jewish Federations of North America

  3. Jonathon Ament says

    March 1, 2017 at 4:34 am

    This is a good start. Here are a couple of other items:

    –How was the survey administered?

    There are a plethora of options (mail, telephone, on-line, to name a few) and within these options an even wider selection. Each method comes with strengths and limitations.

    One example from last year’s elections–the 538 site did a great job of detailing every poll and rating each firm on a variety of methodological considerations. There were certain firms, for example, that in their telephone surveys only called land line telephones. These polls ended up skewed far more strongly for Trump, since older, white and more rural Americans are overrepresented among landline users, while younger, urban and more racially Americans who are overrepresented among cell-phone only users (and who trend Democratic) were not included in the sample.

    –Who is funding this research, and what is the relationship between the researcher and the funder?

    Many organizations that conduct research have a political agenda of one sort or another. This has the potential to impact every aspect of the research process, including how a topic is framed, what issues are not discussed, the ordering of questions, the terms that are used to describe the what or who that is being studied, and how findings and analyses are presented. Even a cursory glance of the questionnaires designed by the media outlets across the political spectrum reveal important differences here. Potential ethical and political minefields abound, and we in the Jewish community need to be acutely conscious of these issues as well.

Primary Sidebar

Join The Conversation

What's the best way to follow important issues affecting the Jewish philanthropic world? Our Daily Update keeps you on top of the latest news, trends and opinions shaping the landscape, providing an invaluable source for inspiration and learning.
Sign Up Now
For Email Marketing you can trust.

Continue The Conversation

  • Email
  • Facebook
  • RSS
  • Twitter

Recent Comments

  • Bruce Powell on An Invitation To Transparency: Reflections on an Open Salary Spreadsheet
  • Sara Rigler on Announcement: Catherine Reed named CEO of American Friends of Magen David Adom
  • Donna Burkat on The Blessings in 2020’s Losses
  • swindmueller on Where Do We Go From Here?
    Reflections On 2021
    A Jewish Response to These Uncertain Times
  • Alan Henkin on Where Do We Go From Here?
    Reflections On 2021
    A Jewish Response to These Uncertain Times

Most Read Recent Posts

  • Jewish Agency Accuses Evangelical Contractors of “Numerous Violations” but Denies They Evangelized New Immigrants
  • Breaking: Birthright Israel & Onward Israel Seek to Join Forces to Strengthen Jewish Diaspora Ties with Israel
  • An Invitation To Transparency: Reflections on an Open Salary Spreadsheet
  • Why One Zoom Class Has Generated a Following
  • The Blessings in 2020’s Losses

Categories

The Way Back Machine

Footer

What We Do

eJewish Philanthropy highlights news, resources and thought pieces on issues facing our Jewish philanthropic world in order to create dialogue and advance the conversation. Learn more.

Top 40 Philanthropy Blogs, Websites & Influencers in 2020

Copyright © 2021 · eJewish Philanthropy · All Rights Reserved