DIY High Holiday Measurement
By Daniel Olson
As a rabbi’s spouse, I am all too aware of the great loss these months of pandemic have brought to our communities: the loss of loved ones, the loss of in-person schooling, the loss of jobs, the loss of time together in community. Leaders of Jewish organizations have worked hard, under great stress themselves, to quickly adapt to the evolving limitations and opportunities posed by this moment. As a researcher and evaluator of Jewish innovation, I am certain that your work is worth learning more about.
Working on new projects in a fast-changing environment, leaders frequently are challenged to take the time to learn from their new work. Jewish leaders may neglect or feel ill-equipped to collect and analyze data in a useful way. The shift to online programming by many religious institutions makes it easier and quicker to engage in these data-driven learning processes.
This short guide offers some easy tips for thinking about what kinds of questions to ask and how to use tech to seek out answers to those questions in real time.
Measure to Learn
Only spend time measuring what you care to know more about.
Assume that members of your community are busy and asking them to fill out a too-long questionnaire is counterproductive. It is important that their experience of data collection be positive, so that the data will be useful in the end. That means you should have a clear set of goals in mind for your offerings and programs, and only ask questions that help you learn about the achievement of those goals.
For example, if you want to elicit a certain attitude among your community, like a motivation to seek out forgiveness or a commitment to working on improving an important social issue, then ask about that specifically. Don’t waste time asking more generic questions about satisfaction with individual elements of the service or program. Some of these attitudes only play out in the long-term, so follow up with congregants or participants throughout 5781 to see if they maintain these attitudes.
If you want people to feel a connection to the community during these socially distant high holidays, then ask them which moments helped them experience a sense of togetherness. This question will indicate if you have been effective, rather than questions that only get at the individual level, like the number of attendees.
Invite participants to share information that you can actually do something about.
Members may desperately want a synagogue to re-open for regular, big, maskless, in-person services, but that desire may not be fulfilled until a safe and reliable vaccine is developed. Jewish organizations accomplish a lot, but they can’t do that. So it doesn’t help you to ask if people want this!
Instead, consider asking a short and specific list of questions about reactions to parts of your offerings that you suspect may need some tweaking or attention. For the high holidays, you may want to know about how your audience reacts differently to live streamed versus pre-recorded material. Knowing which mode comes closer to accomplishing your goals for your community can help you decide what to offer in the future.
Even if your organization has a history of evaluation, you will not be able to compare this year’s experiences to previous ones. There is no baseline of robust data for Jewish life in pandemic. So the task is not to measure yourself against some standard, but instead to learn from scratch what works for your community under these conditions.
You can learn about these reactions using traditional modes of data collection, like surveys and conversations with specific people. These tools remain the best options for communities that do not frequently use Zoom or other online platforms for their programming, including because of Shabbat and holiday observance.
But just as the possibilities of programming have changed as a result of the shift to virtual, so too do the possibilities of data collection.
Real-time feedback on Zoom
Zoom, the ubiquitous software of choice during these months of pandemic, can be used to gather real-time information from call participants in a number of different ways.
The Chat
Take advantage of the “Chat” feature to invite questions or reflections. If you want your community to feel a particular way at a certain point in your program or service, use the chat to ask how they feel in the moment. If you intend for your sermon to inspire action of some kind, ask participants to share something they plan to do after listening.
In the moment, you should validate any and all responses that come through the Zoom chat, but pay attention to unexpected feelings or takeaways. That data can help you adjust in real time as the program goes on and help you plan for future offerings.
Instant Polling
Another prime Zoom feature to use, for instant quantitative data, is polling. Sending out a multiple-choice poll to your participants can enhance the interactivity and even fun of a Zoom gathering – you could have congregants vote on which tune to use for a certain prayer, or to choose which theme the rabbi should focus on as she introduces the parasha – but can also give you solid information about how people are experiencing your offering.
Zoom polling offers more than just fun. You can also use it to learn about your audience. If a sermon or class presents multiple approaches to grapple with a tough issue, you can send out a poll to find out which approach resonates most strongly with participants. I would recommend including an “I’m not sure” option for such questions, because that tells you something important too.
If you decide that gathering information about satisfaction is important, better to do it near the end of a Zoom call while you still have your audience’s attention rather than sending out an after the fact link.
YouTube Analytics and Facebook Insights
The built-in measurement tools on YouTube and Facebook can tell you more than just the total number of views.
YouTube Analytics
YouTube Analytics, for example, can show audience retention stats, that is the percentage of total viewers who were watching the video at a certain point. This information can help you locate precisely where you should make future videos more engaging, to try and retain a greater share of your audience.
For example, I posted a video for Tisha B’Av to Youtube, a rewrite of the kinnah Eli Tziyon about the pandemic. It got over 3,000 views which, while not huge, was a respectable number. Looking more closely at the data, I learned that a good portion of those viewing only watched the first minute of the six minute video.
Youtube helped put this finding into perspective by comparing the audience retention rate to other six minute videos on the site. My audience retention was above that average throughout its length, but this extra insight motivates me to find ways to make the beginnings of future videos more engaging.
Facebook Insights
Facebook Insights can also display audience retention, but offers even more data. Facebook will share demographic information (age, gender, geography) of post viewers and the different reactions people have (like, love, angry, sad, etc.), offering some insight into the emotional impact of the content.
Keep in mind that the implicit goal of these corporate measurement tools is to boost traffic to YouTube and Facebook through your channel or page.
While you might also prioritize engaging more viewers for a longer amount of time, that is likely not your only goal for the High Holidays and beyond. So you will need to supplement with other tools more customized to your specific goals and setting.
Don’t be afraid to try
It is understandable why professionals may resist prioritizing this kind of real-time learning about their service leading and teaching. It does take some time to develop survey and poll questions, and other measurement tools, time that could be going to other essential community functions. Allowing congregants to vote on which theme the rabbi should talk about means she needs to prepare two themes and only share one. Seeing reactions come in real-time could also be unnerving to staff who may have previous bad experiences of community members with strong or non-representative opinions.
The work of Jewish community is complicated and only getting more challenging. We are doing so much work to try new approaches, and measuring our impact against our goals will enable us to focus our work on what we find is actually effective. The only way to navigate these uncharted waters is to do our best to map them as best we can. The tools we’ve already been using to get through this moment can help here too.
Daniel Olson recently completed his PhD in Education and Jewish Studies from NYU. He lives in Port Chester, New York. He is an evaluation consultant for the Center for Rabbinic Innovation.