‘Since this is media, how can you know that you’re really having an impact? What proof do you have that video can build Jewish identity or literacy?‘

[This article is part 8 of the series Continuing Conversations on Leveraging Educational Technology to Advance Jewish Learning. The series is a project of Jewish Funders Network, the Jim Joseph Foundation, and the William Davidson Foundation. For an in-depth look at opportunities in Jewish Ed Tech and digital engagement, read Smart Money: Recommendations for an Educational Technology and Digital Engagement Investment Strategy. Later this year, Jewish Funders Network will launch a new website to help advance the field of Jewish educational technology.]

By Sarah Lefton

BimBam (formerly G-dcast) is a Jewish media studio. Our creative team has worked on over 300 short videos and apps, and we have big league experience from Apple, Pixar, The New York Times, etc.

Usually, people find us through our work – they don’t ask us too many creative questions beyond, “Can I do a part in a video?” That’s because they or their kids already love the programs, and it’s easy to see our track record. Our Judaism 101 and early childhood education videos have clear and easily shared metrics – high viewership numbers, great audience retention curves and accurate aim at the demographics we’re targeting.

What we do get asked routinely is, “Since this is media, how can you know that you’re really having an impact? What proof do you have that video can build Jewish identity or literacy?”

Great questions. What I want to offer is a perspective that we’ve found to be true: well-designed Jewish media programs that are informed by best practices from secular educational media are as effective as their peers. PBS Kids shows (e.g., Daniel Tiger’s Neighborhood, Peg + Cat, Sid the Science Guy), Sesame Workshop programs, and Reading Rainbow have been studied for decades by leading researchers using sample sets comprised of thousands of children and their parents. Interactive educational programs in the app marketplaces are, to some extent, being put under the lens while in development or afterwards.

Funders are advised to read this research – which they will find fascinating not only as evidence supporting certain types of programming but also as former children themselves. Research-based shows are studied for proof of literacy building, numeracy support and social emotional development. They are tested in both formal controlled settings and informally, in homes.

There are protocols, well-established, to choosing groups of respondents to study and there are protocols for discovering not only learning changes but also affect and engagement. For instance, guided by Stanford PhD researchers who work routinely on PBS properties, we tested early Shaboom scripts on eight 3, 4, and 5 year old children – two boys and two girls of each age – in their homes, beside their parents. It did not matter that they all lived in the Bay Area, something that we’d thought would be a factor. We got all of the information we needed to learn from these 12 children in order to improve our pilot episode, through asking establishing questions, observing measures of engagement during viewing (such as toe tapping, pointing at the screen and looking at a parent), and answering questions post-viewing. This formative evaluation allowed us to improve our pilot script – for instance, we added more Hebrew vocabulary words, increased the use of musical chants and changed the name of the show based on feedback we observed and heard from children in this phase.

After producing a finished animated pilot, we did another round of testing, conducted entirely blind to us by Ph.D. researchers who watched the program with dyads of parents and children, and then conducted play testing afterwards to measure what social skills children had developed through viewing the shows. (Children performed little actions with dolls to demonstrate welcoming guests, for instance.) This summative evaluation showed that in fact children were learning social skills and assimilating Hebrew vocabulary for those actions.

We were fortunate to receive a large grant from the Peleh Fund which made it possible for us to bake this crucial, but costly, round of evaluation into our development of Shaboom. But it is unrealistic to expect that any small Jewish nonprofit will produce comparable research on their evaluation budgets, or that funders commissioning Jewish work from secular shops will know how to conduct such research. We learned a lot from our experience on Shaboom, enough that we can – to a great extent – cobble together our own in house studies with our own staff. However, this comes at great cost to staff productivity in a small (5-person) organization – capacity that could be improved either through a major evaluation grant or a full time staff person focused on learning measurement.

Given substantial communal buy-in, I recommend going bigger: establishing a center for Jewish media research staffed with Ph.D. media and learning science researchers who are trained to do this sort of work and dedicated to it, full time. This would show serious intent to invest in effective, high quality media work by the funder community, and would up the game of all of us media producers substantially by having partners for evaluation.

Such an investment might be a pipe dream. No matter: I believe that we have the research that we need from existing educational media companies. The community of funders should hold Jewish media producers accountable to these studies – and show their fluency in them – rather than asking for original research. Playtesting, and viewer testing, are serious endeavors and cannot be thrown together in the “free time” of scrappy small studios or independent artists.

When you work with a well-versed studio or artist, you will find that they are familiar with this research and can tell you, without hesitation, that it builds Jewish identity and interest in practices – from baking challah to singing brachot to trying out bikur cholim. At BimBam we have evidence that our programs and apps have produced these effects – but we also went into the work confident, because we followed research best practices.

So, familiarize yourselves with the research. Browse the archives of the Joan Ganz Cooney Center, of Commonsense Research, the Fred Rogers Center, the Pew Research Center, the American Psychological Association – read up on the work at the university centers doing work in this area (Annenberg at UPenn, the Children’s Digital Media Center at UCLA, Center on Media and Human Development at Northwestern, Learning Sciences and Technology Design at Stanford, Technology, Innovation, and Education Program at Harvard, etc.), and consider picking up some journals that publish new studies on media education research. If you’re really interested in this space, you’ll find them fascinating – and helpful as you partner with Jewish media producers.

Sarah Lefton is the founder and Director of BimBam, formerly G-dcast. She began working in interactive media in the last millenium, at NYU’s Interactive Telecommunications Program (ITP). She went on to produce projects for The New York Times on the Web, the Village Voice, Princess Cruises and several children’s toy brands. Her social media “Save Jericho” project for NutsOnline.com garnered national news attention.

Taking an abrupt turn for the less corporate, Sarah joined Northern California’s independent Jewish summer camp, Camp Tawonga, as their Marketing Director for four years, learning about Jewish outreach and wilderness. Inspired by its Yosemite location, she designed the infamous YO SEMITE tee shirt, and launched her first Jewish entrepreneurial project: Jewish Fashion Conspiracy. Sarah is a past President of San Francisco’s pluralist Mission Minyan, and board member of the San Francisco JCC.

Sarah is a recipient of the 2012 Pomegranate Prize for exceptional young Jewish educators. She was named one of the Forward 50 most influential Jews of 2009, and is a recipient of the Joshua Venture Group fellowship for Jewish social entrepreneurs.