In the summer of 2006, I was set on pitching a new magazine in what I was told would be a four-day global contest for funding held in Jerusalem. The magazine, of course, was PresenTense, an effort a few friends of mine and I started in December of 2005 in Morningside Heights of Manhattan. After months of unsuccessful attempts to raise $10,000 for the first print run and distribution, we ended up printing it ourselves. For $5,000 that a friend and mentor gave me, and around $5,000 that lived on as a (growing) debt on our credit cards, we printed 1000 copies of what we called ‘Issue Zero’ and tried to leverage these copies to gain subscribers and advertisers. We saw this conference as our chance to shine. It brought 120 innovators from around the world and was structured around a series of pitches and workshops supposedly culminating in a grand prize. We were determined to win that prize and scale PresenTense. The conference’s very name convinced us that we were a shoe-in: it was called ROI120, indicating the Jewish community’s desire to get a return on its investment, and was, at the time, a joint effort between the Schusterman Foundation, Taglit-birthright israel, and the Israel Democracy Institute.
We did not win the contest, and from what I understand there was no cash prize at the end. But thanks to the conference, many of the relationships that built PresenTense were formed or strengthened. To go through the list of individuals who attended ROI120 that first year and now are leading major efforts in the Jewish community would be to tempt the consequences of forgetting a name, so I’ll leave others to list them. But what is clear to me is that ROI120 of 2006 concentrated a surge of creativity and innovation that impacted a certain strata of the global Jewish community, and will impact the Jewish community for years to come.
Five years later, what’s changed? A lot. And very little. Some of the larger, more clearly identifiable elements can be sorted out: out of ROI120 came Schusterman’s ROI, which celebrated its sixth summer conference this June bringing 150 young adults from around the world to a four day conference; following ROI120, PresenTense grew from a magazine to a global network of twelve community entrepreneur accelerators launching 155 new social ventures thanks to the efforts of thousands of local volunteers; following ROI120, the Forest Foundation’s Moishe House project received sufficient funding to grow independent, supporting dozens of apartments for young Jews across the world to help them host local events; following ROI the Jewschool and Jewlicious led blogosphere rose and fell in influence, and comparable programs such as the Professional Leadership Project, Heeb, Guilt & Pleasure and others rose and fell in prominence (and out of existence). In other words, if you look to the top of the waves, some of the main actors caught the wave, others did not.
But beneath the surface, I have no clue what has happened. And that is because there isn’t enough data out there to make any good comparison to know whether the Jewish People are any better, or worse, than they were five years ago.
For example, I can tell you that ROI brought together approximately 120 young adults together five times over five years, and 150 one year, to total approximately 650 (adjusting for repeat-attenders) individuals who networked together in Israel over a four day stretch. I can tell you that PresenTense has produced 14 separate issues of PresenTense Magazine due to the efforts of approximately 800 volunteers and the interest of tens of thousands of readers, and run 12 fellowship programs that have engaged approximately 80 young professional volunteers per program and helped 160 community entrepreneurs launch 155 ventures. But I can’t compare the impact of the two in any meaningful way, nor can I find enough data on the former efforts of PLP, KolDor, Guilt & Pleasure, and others, to provide any meaningful conclusions as to whether our efforts have been worthwhile. There is no way to do a cost-benefit analysis, no way for a responsible manager to decide whether the efforts she is working on is “worth it,” or whether she should adjust operations to increase the impact of the dollars under her disposal.
There is a reason there isn’t good data: the capital markets (i.e. philanthropists and foundations) do not demand it, and the organizations, therefore, cannot justify allocating scarce resources to provide it.
As a result, we’re almost a decade since the Jewish community become obsessed with ‘young adults,’ with the ‘un-movement,’ since tens of millions of dollars flowed to organizations such as Heeb, JDub, KolDor, PLP, and so on, and we have absolutely no clue whether we’re better or worse off than we were before.
This irrational behavior is hurting, and not helping, the Jewish People. This is similar to the situation Rob Eshman describes in his article, Count-Less, concerning the lack of data for Los Angeles, “What we are working off are guestimates that are, at best, almost two decades old and, at worst, self-serving and self-aggrandizing.” This lack of information – about the size of the population we’re addressing, about the impact our efforts have had on the population, about how our efforts line up against each other – have caused those excellent professionals among us to get fed up seeing philanthropic investment flow to silver tongued efforts with no base of sustainable support.
If we want the next five years to improve on the previous ones, we need that data, and we need it now. We need to know, for example, whether the dollars spent per participant on birthright israel have the same long-term effect as the dollars spent per (non-birthright alum) PresenTense fellow. Or how Paidea’s JPropel ranks versus the efforts of KolDor. This is essentially a Jewish requirement: we are commanded to have unified weights and measures, and yet our community has not taken the small step of setting up standards and goals, metrics and measurements, done by any political party apparatus who is in it to win it. If we want to grow the members of the Jewish People as bad as the Republicans or Democrats want to grow their party base, we need to measure whether our efforts have any effect.
Five years have passed, and many of the individuals I met at that first ROI120 remain my friends. Five years have passed, and many of the community entrepreneur fellows I met summer after summer, and year after year through PresenTense, remain my heroes. But five years have passed, and with them the blood, sweat and tears of dozens of young community entrepreneurs trying to build a better future for our People, and I cannot tell you if we are any closer to our goal than when we started. We owe it to those activists who toil day in and day out to build a rational capital market, to reward based on measured impact, and to invest in efforts that on comparison are furthering the goals of our People: strengthening our core, building ties of mutual responsibility, and extending the impact of our values through fixing the ills of our world.
Ariel Beery is the co-founder and director of the PresenTense Group.
Send to Kindle
Ariel — Am I wrong to assume that each venture who receives funding from a grant or foundation, must include outcomes (and those outcomes are measureable). So, you mean, the data are there, but just not being shared?
If that is what you mean, this would point to needing a open-repository for this data. Easy enough, especially since non-profits are public organizations, they could be persuaded that sharing of this information is a public duty.
eJewish Philanthropy could house the repository.
Or are you saying, in this day and age, ventures have outcomes that are either unmeasurable or go without being measured?
I hope that is not what you mean.
Great observations, Ariel! HOwever, you have elided the primary difficulty with Jewish ROI. The key to ROI is figuring out what the accurate measurement of success is: it used to be “membership in Jewish communal organization–shul, JCC, Federation” but that no longer is the yardstick for US Jews. What, then does success really look like? How do we measure engagement in a “post-organizational” space? When we know the answer, we will be able to do apples to apples ROI for incubators and new ventures.
The same kind of question always dogs media, as you know. Success can be measured quantitatively by # of audience/click-thrus, but that does not measure impact. At the Media Consortium, we are launching an experiment to measure the impact of memes through digital space–when we get our data, in 2013, I will definitely report back to the Jewish community!
This is an important topic and I thought I’d chime in with a small example from the field that demonstrates some of the complexity here, as well as how seriously we take this topic.
I’ve been trying to write an evaluation plan for G-dcast all year with support from the Joshua Venture Group and some serious expert guidance from impact/effectiveness strategic consultants Wendy Rosov and Tobin Belzer. This has been a hard nut to crack. Certainly, I can share our web metrics easily, but I agree with the other commenters that this does not tell much of a story. Who cares how many people watched one of our YouTube videos? That is an interesting piece of data, but more interesting is how many people learned something from their viewing, or changed an attitude. Also not revealed by simple web metrics are findings like how many schools have altered their pedagogy based on our curricular materials, how many supplementary school teachers feel more confident teaching the parsha because of the availability of our materials, and how many Bnei Mitzvah kids feel more excited about crafting a creative drash after watching our materials.
Our mission is to raise basic Jewish literacy. I unfortunately can’t measure this very easily, especially on our budget.
So for now, we’re going to settle for a somewhat squishier type of metric as far as measuring our impact. For instance, we will be asking teachers (which are only one small segment of our large “market”) to what extent they perceive that their students are more knowledgable or excited about/willing to read Torah stories after watching G-dcast episodes.
This is just a small example. Hopefully this data will be of interest to the community. Along with many other Joshua Venture fellows, we’ll be sharing a report later this year as our study progresses.
Sarah. I like your approach to taking a small sample of users and running with it. You are beginning to describe an applied experiment to evaluate the effectiveness of a “treatment” — G-d Cast, being the treatment.
Why not survey the students, too? Pre and Post.
Add a little more control and you’ll have much stronger finding. Instead of only comparing G-d Cast to nothing (on the dependent variables of knowledge and excitement), compare G-d Cast videos to some other traditional Jewish video.
Doing these kinds of applied experiments to demonstrate the efficacy of your educational method is absolutely in-line with educational research.
It’s ok that this is only with a segment of your intended audience. If G-d Cast videos result in more knowledge and excitement in kids in the classroom, is it realistic to generalize this finding to children at home? Or to adults. Probably. Or repeat your experiments on different subject populations.
There are said to be lies, damn lies and statistics. Some people ‘torture’ the statistics to demonstrate that they ‘prove’ something. Defining what that ‘something’ is the question that funders and entrepreneurs have been assiduously avoiding. I take the position that non-Orthodox Judaism is no longer as appealing. For most secular Jews, however defined, Orthodoxy is not an alternative. We have a dropout rate that is alarming by any measure and a business model for the Jewish Community that no longer makes sense. Individual efforts that affect hundreds of Jews is nice, but unless it can impact hundreds of thousands of Jews it may not be worth funding by the relative handful of available funders. We need more than putting one foot in front of the other; we need to recreate a Judaism that appeals to a population that has left the ghetto far behind. It needs to be a Judaism that is robust and provides meaning in such a way that individual Jews can feel a reason t be part of the Jewish People. It needs to be centered on Jewish Values and how the Jewish People an be a light unto the Nations. I suggest that it may simply be too difficult to imagine and, if so, we will be no further along in the next five years and continuing to go down the slippery slope.
Thank you all for your comments. Howard, I fully agree with Disraeli in that the analysis of statistics is a dangerous, dangerous game. But without the data we can’t even start to analyze! As it is now, we can’t create growth markers so as to know what works and for whom, and then to follow-up that quantitative tracking with asking those people why.
That is not to say that small efforts aren’t important: the Pharisaic movement was a small group of “Chaverim” that took hundreds of years to conquer Jewish theological and philosophical thought. A ‘philanthropist’ in their early days couldn’t know that they’d win – but they could say ‘we’ve grown every year, year after year, while our rivales the Tzadokites have halved, and if you want to support our growth you should know that X shekel will yield Y new Chaverim.”
That’s the key of creating clearly defined metrics: the ability for activists and resource providers to have a conversation grounded in facts and not sales spin. At the end of the year, if the number of Chaverim didn’t grow, the philanthropist could ask why, could analyze, and could reassess. But without the data it’s all one word vs another — leaving decisions of resource allocation open to innuendo and gossip.
Jo Ellen, I respectfully disagree with your comments. The politically aligned press has done a lot of work on this, especially from the perspective of keeping certain organs of ideology alive. There can be inference of influence through public polling, cohort studies and baseline approach.
But your comment leads me to a larger point, one you didn’t make but I’d like to address. For some reason, we Jewish professionals have added this “impact” dimension and argued for years that the ‘impact’ is not measurable and, like porn, one only can know it when one sees it. That just isn’t true. Consumers read and buy something because it speaks to them, and what speaks to them impacts them. The reason Heeb is no longer in print is because Heeb didn’t speak to enough people who would pay the price of a Starbucks Vente to read their fare. Same with Zeek: there simply have not been enough people buying what Zeek was selling to justify its operations as it was. So it moved on to become an online section of Jewcy, then the Forward.
Same with PresenTense Magazine, by the way. We cannot justify printing the magazine if it only serves as a magazine. It simply did not get the subscriptions, and without the requisite number of readers, we didn’t get the advertising either. Which means we failed to attract the sufficient interest of the crowd. We didn’t close down our print edition, however, because we realized that the magazine is one of our best community engagement mechanisms (over 80 volunteers prepare each issue, and each of those individuals is a node on a larger network). So the magazine remains a loss-leader for our larger operation which, as of now, generates sufficient value to attract paying clients.
To say ‘we’re okay but the world is crazy’ underestimates the intelligence of the world. There are smart people out there. And without knowing what grabs them, moves them, and gets them interested we’re only digging ourselves deeper into an echo chamber.
I’m not saying that results can exist over night. Every venture, for-profit or for-social impact takes an investment of up-front capital to build its base, develop a product, test that product and iterate. PresenTense still relies on community investment of approximately 30% to get to its break-even point where the income earned will cover the cost of operations. So does GroupOn. But investors, both for-profit and philanthropic, should be searching for a hockey-puck curve of interest from the community, and for deeper analysis of a subsection of that community.
Which brings me to Sarah’s point about G-DCast. Sarah, I think you absolutely can measure your impact through the number of views and you can absolutely dig deeper into the impact through surveys that can be as engaging as your online work. Pre-tests embedded into YouTube’s A B testing function can work quite fine, as well as hosted video solutions that will invite people to answer a few questions before seeing the video. That’s almost free to do. And post-testing can happen the same way. Beyond that, however, it seems odd that you can’t say “if people like what they saw, they’ll forward.” That’s what makes a video viral. I forwarded ‘Epic Rap Battles of History’ to over a dozen friends, which is why they have millions of hits. G-Dcast is dealing with a smaller market (which we have no data about, like I wrote in my article) but all that means is that you should be targeting 40K views and not 40M.
The key here is the same point I hoped to have made in the article: without knowing who our population is, and how big it is, and what it thinks, we can’t set realistic targets and measure our impact. Forget all of the tools and techniques we’ll need to extrapolate the right data — we don’t have ANY data so we can’t draw any real conclusions. Our work remains in the frame of the theological, which is to say only God knows what we’ve achieved. That shouldn’t be good enough. We have some of the smartest brains in the world involved in trying to develop a foundation for the Jewish People’s future. Let’s do out future the favor of defining what that generally means in numbers and work towards those goals.
(And I have another article that I’ve been sitting on that defines ‘how’ further, Jennie, and how org professionals and philanthropists can work towards that as partners).
Hi Ariel,
I work in the politically aligned press, and I can tell you right now that no one in the progressive media world is satisfied that we have a good way of measuring impact. Let’s dissect that word.
I agree that basic metrics tell us whether a particular audience cares about and to that extent is influenced by a publication–there is no doubt that the print Zeek did not sufficiently appeal to an audience large enough to sustain it. However, audience alone does not tell the story about how ideas travel through a culture. Ramparts was one of the most influential magazines in American history, yet closed down due to lack of paying subscribers. Rolling Stone in its hey-day influenced far more people than ever read any of its pages.
When Zeek published an issue on Food in Fall 2006, did that pave the way for the recent focus on eco-kashrut and the spiritual sustenance we derive from sustainble food? When we published some of the first pieces on gay Jews, did that pave the way for the growing acceptance of LGBTQ Jews in our community? I don’t know–all I know is that one Orthodx man wrestling with his sexual identity wrote me that he was going to commit suicide until he read a Zeek article by a gay Orthodox Jew.
You can’t trace cultural influence through marketing tools like polling or cohort studies because unlike the man who wrote to me, many of those whose ideas or lives have been changed by media may not have actually read/viewed the media directly. They haven’t bought the brand–they’ve bought into the ideas the brand was selling.
In short, the toolkit used by marketers (polling, cohorts, etc) can tell us about the use of our product, but these tools still can’t tell us about the reach of our product, because cultural production is different in kind from cereal or cars.
Those of you who know me know that I’m not the metrics person, I’m the content/language/relationship development person. And that’s why I want to point out one of the impacts that I think may be difficult to map or measure, but is also very important.
One of the powerful things about being in today’s innovation world is the ability to, in the words of my friend and colleague, ROI’s Justin Korda, “connect and create” with other people – those who helm innovative projects and those forward-thinking individuals who are “withinnovating” – working to innovate from within other organizations. Sometimes “connect” and “create” happen together and sometimes years apart. But they are at the core of what I like to call the relationship economy: people meeting people, and forming/expanding networks and relationships; creating an organic, integrated series of relationships with endless, but in some cases, dormant potential.
As someone who has been connected to at least six (but who’s counting?) of the above projects, I also want to clarify that this is not instead of a call for better understanding of community needs or for metrics that are less “squishy” and could help us better understand the impact of individual programs, but in addition to it. The relationship economy is a formidable source of power, creativity and strength, and I’m excited every day about the possibilities that might emerge as connections lead to creativity and creation.
Hi Jo Ellen,
You have much more recent experience than I do in the political press. What I do remember is a debate inside The New Republic, when I was trying to help Marty build younger readership, having to do with metrics and the ability of the press to know whether it needs to call it quits or invest in the future.
At the time, Mother Jones and the Nation were doing very well (the Nation was one of the more profitable magazines out there), and TNR was losing a ton of money. At one point the powers that be just had to accept that American political opinion changed – and TNR was sold off.
I hear you, fully, on the topic of the intangible. There are many failed efforts that changed the world for the better or the worst. My point is that we must affirm that individual people in their every day choices can indicate whether a magazine or publication or project or product is valuable enough to survive.
A great example of this is the Huffington Post, which built itself on volunteer labor, created massive quality, and impacted the world. So far, it’s been doing fine, and impacting the world very clearly. DailyKos, in its day, too. And more examples I am sure can be found from the right (I’m just not familiar with the right).
Without trusting the size of our readership or customer base, we’ll be constantly forced to tell touching stories about that one individual whose life changed because of a program. That individual is certainly important, and every life is a world in-and-of itself. But to imagine that our particular way of doing things is right despite the fact that not many of our peers are willing to spend their time and energy on supporting us is just narcissistic. If people value what we do, they should pay for it in time or energy. If people value the groups we create, the relationships we enable, they should enable its sustainability. If they don’t, we can only suppose they don’t value it enough.
The sooner we learn from metrics the sooner we can catch our mistakes. Like PLP (the Professional Leadership Project), it was all good until the money from an older disconnected individual stopped flowing. And once that philanthropic gentleman passed, PLP suddenly was gone. Wouldn’t it have been better to know that the PLP ‘community’ didn’t value PLP enough to sustain it while it was alive, so that money could have gone elsewhere?
If it isn’t valuable enough to survive, and the human cost of putting it together does not equal or surpass the human value that it creates, well, we should move on and let projects die.
No disagreement from me on your point here Ariel–if a particular project is not supported by a community, it should die. I personally have supervised the death of two projects, and always have that option in mind. My point was only that impact is not the same as viability.
Just to continue the thought–one problem of a capitalist society is that we often see projects thrive that are viable but don’t have a mission-driven impact: viz, reality tv or, more sadly, some of our major infotainment newscasts. As a fan of “fast food” genres like mysteries and sci-fi fantasy, I get the pleasure of immediate gratification–entertainment is a real benefit of cultural production and should not be dismissed. At the same time, cultural production holds out the promise of broadening an individual’s and a society’s notion of the possible.
The history of the West indicates that this kind of cultural production often must be supported, at least in its early life, by patrons rather than the public–Mother Jones and the Nation, to take your examples, began as projects supported by very deep pockets donors. Universities have now taken on that role for most of our literary production, providing sinecures to writers to give them the freedom to write without relying on an audience for their livelihood.
The Jewish world sorely needs this kind of cultural production right now, as I believe we are in real danger of losing a living Judaism, a Judaism that can adapt and grow in a postmodern era. We need new ideas. We need ideas driven by products that work (e.g. minyanim, eco-kosher meat vendors) and we need big abstract mind-blowing ideas of the kind that show up in cultural products. Jewish philanthropists must understand that this latter kind of cultural production does not always, at least at first, pay dividends in cash or audience but in ideas.
If the ideas are strong enough, the project should become stickier–it should attract attention; it should attract volunteers; as I said in my first sentence, if a project doesn’t attract a community that values it, it should die. I underscore your point that the attraction may be in time and energy and not cash–cultural products may NEVER attract a large audience or cash.
One last thought: what’s hard to know if how long to give a project to develop before assessing its success or failure by these more qualitative metrics. We know that right-wing think tanks nurtured cultural products and individuals for over a decade– through the 1970s–before they started to see results. The U.S. civil rights movement likewise incubated for over a decade before Rosa Parks got on that bus. Most funders give a project 2-3 years. I am sure that is not long enough for a cultural product of the type I am discussing to bloom. Is 5 years? 10 years would seem more than sufficient.
I don’t have the answer, but this conversation you have initiated is the one we must have in the Jewish world, and frankly, in the United States, if both of our cultures are to move forward.
Best,
Jo Ellen