Evaluation: Who Decides?

A recent report by the James Irvine Foundation finds that “Trustees care deeply about impact. Understanding results is part of their fiduciary duty. As foundations strive to improve performance, advance accountability and share knowledge, their desire for evaluation – reliable data on organizational effectiveness – grows.” Just as foundation staff want to measure their grantees’ performance, trustees in turn want to make sure that their foundation’s strategies are productive.

It is certainly in everyone’s interest to spend money wisely. But when it comes to evaluating the results, it’s all too easy to assume that methodologies and numbers are objective and that strategic success can be quantified. It’s true that you can count students enrolled at a school, hot meals served at a soup kitchen, or attendance at an art exhibition. It’s not so easy to measure how good the teachers are, the overall well-being of the clients of the soup kitchen, or the quality of the art on display.

What’s more, collecting and analyzing data can be expensive. Meanwhile, Jumpstart’s 2008 Survey of New Jewish Organizations reported that “few organizations have large numbers of participants and constituents; smaller, more intimate organizations are the norm.” Spending money on professional evaluators is simply beyond the means of many of these startups, and of many other nonprofits as well.

The Irvine Foundation’s Evaluation Kit for Trustees recognizes this and counsels foundation trustees: “Cultivate a pragmatic attitude toward data collection. Rigorous studies that prove the impact attributable to a foundation grant are costly and time-consuming. Consider other kinds of data to inform the board along the way. Discuss which kinds of information would be ‘reliable enough’ to support a decision.”

Evaluation professionals, understandably enough, would prefer to see the money spent on data. Ken Berger, the president of Charity Navigator recently turned aside an objection about the cost of tracking data by suggesting “One possible solution is to explore collaborations and mergers as a way to become more financially strong and large enough to have a more diverse base of staff.” In other words, nonprofits should merge if they can’t afford to produce enough data! I wonder if most funders believe that numbers are so important that nonprofits ought to give up their independent existence in order to generate them.

On his blog Berger repeatedly urges relying on professionals. “Read up on the matter and engage the experts. There has been a tremendous amount of work with at-risk youth and outcome measurement,” he tells a representative of the Dare to Dream Children’s Foundation. “Constituency voice tools are one way to measure museums,” he advises a museum board member. His solutions, in short, are technocratic.

Harvard Business School professor Alnoor Ebrahim has studied accountability in public organizations for some years. He concludes that it can be seen as “a social phenomenon, in which its actual impacts are a result of relationships of power and interplays among actors.” He continues, “the social nature of accountability brings into question the metrics used for performance assessment. It leads to questions such as: What kinds of knowledge are considered legitimate or valid in designing and implementing accountability and performance systems? What kinds of information, knowledge, and expertise are devalued? … Metrics thus become linked to concerns about power: why certain metrics are selected over others, who decides what metrics are important, who collects the data, who interprets it, and who decides how to use it. … [I]t suggests that the reality of accountability is ambiguous.”

That’s a crucial point. Choosing criteria and evaluators is neither neutral nor unbiased, and the evaluators hold a lot of power. Think of the committee that determines whether a film is rated R or NC-17, or the board that decides whether a textbook is approved for schools in Texas. As formal, quantitative evaluation becomes more widespread, there’s a real danger that assessment specialists will exert more influence over spending decisions than nonprofit professionals, especially in areas like education or culture. It’s hard to see how that would be good for the wider community.

Bob Goldfarb, an alumnus of Harvard Business School, is president of the Center for Jewish Culture and Creativity. A regular contributor to eJewishPhilanthropy, Bob lives in Jerusalem.