WolfBrown: On Our Minds

Posts tagged ‘evaluation’

How Trustworthy are Charities?

February 17th, 2011

Given that much of WolfBrown’s work helps organizations demonstrate accountability through rigorous evaluation of their programs, I found a recent article in The Economist quite interesting. The article states that “70% of Americans trust non-profit outfits more than government or business to ‘address some of the most pressing issues of our time.”

A survey by the Center on Philanthropy of affluent households found that over 36% had a “great deal of confidence” in nonprofit organizations to solve domestic or global problems (fewer than 10% felt the same of corporations). The Economist posits that much of that trust may be tenuous due to the fact that “warm sentiments towards charities may be based on a wider misunderstanding of what they do and how much they cost,” and points to the anger that erupts over charity scandals like those involving Unicef in 2008 or the Red Cross in late 2001.

Accountability rests on transparency and thorough evaluation. This involves not only nonprofits making facts and figures public, but their funders as well. One positive recent trend among funders is to publish the results of evaluations on their web sites (For examples, see WolfBrown’s evaluation of the Knight Foundation’s “Magic of Music” program, or this recently-released report for the Nonprofit Finance Fund. The increasing availability of comprehensive analysis drives transparency, reduces reliance on financial indicators to appraise nonprofits’ net impact (more on that from Jane below), and increases awareness of how nonprofits operate.


Conversations with nonprofit leaders about evaluation and accountability usually focus on intended outcomes, impacts, and benefits and how to measure them. In our evaluation work, however, we often observe unintended outcomes, which can be favorable or unfavorable. Recently I ran across a provocative paper by Mark J. Stern and Susan Seifert of the University of Pennsylvania’s Social Impact of the Arts Project (SIAP) that explores just such consequences. Their brief about the social effects of creative economy policies addresses the downside of urban policy makers’ reliance on “creative economy” thinking as a strategy for urban revitalization. They acknowledge that “the logic [behind these strategies] is that attracting the ‘creative class’ to [a] region will generate jobs and tax revenue, a trickle down of benefits to all citizens.” In reality, however, the result is too often not only the gentrification of neighborhoods, but also the “gentrification” of culture – unwitting exclusivity. They discuss research and policy related to the role of culture in urban revitalization and propose a new model – a “neighborhood-based creative economy.” Their thinking resonates with my own feelings about the organic, close-at-hand nature (e.g., shared interests and joint efforts) of what spurs creativity, and its power to transform individuals, families, and neighborhoods.

What is ‘Real Value?’

July 10th, 2009

If you want to put a room full of artistic and managing directors to sleep, start talking about evaluation. While most of them care deeply about the impacts of their programs, talk of evaluation reliably produces anesthesia. Why? Some say they’ve been subjected unfairly to funder-mandated evaluations. Others say that the outcomes of arts experiences are inherently mysterious and cannot (or should not) be measured. A few honest souls will confide that ‘measurable outcomes’ feel antithetical to their sense of artistic autonomy. Maybe part of the problem is that we have yet to define outcomes that speak to the real benefits of arts experiences.

Understanding the accountability environment is, perhaps, one of the most important dialogues we can have as a sector. Is outcomes-based evaluation a distraction or a necessity for nonprofit cultural organizations? What expectations should funders have of grantees with respect to their capacity to undertake evaluation? Why should nonprofit cultural institutions voluntarily hold themselves accountable to a higher standard? As we investigate these and other questions in more depth, I hope you will send us your thoughts and ideas.

Making Evaluation Metabolic

July 10th, 2009

Our research also looks beyond the cultural sector to other kinds of nonprofits. Recently, I spoke with an education research and evaluation organization and a venture philanthropy investment group. A common theme stood out: the process of gathering information for evaluation needs to be embedded in the functioning of an organization or project. Otherwise evaluation becomes a burdensome add-on. As Michael Gilbert argues in Integrated Program Evaluation: A Three Part Vision for Better Leadership, Planning, and Effectiveness, “good evaluation is integrated evaluation.” The implication of course, is that the organization has to have the capacity in its leadership and staff to conceptualize strategic outcomes, and the systems to gather, analyze, and summarize data. The venture philanthropy group I spoke with takes a serious and far-sighted approach, funding organizations to create a theory of change, define outcomes, measure outcomes, and train staff to manage performance measurement.

As part of our research, we have been speaking with executive directors and development directors of state arts agencies and cultural organizations. For most of these people, evaluation and assessment is taking a back seat to the more urgent work of survival. While all agree that ticket sales and attendance metrics do not convey the real outcomes of arts programs, lack of staff time and other resources keep even those who are supportive of evaluation from moving beyond these rudimentary statistics. Where more extensive evaluation is happening, it is primarily because a funder has required it, and, in many cases, is funding it. But there are exceptions. For example, one contemporary arts organization is building evaluation into some of its programs, such that programming will change from one event to the next based on the results of evaluation efforts – sometimes to the discomfort of staff. In general, there is interest in moving beyond “counting heads,” but in a time of scarce resources, doing so remains a vague dream.

Money Well Spent

July 10th, 2009

Of all the foundation presidents active today, Paul Brest of the William and Flora Hewlett Foundation is one of the most respected. He has written extensively on evaluation and accountability, most recently in his book, Money Well Spent, co-authored with Hal Harvey. In a recent conversation, I asked him how his foundation balances the need to come up with significant dollars to do effective evaluation with the desire to put as much money as possible into supporting programming and mission-related activities. Brest replied that this is the wrong way to frame the question. Foundation dollars are intended to drive towards impact. Evaluation is critical in helping to achieve that goal. It is not a separate add-on expenditure, but fundamental to the grant because it is crucial to know if the goals of the effort are being achieved. There is no point in funders making grants, he suggests, if they do not know in some kind of objective way if they are being successful. Indeed, Money Well Spent was written partly to change the paradigm in funder thinking that evaluation takes money away from program funding. Both are needed if outcomes are to be achieved.

Returns on Investments

May 15th, 2009

At WolfBrown, we think a lot about effectiveness and impact and how to frame our understanding and assessment of them. I was recently fascinated by an article in the Chronicle of Philanthropy detailing the Gates Foundation’s innovative effort to assess the return on its investment in schools.

With the help of the Center for Effective Philanthropy and AmericaSpeaks, Gates went directly to students – not principals, teachers, or data-tracking administrators, but 5,400 students at 20 of the 2,000 high schools it has worked with – to take the “YouthTruth” survey. According to Fay Twersky, Gates’ Director of Impact Planning and Improvement, the “YouthTruth” survey will provide feedback to its grantees and “add rigor to Gates’ process,” though it won’t have a direct bearing on their future funding. The article focuses on Woodrow Wilson H.S. in DC’s Northwest Tenleytown neighborhood and describes how the young people were equipped with wireless devices whose instant polling feedback immediately compiled results on a video screen for everyone to see. The devices were a tool to capture the kids’ attention and prepare them for more serious questioning that followed in an online survey later. The article suggests the survey may also prompt students to think in ROI terms. Questions like “What obstacles, like drugs, crime, or family responsibilities make it difficult for you to perform well in your studies?” were designed to keep them thinking about effectiveness and impact, too.

Creative Communities

January 23rd, 2009

On January 11th at the annual conference of the Association of Performing Arts Presenters, Nancy Cantor, Chancellor of Syracuse University, gave a fantastic speech about how presenters, universities, and communities can work together to achieve a higher purpose of creativity. Her remarks punctuated a gathering of grantees and other presenters who are involved in the Creative Campus Innovations Grant Program, funded by the Doris Duke Charitable Foundation and administered by Arts Presenters. WolfBrown is evaluating the initiative. Cantor sees new possibilities for interconnecting people, programs, ideas, and spaces through the arts. I was particularly interested to hear of the giant outdoor video screen that has been installed on the exterior of an old furniture warehouse in Syracuse, which will be illuminated from nightfall to 11:00 p.m. every night of the year to showcase videos made by artists, students, and community members. The overarching idea is a radical redefinition of the role of the arts presenter, from a booker of touring artists and attractions to a catalyst for creative development.

Proudly powered by WordPress. Theme developed with WordPress Theme Generator.
Copyright © WolfBrown: On Our Minds. All rights reserved.