|
[Printer Friendly Version]
Integrated Program Evaluation: A Three Part Vision for Better Leadership, Planning, and Effectiveness
By Michael C. Gilbert, April 16, 2009
"This is how we do evaluation in the nonprofit sector: We feed people carrots. After it's all over, we administer a survey. We ask them: Did they taste good? And is your eyesight better?", Jan Masaoka, Blue Avocado.
Would you drive a car whose speedometer was bolted on after the fact? On the outside of the car? With the only measurement being the average speed since your last stop? Based on an end-of-trip poll of the passengers in the car as to how fast the trip felt? Oh, and finally, the results of which required an expert to read and interpret?
Right. I didn't think so.
But, that is more or less how civil society organizations evaluate most programs, if we do any evaluation at all.
Scaling Up Bad Evaluation
Many years ago, I joined others in becoming more outspoken about encouraging critical thinking about outcomes. Several years later, I started noticing some unfortunate results of this advocacy. We succeeded in encouraging more evaluation, to be sure. But to some extent, the result was just larger quantities of bad evaluation.
The biggest and most obvious problem was that people fall into the classic trap of valuing that which could be measured, rather than measuring what was valued. Measuring website traffic is one common example. Confusing outcomes with outputs is another.
A second problem is more subtle. Rather than encouraging deeper systems thinking, as I had hoped would happen as we became more and more adept at identifying and testing our logic models, we became increasingly reductionistic. Turns out that, whether you're a funder or a grantee, paying attention to "outcomes" is much easier than deep systems thinking about strategy.
A third problem was caused by consultants and vendors who rushed in with superficially simple solutions to the challenge of outcomes. This served as a multiplier of the first two problems, making it easier and easier to measure the wrong things and to get trapped in shallow thinking.
By far and away the largest problem is that "outcomes" have come to be seen as an add-on, kind of like a final report, ultimately separate from the project itself. By treating it as separate, we can hire people to do our evaluation for us, rather than integrate fully into our planning.
Therein lies the critical insight: Good evaluation is integrated evaluation. What I'm calling "integrated evaluation" isn't anything new. It's what the best advocates of effectiveness, logic models, and good program planning have been saying for a long time. But I hope that by presenting it in terms of the theme of integration, I can help encourage better practices.
So, what exactly am I talking about, when I use the word "integration"? What is evaluation supposed to integrate with? For this short piece, I will identify three major aspects of program work with which evaluation should integrate deeply. It should integrate with leadership, with planning, and with the work itself.
Integrating with Leadership
It's common for leadership to avoid rigor in their thinking about what constitutes success and how they'll know it when they see it. Why is this? Maybe it's because many great leaders are motivated by stories, and rigor can feel cold and inhuman in comparison. Maybe it's a healthy desire to avoid reductionism. Maybe it's because in many organizations, failure is not a comfortable part of the work and thus it's in their interest to have success criteria be a little fuzzy. Maybe it's because top level managers think they are safer in their jobs if they define success after the fact. Maybe it's a matter of training and culture. Maybe Grantmakers for Effective Organizations knows the answers.
Whatever the reasons, leadership plays a huge role in shaping program evaluation. Although there are grantmakers that have helped stimulate interest in certain ways of thinking (such as theories of change and logic models), I'm not sure how far past the grantwriters these messages and methods have gotten. Indeed, it's not clear that the grantmakers themselves even practice what they preached, at least at the highest levels. What that means is that, despite the good intentions and practices of others, as program ideas move from layer to layer of the organization and as program activities move through month after month of implementation, solid evaluation can become more and more removed.
Integrating with Planning
Despite the fact that "planning and evaluation" are linked together in common usage of the terms, in common practice they are separated by the gulf of activities of the programs themselves. Planning happens at the beginning. Evaluation happens at the end. If at least a little attention has been paid to the tropes of modern planning, then the evaluation may at least be couched in terms that were used in the planning phase. But often, that is as integrated as it gets.
The larger the gap between planning and evaluation, the less relevant each become to the work itself. Without the benefit of iterative, systems-based development of evaluation, planning becomes unanchored. Without an anchor, it seeks whatever stability it can find, most likely the stability of existing roles and relationships in the organization. Without anyone meaning to make it so, the program comes to serve those roles, rather than the cause. Without the benefit of the planning process, evaluation becomes post-hoc, exhibiting all the symptoms described in the opening of this piece. The data gathered and the conclusions drawn may be framed as relevant, but they aren't.
Integrating with Work Itself
Up to this point in our discussion, new communication technology plays a very small role. It plays almost no role in the topic of integrating with leadership, except for perhaps the presentation of tools and dashboards that can make bad evaluation even more influential. It has played and will continue to play a modest role in helping integrate evaluation with planning. Anything that makes it easier for inexperienced planners to use theories, logic models, and other tools to help define success and be able to assess it is a good thing. This includes software and online services in the field of planning, systems analysis, and so forth. But these things will only take us so far.
What has the potential to truly leverage us past the barriers to successful evaluation is the integration of evaluation into the work itself. The reason we resort to bad, post-hoc methods for gathering data about our programs is in part because this was, despite being horribly expensive, actually the cheapest thing we could do. Whether you were a social worker or an activist or an arts manager, measuring your work has always seemed like an extra burden. When choosing between record keeping and actually doing the work, we choose the latter, every time. In fields where reporting is mandated, such as government funded social services, filling out those forms is a loathed activity that contributes profoundly to burnout.
This is where new media comes in. So much of our work is electronically mediated or so easily amenable to automatic tracking that we are at a point now where, for a great many work processes, we can think about making them essentially self-documenting. We've barely scratched the surface here, but the potential exists for dramatically lower costs of getting truly relevant data. Evaluation can be based on real facts, made visible in the course of real work.
Benefits of Integration
There are at least five benefits to integrated evaluation.
(1) Evaluation as currently practiced is a clumsy and expensive endeavor. If the design of evaluation is incorporated into planning and the collection of data is incorporated into the work itself, there is a lot of money to be saved.
(2) Evaluation as currently practiced is a highly political process, full of risk, anxiety, and distortion at every level. It puts burdens on everyone who is actually doing the work that we're here to do, whether on the front line or in management. If data gathering becomes more seamless and leadership is committed from the beginning, we will all be able to breathe more easily.
(3) Evaluation, regardless of how it's practiced, has influence. It is first and foremost a tool for learning. If evaluation becomes more relevant, and if we're paying attention, we will all learn the right lessons.
(4) Post-hoc evaluation ends up being a kind of final judgment on a project and doesn't give us much room for mid-course corrections. If evaluation is part of the program work itself, then we get real-time feedback and can make real-time improvements.
(5) Finally, the most important result will be seen in the outcomes themselves. We will have more money for them. We'll be able to focus on them. We'll be able to improve how we pursue them.
And that's why we're all here, after all.
If you found this article interesting or helpful,
please consider making a donation to Nonprofit Online News.
It will probably feel good!
|
RSS