As a nonprofit organization, the question we should ask ourselves is “how do I know if some new approach is better than what I’m already doing” for this community?
Many programs don’t improve lives that much on the whole. But there are several simple ways to quantify exactly how much benefit they have, compared to something else. The “something else” can be quite simple, as you will see.
1 – For every intervention, the control group gets cash instead.
GiveDirectly takes money and gives unconditional $1000 cash to poor people, then tracks what they do with it and how well they live afterward. Chris Blattman believes this should be the null hypothesis for all foundations: What do people do with money themselves, given unconditional cash?
#2 – Citizens allocate the dough. Compare with foundation experts.
Another example is what I call “Pay it Backwards” where we take the thousands of do-gooder citizens that people have already written a story about in our storytelling project and give them $100 to do something good for someone else with. Afterwards, we ask the do-gooder, the inidivual donor (who put up $100), and the beneficiary what happened and were they happy with it? Here it is not GiveDirectly choosing the people, but rather, good citizens whom we empower to act and little micro charities. And the question is, “are people happier when individual helps than when foundations help them? What do they do well? And what do they do poorly (compared to charities)?
#3 – Pay the actors cash not to do something you think is important.
A third model of rethinking the null hypothesis is to split the pool of qualified organizations (that you would normally give a grant to) into two groups. To one group you deny them the grant for a year in exchange for a little cash now and their commitment to give you data about what they are doing. Then you can compare the data with data you collect on your grantees. You should expect that your grantees are doing better work, if you data system is worth anything. Otherwise none of your data is reliable and it’s possible that your grantees would be better off being given nothing but grants. We at globalgiving have considered this. It shows what drastic extremes we in the nonprofit world must go to. For in order to “prove” that our products have value, we have to take some of our customers and pay them not to buy our products, so we can show the product works.
#4 – How much much you pay someone not to doing something?
A variation on idea #3 is to find out how much a person needs to be bribed to try something new. If a foundation isn’t collecting feedback from the people they serve, how much could you pay them to do it? And if they are already listening to a community (because experts at foundations encouraged them to do it), how much would you need to pay them to leave it out of a proposal? If they truly believe that the information is valuable, you would need to pay them to omit it, and how much you need to pay is a measure of what perceived value it has.
The same applies to microloans. If training with a loan is really valuable, could you ask people to give back 10% of their loan to pay for the training? That would be a strong bit of evidence that the training has value.
#5 – All projects must have at least a 25% local matching effort to proceed.
I am a huge fan of the Peace Corps Small Project Assistance (SPA) program. When I was a PCV in Gambia 99-01, I never used SPA myself, but many other PCVs did so successfully. The design of a small budget with minimum 25% community effort match should be the design that all other projects are benchmarked against. If a community is unable or unwilling to quarter-match a project, then the budget is too large or the money going to the wrong thing.
Conclusion: There’s too much talk about impact from people who aren’t able to measure it.
If you’re thinking, “I don’t need to know whether we are better than something else” because your grant funding is restricted to exactly the one thing, you should give up talking about impact. You cannot prove you had a big impact on anyone if you are unable to offer more than one kind of aid to a person. Anyone who says otherwise is selling something.
If your next grant proposal is all about implementing one idea without comparing it to anything else, then don’t try to claim you’ll know what its impact will be. Instead, rewrite your grant so that you can at least say that it was better/worse than cash, or use one of these other approaches to demonstrate there is demand for it.
The ‘null hypothesis’ in any experiment doesn’t need to ask, “is this new thing is better than some old thing?” Instead, show that the intervention is more effective than cash. Cash is the equivalent of placebo for international development.
If you want to say something like “X foundation is 25% more effective than the rest,” you’ll need a better baseline to compare against. That means you need benchmark data and a null hypothesis to test against.