This week our team at GlobalGiving upgraded our website and expanded the way we measure and reward our thousands of partner organizations. Two years ago we began this journey on the hunch that curious, learning-focused organizations grow up to become the most effective organizations at changing lives — the ones that deliver high-quality services to those in need. We believed that cycles of learning turn into virtuous cycles of impact.
Today I took 1,325 evaluations from hundreds of in-the-field travelers to GlobalGiving projects over the past 5 years and analyzed them. BigML, the DIY machine learning site, allowed me to understand what defines a great organization in about 15 minutes. That alone is cool, as this type of analysis would’ve taken weeks just 5 years ago! Given one column in the data that represents the outcome you want (or don’t want) to achieve, BigML organizes the rest of the data into a branched contingency tree, like this:
Reading the tree reveals which other questions in the evaluation are the most reliable predictors of answers in that primary outcome column. Statisticians run something similar called a principal component analysis. The labels at the top of each branch of the tree define what makes a great organization, apart form an average or a poor one.
I chose for my outcome column the net promoter question:
How likely are you recommend this organization to a GlobalGiving corporate partner, on a 0 to 10 scale?
So what are the most strongly correlated answers with great organizations?
- transparency organization management: at least average
- online fundraising capacity: high
- collecting feedback from the community: yes
I was excited to see this. Although more than 50 possible features were in the mix, the top three align exactly with what we encourage oraganizations to do in our GG Rewards program (launched today).
This is a totally separate line of evidence. Evaluations didn’t inform our design; partner organization feedback did. But the results of this analysis validate the direction we’re taking. Great organizations are more likely to have a clear management structure and be actively collecting community feedback, just as we seek out feedback from our community of partner organizations and work to clarify how we decide what we do.
Curious what else made the “great organizations” list?
4. this organization is a good fit for our storytelling project
5. project descriptions on GlobalGiving are reasonably accurate
6. the in-the-field traveler was able to visit other projects during their site visit beyond the intended project
What is characteristic of poor organizations (those least likely to be recommended)?
- not very clear how organization is using funds
- unclear management structure
- poor online fundraising capacity
And what are common characteristics of average organizations:
- they are clear how funds are being used
- project leader was very transparent (this is a separate question from whether the organization is transparent as a whole)
- they don’t excel at the things that great organizations do (above).
I hope this illustrates what evidence-based planning is all about, and how it fuels smarter, leaner social change. If we are ever going to transform the way do-gooders make progress around the world, it is going to come on the heels of learning more, faster. That’s why I’ve chosen to work on a group of aligned projects, including GG Rewards, the storytelling project, Keystone’s Feedback Commons, The feedback loop diagnostic quiz, and Feedback Labs. They’re all components in a system to evolve smarter solutions to the world’s most pressing problems.
Nick, a colleague, thought we could be more rigorous in this experiment. So he suggested I use a different outcome for this analysis: the number of rewards points these organizations had earned prior to the day we launched our GG Rewards program. So I did.
A slightly different set of the 64 characteristics we tracked in evaluation profiles seems to best correlate with earning enough GG rewards points to earn Superstar status.
Superstar organizations are most likely to…
- have high fundraising capacity (we already knew that our legacy rewards system was too-heavily weighted towards this one factor. That’s why we revised it to weight learning higher yesterday.)
- not use our “tribute cards” feature (this one is baffling. But it’s data all the same)
- evaluator comments didn’t include the word “visit”
- the organization is pleased with GlobalGiving
- introduced the evaluator to beneficiaries – the people served by that project – during their visit
- have at least an average level of transparency around organization management
- be meeting the needs of their community, at least in the eyes of the visitor
In a similar analysis that only uses the points earned for engaging on our platform (and not any points awarded for learning outside of GlobalGiving), we found a similar but encouraging set of features correlating with the best organizations:
- evaluators would recommend them to family and friends
- organization is meeting needs of the community
- community identifies the needs
- the community works with other organizations, not just this one
- the visitor spoke with the GlobalGiving project leader directly
Because “engagement points” are so closely tied to tools, training, and fundraising that GlobalGiving offers, it is wonderful to see that many of the features of great organizations are community-centered, rather than centered around our website and products. The GG Rewards status of great organizations is less correlated with their capacity to fundraise on GlobalGiving and more correlated to their commitment to be a community-focused organization.