[Part 1] [Part 2: Strong patterns] [Part 3: Scribes’ feedback]
Nearly 3 years ago, Nancy McPherson (director of evaluations for the Rockefeller Foundation) and Irene Guijt started a conversation with GlobalGiving that presumed many international development efforts are inherently complex. This means that few social, economic, or health problems can be solved by simply supplying the right ingredients. An example of a simple, solved global problem was small pox. We found a cure, applied the vaccine to everyone in the world, and the virus disappeared from the Earth (except in CDC level 4 bio labs).
Today, after collecting over 40,000 stories in Kenya and Uganda, we can actually look at patterns in these stories and predict which kinds of interventions are more complex than others.
The Storytelling Question
Every person chose one community effort – a ‘thing’ that a person or organization tried to do in their community to improve it – and wrote a short story about it. In addition, they answered some questions that mapped out the elements of that story. In this analysis I used 6 of those questions (ignoring for the moment the very important role that one’s role (actor, observer, affected by the events) plays in interpreting the meaning of these stories). Here are the story elements and some lessons from analyzing them:
Need-Problem-Solution
Might not seem like a logical split, but many community stories are about an unmet need that may not have any evident solution. From the authors’s perspective, the story may better characterize the community need than the community problem that is being addressed by the need.
Typical data looks like:
Breakdown by number of stories per corner (see diagram below for meaning of ‘mixed’):
success failure mixed = outcome need 3201 190 4410 problem 2355 254 7566 solution 2777 537 2934 mixed 5730 378 11137
In all 4 cases, success stories are more common than failure stories, but less common than stories with “mixed” outcomes – meaning some aspects were good and others not-so-good. ‘Solution’ stories were the clearest place to find examples of success (as you’d expect). A larger portion of ‘Problem’ stories had mixed outcomes than ‘Need’ stories. The little diagram of what ‘mixed’ means reveals that there are four more clusters we could analyze, but I’m ignoring those too for the moment – there really is too much data to wrap your head around all at once.
Social-Physical-Economic
Breakdown:
success failure mixed = outcome social relations 5007 593 7068 physical well-being 2214 228 5445 economic opportunity 1491 207 3449 mixed 5351 331 10085
These three areas can be combined with ten more topics (derived from the hierachy of needs) to map what kind of community effort the story is about:
[ ] creativity [ ] freedom [ ] fun
[ ] knowledge [ ] respect [ ]self-esteem
[ ] family & friends
[ ]food & shelter [ ] security [ ] physical needs
Storytellers were asked to check 3 of these 10 boxes that best described their story.
The checkbox approach – like any survey – is still an abstraction of reality. We compared the words in the stories against the categories that authors placed their stories in using check boxes, and this is a more realistic view of the fuzziness of each story mapped against the Mazlov’s hierarchy of human needs:
Success and Failure
Actually this question is not directly asked, but we compile a score based on 3 questions:
So the highest success score would be a story that was a [x] good idea that succeeded, benefiting the right people, and made the person feel happy, hopeful, or inspired.
Breakdown:
Note that there are TEN TIMES MORE positive stories than negative ones, but only HALF as many clear positive outcome stories than mixed outcome stories.
Count Percent of total Success 14,063 33.9% Failure 1,359 3.3% Mixed 26,047 62.8% Total 41,469 (on 7/19/2012)
Throughout this experiment, we were aware of a huge positive bias in stories. This is a built-in human behavior. If you give someone two chances to tell a meaningful story about their lives, 9 out of 10 people will pick positive stories unless the negative experience was really negative. In contrast, if someone has a bad experience at Dunkin Donuts they’ll tell 30 people about it (I’m lookin at you @akmcquade), but tell only 3 people about a positive experience. This is yet another reminder that international aid is not like democracy. The beneficiaries do not behave like voters and they don’t demand services – they act like the recipients of gifts from above, and very rarely criticize. Any rational person would weigh the pros and cons and realize that criticizing something would only risk losing what little benefit it offered them. This makes it really hard to get honest feedback, and in a sense, untrains citizens to demand services from their own government too. So unless international development wants to behave like democracy, African “democracy” will tend to behave like aid.
[Stepping off my soapbox…]
Q: How is this success-failure-mixed stuff related to democracy and power?
- Feedback is always much more positive than reality. (10 times more common, actually)
- We can’t improve things without honest feedback.
- This indirect approach really works, because ‘mixed’ outcomes are twice as common as positive outcomes, which is closer to the truth.
- A more honest measure of success will enable organizations to do more reliable work in the future.
Project/Org focused story?
This story element was a bit more opaque. We are asking (without actually asking) “how important was some organization to your community effort story?” And while we tried various approaches at this question in the survey, looking for organization-type words in the story itself seems to work better. So using a patternizer I wrote in python, I was able to come up with the most common words that are organization-focused:
organization group started support gives giving helped helping help project organis organiz provide providing ngo cbo
Breakdown:
Project focused 30,814 74.3% Non-project focused 10,655 25.6%
Or only using the most conservative filter (stories containing cbo, ngo, organization, organisation, group, or project)
Project focused 17,770 42.9% Non-project focused 23,699 57.1%
The true split is probably a little lower than 74:26, but most of the ‘project focused’ stories are about community efforts, whereas the non-project focused stories are a mixed bag : (some situation descriptions, some stories without a narrative, some without an actor).
GlobalGiving Org related story?
This was, by far, the hardest one to assign. Most storytellers don’t know which organizations are involved, or use a different name for them, and none know whether the org is on GlobalGiving. Whereas all of the previous data could be organized using a single MySQL query of our data, this one required a lot more work and several iterations of checking and rechecking the organization named within each story against our list of 12,000 past and present partners:
GlobalGiving partner organizations were named in 4,413 stories out of 41,469 total = 10.6%
I believe most of the remaining stories are relevant to the work our partner organizations do because of overlapping social problems and similar topics, but this number represents our best estimate of specific attribution. (But please take a look at this post on all the problems with attribution, and maybe we can all agree to focus on contribution instead in the future.)
Interesting ratios
Now that we have a bunch of story elements and three kinds of outcomes (success, failure, mixed), there are three ratios I find useful to look at with each combination of elements:
- Outcome (1) Success stories : Failure stories — high numbers = lots of success. This is the most intuitive ratio, but less descriptive than the other two.
- Clarity (2) (Success + Failure) : Mixed — high numbers mean that the outcomes of this cluster of stories are clear, either success or failure. Low numbers mean a lot of mixed results, and this sort of thing is inherently complex and muddled.
- Clear success (3) Success : (Mixed + Failure) — high numbers mean this cluster has consistently clear success with minimal failure and mixed outcomes.
Great! Now we can look at all these patterns.
Western HIV/AIDS Network still needs your assistance for the success. Thank you for the goodwork both nationally and internationally. GOOD EFFORTS.