Is international development complex? (meta analysis of 40,000 community stories)

[Part 1] [Part 2: Strong patterns] [Part 3: Scribes’ feedback]

Nearly 3 years ago, Nancy McPherson (director of evaluations for the Rockefeller Foundation) and Irene Guijt started a conversation with GlobalGiving that presumed many international development efforts are inherently complex. This means that few social, economic, or health problems can be solved by simply supplying the right ingredients. An example of a simple, solved global problem was small pox. We found a cure, applied the vaccine to everyone in the world, and the virus disappeared from the Earth (except in CDC level 4 bio labs).

Today, after collecting over 40,000 stories in Kenya and Uganda, we can actually look at patterns in these stories and predict which kinds of interventions are more complex than others.

The Storytelling Question

Every person chose one community effort – a ‘thing’ that a person or organization tried to do in their community to improve it – and wrote a short story about it. In addition, they answered some questions that mapped out the elements of that story. In this analysis I used 6 of those questions (ignoring for the moment the very important role that one’s role (actor, observer, affected by the events) plays in interpreting the meaning of these stories). Here are the story elements and some lessons from analyzing them:

Need-Problem-Solution

Might not seem like a logical split, but many community stories are about an unmet need that may not have any evident solution. From the authors’s perspective, the story may better characterize the community need than the community problem that is being addressed by the need.

Typical data looks like:

youth-need-prob-soln-sensemaker — 741 stories mentioning “youth.” Each blue dot is the author’s perspective on one story. This image was generated using SenseMaker(R), licensed from Cognitive-Edge.com

Breakdown by number of stories per corner (see diagram below for meaning of ‘mixed’):

         success  failure  mixed = outcome
need     3201     190      4410
problem  2355     254      7566
solution 2777     537      2934
mixed    5730     378      11137

In all 4 cases, success stories are more common than failure stories, but less common than stories with “mixed” outcomes – meaning some aspects were good and others not-so-good. ‘Solution’ stories were the clearest place to find examples of success (as you’d expect). A larger portion of ‘Problem’ stories had mixed outcomes than ‘Need’ stories. The little diagram of what ‘mixed’ means reveals that there are four more clusters we could analyze, but I’m ignoring those too for the moment – there really is too much data to wrap your head around all at once.

Social-Physical-Economic

Breakdown:

                     success    failure  mixed = outcome
social relations     5007	593	 7068	
physical well-being  2214	228	 5445
economic opportunity 1491	207	 3449
mixed                5351	331	 10085

These three areas can be combined with ten more topics (derived from the hierachy of needs) to map what kind of community effort the story is about:

        [ ] creativity [ ] freedom [ ] fun
     [ ] knowledge [ ] respect [ ]self-esteem	
            [ ] family & friends
[ ]food & shelter [ ] security [ ] physical needs

Storytellers were asked to check 3 of these 10 boxes that best described their story.

The checkbox approach – like any survey – is still an abstraction of reality. We compared the words in the stories against the categories that authors placed their stories in using check boxes, and this is a more realistic view of the fuzziness of each story mapped against the Mazlov’s hierarchy of human needs:

Hierarchy of needs — Comparing two approaches to measuring where each story falls on a hierarchy of needs. Scores on the Y-axis are calculated by comparing story overlap with a dictionary of words representing food, self-esteem, creativity, and so forth. Image was created using SenseMaker and Paint. There is some correlation between the check-boxes and the story-words methods, but not a strong one. This is precisely why we ask every storyteller to choose 3 categories – stories are rarely about one thing.

Success and Failure

Actually this question is not directly asked, but we compile a score based on 3 questions:

So the highest success score would be a story that was a [x] good idea that succeeded, benefiting the right people, and made the person feel happy, hopeful, or inspired.

Breakdown:

Note that there are TEN TIMES MORE positive stories than negative ones, but only HALF as many clear positive outcome stories than mixed outcome stories.

         Count     Percent of total
Success  14,063    33.9% 
Failure   1,359     3.3%
Mixed    26,047    62.8%
Total    41,469 (on 7/19/2012)

Throughout this experiment, we were aware of a huge positive bias in stories. This is a built-in human behavior. If you give someone two chances to tell a meaningful story about their lives, 9 out of 10 people will pick positive stories unless the negative experience was really negative. In contrast, if someone has a bad experience at Dunkin Donuts they’ll tell 30 people about it (I’m lookin at you @akmcquade), but tell only 3 people about a positive experience. This is yet another reminder that international aid is not like democracy. The beneficiaries do not behave like voters and they don’t demand services – they act like the recipients of gifts from above, and very rarely criticize. Any rational person would weigh the pros and cons and realize that criticizing something would only risk losing what little benefit it offered them. This makes it really hard to get honest feedback, and in a sense, untrains citizens to demand services from their own government too. So unless international development wants to behave like democracy, African “democracy” will tend to behave like aid.

[Stepping off my soapbox…]

Q: How is this success-failure-mixed stuff related to democracy and power?

Feedback is always much more positive than reality. (10 times more common, actually)
We can’t improve things without honest feedback.
This indirect approach really works, because ‘mixed’ outcomes are twice as common as positive outcomes, which is closer to the truth.
A more honest measure of success will enable organizations to do more reliable work in the future.

Project/Org focused story?

This story element was a bit more opaque. We are asking (without actually asking) “how important was some organization to your community effort story?” And while we tried various approaches at this question in the survey, looking for organization-type words in the story itself seems to work better. So using a patternizer I wrote in python, I was able to come up with the most common words that are organization-focused:

organization group started support 
gives giving helped helping help
project organis organiz provide providing
ngo cbo

Breakdown:

Project focused     30,814      74.3%
Non-project focused 10,655      25.6%

Or only using the most conservative filter (stories containing cbo, ngo, organization, organisation, group, or project)

Project focused     17,770      42.9%
Non-project focused 23,699      57.1%

The true split is probably a little lower than 74:26, but most of the ‘project focused’ stories are about community efforts, whereas the non-project focused stories are a mixed bag : (some situation descriptions, some stories without a narrative, some without an actor).

GlobalGiving Org related story?

This was, by far, the hardest one to assign. Most storytellers don’t know which organizations are involved, or use a different name for them, and none know whether the org is on GlobalGiving. Whereas all of the previous data could be organized using a single MySQL query of our data, this one required a lot more work and several iterations of checking and rechecking the organization named within each story against our list of 12,000 past and present partners:

GlobalGiving partner organizations were named in 4,413 stories out of 41,469 total = 10.6%

I believe most of the remaining stories are relevant to the work our partner organizations do because of overlapping social problems and similar topics, but this number represents our best estimate of specific attribution. (But please take a look at this post on all the problems with attribution, and maybe we can all agree to focus on contribution instead in the future.)

Interesting ratios

Now that we have a bunch of story elements and three kinds of outcomes (success, failure, mixed), there are three ratios I find useful to look at with each combination of elements:

Outcome (1) Success stories : Failure stories — high numbers = lots of success. This is the most intuitive ratio, but less descriptive than the other two.
Clarity (2) (Success + Failure) : Mixed — high numbers mean that the outcomes of this cluster of stories are clear, either success or failure. Low numbers mean a lot of mixed results, and this sort of thing is inherently complex and muddled.
Clear success (3) Success : (Mixed + Failure) — high numbers mean this cluster has consistently clear success with minimal failure and mixed outcomes.