[Part 1] [Part 2: Strong patterns] [Part 3: Scribes’ feedback]
In part 1, I summarized each of the elements in our 40,000 stories about community efforts. Now, let’s put all of these elements into one big matrix and look at which combinations lead to the cleanest or most-muddled patterns.
Quick Recap on the ratios:
For all 14,590 stories that are tied to some organization not associated with GlobalGiving, this is what we find:
What do you see?
- First, if you were only looking at the SUCCESS:FAILURE ratios, you might be misled into thinking that organizations that engage in solving physical problems have the greatest success (there are 377 positive stories in this cluster, compared to only 2 negative ones.) Ratios can be misleading that way when the denominator gets too small. Assuming many of the mixed stories are also not great examples of success, the clear_success column points to some other types of stories. In fact, stories primarily about “the problem” and “physical well-being” are more likely to have mixed outcomes than be a success.
- My interpretation: the ratios in the success_2_fail column are typical of the positive bias that organization self-reports have, but the ratios in the clear_success column recognize the importance of mixed outcomes – these ratios present a much more realistic view of which types of interventions lead to greater success.
- Stories primarily about a solution involving a mix of social, physical, and economic components have the clearest pattern of success. At least for non-GlobalGiving organizations, that is. Actually (jumping ahead), it is true for GlobalGiving partner organizations too, even more so. Twice as many “mixed+solution” stories have clear outcomes than those with mixed outcomes, and many more successes than outright failures.
- Authors who describe their story as “solution” focused are more likely to tell a simple, clear success story. This re-sorted chart might make that clearer:
- The complexity comes when organizations try to do more than provide basic needs, such as food, shelter, security, blankets, and temporary relief. Note how the numbers in the last three columns change (roughly related to the percent of people checking a box to say their story is related to basic needs, psychosocial needs, or higher order emotional needs) when a story is about a ‘social’ need, problem, or mix of these. Stories that are “needs” or “mixed” (need/problem/solution) about organizations are mostly about basic needs, not higher level needs. This might be obvious, but you’d be surprised how often organizations struggle in transitioning a project from A to B (A being immediate disaster relief and B being longer term disaster recovery).
What is in these stories?
Focusing only on 2451 stories that match three criteria (about an organization, are primarily about social relations, and are a combination of need, problem, and solution), there is a map of all the common words that appear in this cluster of stories:
This image was generated using Gephi. To convert 2451 stories into a network of words (and hopefully, concepts), I wrote a python program that recursively identifies the main top-level words, and then the most common words within stories that contain these top-level words, and so on, until you have branches of related words representing major trains of thought within these stories.
Kinda amazing that we have this universal encoding system for concepts, called language. It can reveal some pretty nice patterns when we learn how to decode it. Not that this stuff is as amazing as the first time you used a google search engine, but it does constitute an instant way to build a taxonomy of concepts within any set of narratives. Let’s look at what each of these branches focuses on. Remember, all that these 2451 stories have in common are the three characteristics (social + org-related + mixed need-problem-solution).
The education branch (primary & secondary schools, students, parents, and school fees)
The work, agriculture, and livelihoods branch
The HIV, child abuse branches
Also has a smaller cluster around clothes & blankets, street kids, and the disabled.
There are a few other branches, but they are harder to decipher. The center of this map contains all the NGO-ish jargony words that organizations used to describe themselves: organization, providing, helped, community, living, care, standard, and a few interesting words like orphans – which is connected to all of these. Orphans seem to be common to a wide variety of stories with words that overlap with all branches.
Here’s a little more detail on the child abuse branch. In this case it is only maps 230 stories that include the phrase ‘child abuse‘, ignoring all other factors:
If you like, you might want to look at a related map of stories about street children I posted earlier.
What do SUCCESS outcomes stories talk about (within this group of social+organization-related+mixed need/problem/solution stories)?
Vs Mixed + Failure outcomes stories
Describing community efforts that work is much easier than describing efforts that did not work. These maps instantly give you a feel for the depth in different types of narratives. Later on in this series, I’ll demonstrate other methods to measure story quality and depth.
Another pretty strong signal is that stories about organizations that help children can be the simplest and most positive stories, yet a lot of failure and mixed outcomes stories also mention and/or target children.
Summary: Which community efforts are most complex?
This turned out to be just the tip of the iceberg to this question. But hopefully it should clearer that there are patterns to be discovered within this large data set. Here are some take away points:
- Few community efforts are all positive or all negative. We should never assume that a set of qualitative data that lacks mixed outcomes is going to tell us anything we didn’t expect.
- Stories with mixed outcomes (with both positive and negative aspects) are valuable for learning.
- Helping children, especially with their education needs and basic needs seems to be the clearest and simplest form of community effort.
- Even within the clearest cluster of most-likely-to-be-about-success stories (those with a social focus, mixed need+problem+solutionl, and discussing the work of an organization) there is significant diversity in what gets talk about. Even the same branch of words can appear in both success and failure stories.
CORRECTION: Where I mention analyzing 2451 stories in this post, these are all social + mixed need/problem/solution stories with all kinds of outcomes and include stories not specifically about an organization. My mistake – i’ll try to re-analyze this data with only those stories that are org-related.