Examples of meta stories from narrative analysis

In my previous post, Narrative analysis with benchmarking, I explained how you can search and filter among tens of thousands of stories in the GlobalGiving Storytelling project in a few steps:

My hope is that by making it easy to explore the rich data we already have, we encourage project leaders, community activists, entrepreneurs, researchers, and other curious globally-minded people to think about our world, and continuously refine their ideas:

This behavior is the essence of a knowledge feedback loop; you learn things that help you, so you keep trying to learn more. As the diagram also shows, the tool requires your own curiosity, ideas, and sweat to work.

As a tool builder, I can help by creating a simpler interface and the means to manage knowledge. I have started baking in controls that hide data when the quality is poor, so that you can trust what you see. Future upgrades will allow users to import any data set from a spreadsheet (CSV), google spreadsheet, or RSS. And more advanced statistics are coming. The system is already extensible, for those who are thoughtful and creative in how they filter stories.

But this tool will only change projects — and improve lives — when the people who use it are free to work within their organizations in a true idea-experiment-cycle:

As of today, I’m am happy to announce that everything one needs for the analysis and experiment parts of the loop is available online, for free, and has been extensively tested:

We’re just looking for the missing ingredients – thoughtfulness and curiosity – that only you can provide. If you work for an organization, you should sign up for a training program that will not only help you “jump tracks” onto the innovation-cycle one shown here, but might also help you win more funding grants:

 Apply to the storytelling-grantwriting programme

Owen Barder recently wrote an essay which underscores the importance of putting more tools like these into the hands of those who will change the world, because “solutions” cannot be directly copied. They must be reinvented for each local context:

Where it is not possible to replicate success directly, it may be possible to support systems to enable them evolve more rapidly and more surely towards the desired goals. – Owen Barder

“Evolve” is precisely the right word, as I’ve explained previously. If you’re looking for ways to boost the rate that your organization learns, you may find these next illustrations inspiring.

This approach is about applying simple rules to semi-structured content, with complex consequences. The compare tool allows you to search for two collections of stories. You “build” a collection by choosing which answers to questions matter to you, and which words in stories people share are relevant to your idea.

Compare

Example: Female Circumcision vs Female Genital Mutilation FGM

There is an organization in Kisii, Kenya that rescues girls from families and gives them a home in a boarding school, so that they can escape female genital mutilation (FGM). The language they use is very different from the language that Kisii tribe members use to describe the same thing.

On the left: stories about “female circumcision” excluding the word “hiv.”(male circumcision has been shown to reduce HIV infection rates, so I’ve excluded those stories)

On the right: “genital mutilation” or FGM:

The size of the people represent the proportion of stories that come from those demographic groups. The color (red-yellow-green) represents how negative or positive stories were compared to what we expected (based on all stories collected). This is how you read it:

The teenage boy icon is larger because they are more likely to talk about “female circumcision”; the teen woman icon is smaller because they are less likely. No girl icon appears on the left because no girls used those words at all. Some girls did talk about FGM, and these icons are red because these stories are associated with negative emotions. (We asked them how they felt about the story they told and they checked the box for a negative emotion):

Upon merging (dividing left by the right), you see that women and men have very different perspectives on the issue.

Men are very positive about “circumcision” whereas women are negative. The women icon is smaller, because women are more likely to talk about FGM and not “cicumcision.” And looking back at the demographics that were merged, men were less likely to talk about this than some other topic. It seems to be unimportant to older men and women, and more important to younger people.

How is this useful? If you are Kakenya’s Dream, you could use this in a grant to underscore just how divided the community is about FGM. It would also be fair to include narratives from those who vehemently oppose your work, so you can talk about your efforts to reconcile “tradition” with the rights of women.

While I’m at it, why don’t we just broaden our search and see how people talk about those two ideas? I’ve searched for two new collections. On the top, stories with  (women and rights) or (FGM or mutilation). And on the bottom: (tribal tradition ethnic kisii) and “practice”:

Explore narratives with bubble plots

The bubble plot tool puts all the words that get used “enough” into bubbles and sorts them up or down, depending on how often they tend to appear in either the top or the bottom collection. Words more likely to appear in the overall 57,220 stories are excluded. Common two-word phrases (“human rights”) also appear and gobble up the individual words (“human” and “rights”).

Bubbles are more like the tea leaves of understanding people and cultures. Sometimes the patterns are meaningless, and sometimes they offer deep insights. It is up to the reader to decide what to focus on, but the basic computer filtering ensures that anything you see appeared in a good portion of the stories. When you understand what you are looking at with the basic bubble plots, click the [CUSTOMIZE] button next to the bubble button and change how it calculates and displays patterns. Like with the compare tool, I tried to hide the full barrage of options. This is a new way to interact with data, and it may take an hour of playing iteratively before you can really get the most out of it.

So what do these bubble say about women’s rights vs traditional, tribal practices? Well for starters, women’s rights are human rights (to those who talk about it) and female circumcision is NOT a part of it. Also – the other side is very concerned about HIV/AIDS, and stories about “practices” include specific mention of “old men,” “early marriages,” and “young girls.” So I would venture to say that any successful program needs to deal with the practice of old men marrying young girls under the guise of “tradition” head on to be effective. Rescuing girls from homes may not do much to improve their lives if they later return to the village and are forced into marriage to old men.

Example: What is “food security,” really?

By comparing stories with the NGO speak “food security” against a much larger collection of farm-grow-plant stories, we see who talks about it, and what words they use:

Adult women talk are more likely to about food security. The topics generally are more positive than the typical stories. Below: words above the dividing line are more likely to appear in stories about “food security” than other stories about farming, growing, or planting.

What do girls dream about, hope for, or want?

By searching the texts for phrases “I dream” “I hope” or “I want” and then splitting left/right by female-male, you can see…

Girls are much more likely to frame their aspirations in a “if I work had… then…” mindset than are males. (Words above the line are more often in female-narratives; below = male centric words). Boys talked about World Vision much more. And both sexes talked about education and starting a business equally (bubbles on the line).

Taking a broader step, you can see cognitive patterns change in women throughout life in rather interesting ways:

In stories where women talk about “hope” and use at least one thinking word, they tend to be more negative than women who hope for things without thinking much about it. As women get older, stories of hope are more likely to be negative, but especially so if they have also thought and written about examining it.

The author of the book, The Secret Life of Pronouns, finds a similar pattern as we see here – critical thinkers are more negative about the events:

But the other trend is something unique to our international development storytelling: People become less likely to describe a story with introspection as they age. I’ve speculated this is because government and civil society don’t listen.

Returning to the “hope, dream, want” collection – after you merge both collections, divinding patterns on the left by those on the right –  fun, freedom, and respect are the talked about more by women than men. And whereas women are more positive in their stories tagged with fun and freedom than men, respect is neutral. From the two stand-alone data sets (upper left and upper right) it is clear that these aspirational stories tend to be more negative than the typical East African story collected.

School Uniforms in Busia

Innovations for poverty action ran a randomized controlled trial in Busia a decade ago, proving that providing school uniforms improves school outcomes and is more cost effective than school fees. Looking only at stories from Busia and comparing “uniforms” stories to “school” stories (it will automatically remove overlapping stories from the benchmark for you), we see:

Teen women talk about uniforms positively, but younger girls are slightly negative, compared to stories from them about school. Adult women are also negative, and men 17-30 do not talk about uniforms at all. The topic analysis shows that uniform stories are much more about security, and less about knowledge (the books are smaller). Looking at Busia, then Kenya, then East Africa – I find that uniforms are a much less talked about problem than school fees.

Point of view: When ” I ” go to school

Words above the line come from stories that mention “school” and include first person words, such as “I” or “my”. Below: a random selection of school stories. Note how the top is about who the person has to thank for education (mother, father, god, family) and include a lot of positive words. Below, impersonal groups appear (orphans, students, village, pupils, schools, youth, teachers, needy, girls). “I” stories are much richer (higher quality data) because they can teach us more from specific anecdotes than the generalized observations of the stories below the line.

Analyzing Tip: search for ” I ” instead of just “I” so that djotjog.com/compare/ finds the space before and after the “I”. Otherwise, all stories containing a word with an “i” in them would be included.

When you look across two of these examples (people who are thoughtful and introspective in their stories vs those who tell a school story from their own perspective, we achieve nearly opposite patterns in what is positive and negative:

Children (especially boys) are more likely to ask “why” in a story about anything, and these stories are always more negative. Telling a story about school and putting yourself in the story is typical neutral. But when it comes to stories about the topic of respect, both groups are more positive than the rest.

Program-specific benchmarking

The next four examples use stories about specific organizations or projects and compare them to their respective issues.

• The Mrembo project was designed to train adolescent girls about life skills and avoid teen pregnancy and early marriage. It eventually focused on preventing sexual assault and rape because of the storytelling project.
• Tysa is a youth-sports organization. Looking at stories about them compare to their benchmark (youth sports stories), they do a lot more to pay school fees, target young girls more, and work with parents more.
• Retrak is an organization that works with street children in Kampala, Uganda. These stories show that family problems are a major influence in why kids run away.
• Comparing the “Street children” stories to a random sample, we see what is least related in all of development: water, health, HIVAIDS, development, business, and women. All these other issues come up more often in stories NOT about street children.

Extensibility

Extensibility is the degree to which an existing system can accommodate new features with a minimum of changes. It’s a word that never escapes the lips of monitoring and evaluation experts, because evaluations rarely boast this feature (By rarely I mean, never, period. Until now). Not that that they couldn’t, mind you – it just requires reworking the way we gather evidence and rethinking the way we organize it.

I made these tools extremely flexible, both in how data gets in and how we pull insights out because it is much easier to innovate by changing your own world than to wait for others to change theirs. But enough philosophizing. Here are examples of new, meaningful ways to interpret story data that are as powerful as if we’d asked users more survey questions. In every case, you can simply cut and paste the “how to ask it” text into the story text search box, and it is as if you are filtering by answers to the question in the “question” column. You can combine them with specific topics (i.e. (“thank you” “to thank” ) and school):

Category	Question	How to ask it in djotjog.com/compare/
gratitude words	Is this story about thanking an organization for their effort?	(“thank you” “to thank” )
cognitive words	How thoughtful were you in the story you just told?	(know knew realize understand understood think thought consider ponder wonder remember cogn conceive believe speculate why )
exclusives		(but without except however )
aspirational words	Did the storyteller hope for more than what actually happened?	(hope aspir promise predict ambition )
organization words	Words associated with narratives where an organization was involved.	(organization organisation admin accountable addressing collaborating development association “women group” “self help” cooperative constituent intervention “youth group” ministry foundation project program initiative )
negative  words	How bad did you feel about the story you told?	(” no ” ”  not ” never noone nobody )
negative emotion words		(angry depressed confused helpless irritated upset enraged disappointed doubtful alone hostile discouraged uncertain paralyzed insult shame indecisive fatigued powerless perplexed useless annoyed “not happy” embarrassed inferior upset guilty hesitant vulnerable hateful dissatisfied empty unpleasant miserable offensive detestable disillusioned hesitant bitter despair despicable skeptical frustrated resentful disgusting distrustful distressed terrible pathetic despair unsure tragic infuriated uneasy ” bad ” pessimistic indignant )
positive emotion words	How good did you feel about the story you told?	(open happy good great playful calm confident courageous peaceful reliable joyous energetic “at ease” easy lucky liberated comfortable amazed fortunate optimistic pleased delighted provocative encouraged sympathetic overjoyed joy impulsive clever interested glee surprised satisfied thankful frisky content receptive important animated accepting festive spirited certain kind ecstatic thrilled relaxed satisfied wonderful serene glad cheerful bright sunny blessed merry reassured elated jubilant love strong loving eager considerate keen affectionate fascinated earnest sure sensitive intrigued intent certain tender absorbed devoted inquisitive inspired unique attracted determined dynamic passion excited tenacious admir engrossed enthus hardy warm curious bold secure touched brave sympathy daring challenged loved optimistic comforted drawn confident hopeful )
question words	Did the storyteller ask a question in the story?	why
discrepancy words	Did the storyteller talk about what could have happened?	(could would should )
tentative		(maybe perhaps sometimes might almost “more or less” )
first person	“ I “ is used more by followers than leaders, more by truth-tellers than liars,	(” I ” “I’m” “I’ll” “I’ve” “I’d” )
cause-effect	Story shows cause-effect thinking	(because reason effect ” if ” )
analytical		(but without except) and (because reason effect) and [cognitive words above]
black-white thinking	Does he/she see world in absolutes?	(always never absolutely surely )
relationships		(mother father sister brother son daughter grandfather grandmother parent friend lover husband wife relative uncle aunt )
time-space words	Associated with truthfulness	(day time started year morning evening night) and (after before while next around above often )

My “Claimer” (e.g. the opposite of a disclaimer)

If this kind of analysis seems too abstract to be useful in international development, I’d caution you to try using community feedback to think about the root causes of the problem before jumping to the conclusion that by measuring the countable goods and services delivered better (the “outcomes”), we solve the problem. Today you can study the root causes much easier than ever before, and our understanding of the problem is ultimately going to be the less complex part of the problem to “fix.” As Anais Nin says,

We don’t see things as they are…

…we see things as WE are.

Outcomes vs monitoring: While the logistics of every intervention requires a quantitative measurement and real-time tracking approach, this is not that. USPS, UPS, and FedEx are masters of logistics, but they can’t tell you what to buy your mother for Christmas. This is a tool to understand your mother.

Impact evaluations answer the question, “what would have happened if we did nothing?” and  “What tangible improvements with we make?” This also, is not that. It doesn’t want to be that. Take education, for example. If educators applied the “impact question” they would ask, “how will this lesson plan change a student’s income 25 years from now? How will it make them more likely to vote, to volunteer, to avoid breaking the law, or cheat on their taxes?” This is a performance monitoring system, with aspirations to be a real-time feedback loop system between citizens and civil society/government/corporations/media. I take my lessons from educators who learned long ago that the “impact question” cannot be answered quick enough to provide course-correction (pun intended). Instead, they ask, “what are students retaining from this lesson plan?” and “can they apply what they learned today to real world problems tomorrow?”

Yeah, we’re doing that too. This is part of a larger program GlobalGiving is launching next month to provide all of our partner organizations with real-time feedback on their performance. Specifically, how well they listen, act, and learn in cycles as they do the work they are already doing. By interacting with a website (globalgiving.org) for a few years, they generate a behavioral profile that they can learn from, especially when benchmarked against similar organizations.

If the aggregate-filter-contexualize-benchmark-visualize features of that system resemble this system, it’s because I helped to create both. I think these may become the future steps of all big data learning feedback systems, but what do I know? We in the aid world are still talking about samples in the hundreds when corporations are talking about the coming brontobyte era – where we archive more data each day than we created in past 2000 years. Data, mind you, is not knowledge. You need to aggregate-filter-contextualize-benchmark-visualize it before you can listen, act, and learn from it.

Better narratives are simply better data: The problems in international development are going be an order of magnitude easier to solve if we have richer data. To get it, simply (1) Ask people, on a large scale, what they want. (2) Demand that they get involved in the process if they want it. Many will volunteer to improve their own lives. (3) Work with them to make sense of their own world, and fix the work being done “to them” and not “for them.” (4) When get it, you’ll “get it.”

This iterative approach requires more aggregation and less structure in the data itself. My next batch of tools released will support that need exactly.

Quality control: When working with narratives, “quality control” is not as hard as you think – but you need to follow the rules.

1. Collect enough: Minimum viable sample size seems to be around 100 stories (when all answer the same prompting question). Collections of stories can be used to build a meta-narrative (and are viable for statistical significance testing), whereas individual stories can only be trusted as anecdotal evidence to support an idea.
2. Calculate statistical power: Power is the chance of seeing a difference, if there is a difference there to see, and nobody pays attention to it enough. Power is related to sample diversity. Did you get enough independent sources? If you think this is “staff work” and not “community work” then you will fail. There are no experts you can outsource the evaluation to. You must engage your community for it to work. Future tool upgrades will auto-calculate the “statistical power” of collections for you. When you survey both the community and your organization’s beneficiaries you will have more power to detect patterns.
3. Diversity makes it a meta analysis: The tool tells you how many scribes, storytellers, organizations, and locations are represented in each story collection you build with djotjog.com/compare/. Future versions will have the power to calculate meta analyses, with the power to provide results as rigorous as randomized controlled trials, if you have the power in your sample to detect differences, but without depriving people of what they deserve. In the school classroom example, to truly measure the impact of education, you would need to randomize the classroom and deny half the class an education, just to prove that it matters. We can’t do that. Instead, we have to aggregate real world data and look for natural experiments, such as comparing the thousands of narratives of people already denied an education to those who got the opportunity. It’s not as rigorous, but it will reveal the same answer. The way we “control for other factors” is to have a colossal sample of narratives so that these other factors are in the story but differences in them cancel out. This is the future. And the greater the diversity of sources, the more likely any differences are to be real, robust, valid predictors on a vast scale.
4. “Ahem, they can see your raw data”: Because the data is public, we can check claims people make against their data in minutes. Wildly speculative claims will be easy to refute. And smart advocates will make more public use of the data in their arguments and reasoning to lend credibility to their claims in a trackable way. It’s like “open-sourcing” the evidence-based-decision-making of the aid world (not that I actually think they make evidence-based decisions, but now they can stop pretending and start attending to the needs, opinions, and insights of citizens.)

Why this is better:

1. Easier to manage than “quantitative” indicators: Collections are extensible, aggregatable, and comparable.
2. We can detect and correct bias with narratives, as explained in The Secret Life of Pronouns (James Pennebaker).
3. Meta analyses with statistical “power” (Frank L Schmidt).
4. Emergence: narratives and brief surveys provide “enough”.
5. Focused on listening and collecting multiple perspectives.