Designs on Systematic Listening

In my last blog post I introduced five testable hypotheses that apply to many designs for helping people. Some examples of those were:

  • cash-48Cash control group: Instead of the program, give them the equivalent cash it would cost.

 

  • decision-making-behaviorDecision control group: Let the people decide what intervention they want to join, and compare to the choices experts make for another group of people. For example, do food stamps recipients make smarter decisions and get better value with cash than with vouchers that are limited to only what the experts think they should buy?
  • effort-matchEffort matched control group: Split the recipients into smaller groups (like micro-lending does) and require each group of say, a dozen people, to match the aid money with some effort of their own. For example, in exchange for getting a bio-waste energy tank, a group could be required to clean trash in the neighborhood regularly – to be verified by satellite imaging. This commitment and verification would prioritize allocations to groups that are most committed to “working” for them. And it lets them “earn” aid instead of simply passively receiving aid.

whole bull planning process

Story-centered Learning Designs

Here I introduce different ways to design a storytelling project with one goal in mind: Systematically listening to people and benchmarking narrative patterns against some control group. It dovetails with the previous post because program design dictates which listening design will work best.

If the goal is a needs assessment, do community mapping

Focus on the location and the people that live there. Ask an open-ended question like,

“Talk about a time when a person or organization tried to help someone or change something in your community. What happened?”

From 2010-2012 we collected nearly 60,000 responses to this question, searchable online.

All stories in a collection will be connected by their proximity in space and time, and often this alone is able to reveal patterns that should inform project design. For example, the Kenyan NGO VAP interviewed girls in their program and learned that rape was a major life issue. Too many girls were bringing it up in generic “community mapping” stories – more than a third – to ignore. So they changed the program to address this immediate need. The following year they used this process to hone in on aspects of youth crime that were amenable to after-school lessons.

Blogs on VAP forming a “case study”:

Mrembo Program

Comparing two rape-prevention programs – this one illustrates how a nearby outside organization can be a good benchmark for your program.

In 2012 our GlobalGiving storytelling project was based in many parts of East Africa. We collected and published community maps for each community:

Kibera meeting and Kibera follow-up. Map Kibera project

Kakamega and Western Kenya

Kisumu

Kampala

In the past creating these maps was a manual process, but soon they will be automatically visualized from the stories and their meta data. “Meta data” are the little bits of related data around stories, such as who scribed them and where the story took place. It is safe to assume that stories with overlapping people and places and dates are important signals for program managers, and that visualizing it can help them make smarter decisions (or even make them smarter decision-makers).

If goal is measuring a program’s impact, these designs will help…

Impact is a messy, vague, ambiguous loaded word. It gets tossed around casually by board members and funders and served on a silver platter to rich people who want to try their hand at being a development agency. This is my attempt to disambiguate Impact into different approaches to measuring them, and tying these back to the best way to gather signals from people.

Customize the story prompting question to map the root causes of a problem

Some past storytelling questions give you a sense for how to map the problem, or issue, that comes to mind in association with a topic:

Please tell a story about a time when you had to choose between protecting the environment and maintaining a livelihood. Include if/how individuals or organizations were involved in the conflict.

אנא ספר סיפור על זמן שבו אדם או ארגון ניסה לעזור למישהו או לשנות משהו בקהילה שלך.

Please tell a story about a time when you tried to get a job. What helped you get a job?

Please tell a story on a most significant change that you have observed based on your experience as a participant in our program(s).

Please talk about a specific time that you felt more visible in your community. What happened and how did it expose some hidden need or issue? What would you like to do to help address it?

In the space provided, please tell us about a childhood experience when you did something you believed you never could have done.

And these illustrate how GlobalGiving used it in our own network:

Talk about your experience approaching a grantmaking or funding organization that either did or did not grant you funding. What was your relationship like? Did you receive support from them?

Please tell a story about a time when when a nonprofit listened, acted, and learned to become more effective at fundraising on GlobalGiving. Did they became more effective in real life?

As a past global giver, why did you give to GlobalGiving or to this project in particular?

Good story prompting questions undergo design evolution based on early testing:

Version 1: Please tell a story about a time when you had to work with someone different from yourself.

Version 2: Please tell a story about a time when a conflict arose because you had to work with someone from a different background (religious, cultural, ethnic etc.) to yourself.

Version 3: Please tell a story about a time when a person changed someone else’s perception of them or challenged a prejudice or misunderstanding.

Some prompts are too specific to be comparable to any other stories:

Please tell a story about a time when YaLa Africa tried to help and empower you or your community through micro-gardening and nutrition training.

You’ll notice that these questions are still much more open-ended than program evaluations use. We don’t ask them to describe the impact directly – we ask them to describe specific events and categorize experiences, so that impact can emerge from the collect as a whole in an organic (less biased) way.

Add survey questions

Certain follow-up questions will extend these narratives in ways that allow for specific quantitative comparisons:

Hierarchy of needs

Which of these relate to your story?
Choose three.

Freedom |  Fun

Knowledge | Respect |  Creativity | Self-esteem |

Food and shelter|  Security|  Family and friends|  Physical needs

Root causes

What is needed to address the problems in your story?
Choose all that apply.

Money | Change to government and institutions | Individual behavior change | Change to society’s attitudes

Give two words to define this problem 
The events in this story…
Choose a point on the line.
slider
Happen often exactly as I told it
Has a different ending from what usually happens

What else would have made a difference in your story?

Power relationships and social hierarchy

Who would you go to if you wanted to solve the problem in this story?
Choose only one.

Chief or local authority | Family member | Religious leader | Vendor or business leader| Friend, neighbor, or community member | Teacher, health, or government worker | Somebody else (none of these)

The events in your story happened mainly because of…
Choose all that apply.

The circumstances people found themselves in | The resources people had available to them  |

The actions people took | The way people felt

Ask beneficiaries two stories

Sometimes it makes no sense to interview neighbors of the people you serve. In that case, each program participant can serve as his or her own control if you invite them to share two stories. The first story can be “How does organization X help you?” and the second one, “how has some other organization helped you?” With variations on this within-subjects control design you can make many comparisons.

What else do the people you serve care about or need?

What other organizations are having an impact on the lives of the people you serve?

How do people feel about various life issues that intersect with the problems your program claims to address?

Is there one demographic group that you are reaching more (or failing to reach)?

Trigger conflict narratives

Good writers know that narratives require conflict to be interesting. There are as many different kinds of conflicts in fiction as there are in program design. Conflicts can be internal and external. They involve human against nature or against each other. Family conflicts differ from community ones. And all of this needs to fit inside 150 words for good storytelling. Frequently, our stories are boring. The authors are reluctant to describe the conflict because it is not their own self-interest, or because their culture forbids it (Lookin at you Japan!):

exciting-narrative-chart

As a result, evaluations are often conflict-free, or at least the conflicts are severely dampened and couched in euphemisms. We’re trying to change that. We’ve giving organizations to probe for conflict, and training scribes to give citizens permission to give feedback in a safe space.

In the example where the person was asked to tell a story about having to choose between protecting the environment and securing a livelihood, we are mapping out an internal values conflict. Other conflict stories can be about working with “other” (see above) or understanding corruption. Impact is not the absence of conflict or measured by how many people make the “right” choice (e.g. choosing the environment over one’s livelihood is not success), but rather comes from understanding the issue on a deeper level and building bridges or designing projects that allow people to have more prosperity and face fewer tough trade-off decisions.

Measure change with a before, during, after program listening design

Impact is positive change over time. The number one reason Impact is hard to measure is that the people with the money and the power don’t want to wait for time to pass – they want to know immediately. But if you don’t ask people to describe life before an intervention, you will find it hard to measure change. Mathematically, it is impossible, though people often use weak data from elsewhere as a proxy for the baseline.

To be able to look at how a collection of narratives is changing over time, you need (at a minimum) to ask people before and after the program. If programs are ongoing, then you can ask periodically. Very strict researchers would ask the same people at regular intervals, but in the real world getting an organization to just ask two times (before and after) would be a huge improvement over what they have done in the past.

Journaling – if you have volunteers working for weeks or months, have them keep a journal. After, scan and datify the content as stories. So instead of two stories (pre and post), you would have a dozen or more stories from the same person about an issue. Growth is easy to see with journaling.

Focus groups – Informal discussions can be augmented by during a transcript of stories shared into data. Leave a tape recorder running, or use an app like dictadroid to convert it to MP3 and email for transcription immediately after.

Use our 60,000 story repository to build a reference collection for comparison

The best data is the kind that already exists. All you need to do is add your unique part and use our comparison tools to look at how peoples’ experiences differ. This isn’t as powerful as some of the other techniques, but it can be done for free, and sometimes done even when the project is being designed, before there are any “beneficiaries” yet.

Example: Two rape prevention programs or Planning to fight stigma after the ebola epidemic

mrembo-vs-sita-kimya-demog

These tools live on storylearning.org.

What next?

This is Part II of a blog series on story-centered learning and hypothesis-based international development. Read more!

Part I: Null hypotheses in international development

Advertisements

One thought on “Designs on Systematic Listening

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s