Kisumu community feedback focuses on conceptual evaluation

The plan

was to collect thousands of stories from people in the area and then give them to local organizations, so that they could learn what was happening (…or not happening). After collecting 25 thousand stories, we’re faced with a new challenge – how to visualize and summarize the overall picture, and how to deliver this information to the right people, with the right frequency, in a way that transforms how organizations operate.

Community Feedback Sessions

Where: In Kenya, we’ve held local NGO meetings in Kakamega, Bungoma, and now Kisumu. Next up: Kibera and Kampala, Uganda.

Kisumu Stories Map: Any story mentioning Kisumu was used, and they come from many places across Kenya. Also shown: Whether story was about an NGO, Government, an Individual, or something else.

Click to explore in BatchGeo

In Kisumu Collins from the Upendo Development Group organized a meeting that 9 men from 8 local organizations attended. (Where were the women? “busy” I was told. We will try to partner with a women’s organization next time.)

First I illustrated the picture of Kisumu overall…

Above: An experimental visualization showing what gets talked about in stories. You can parse the data by who (which NGO) or where (which town). Kisumu is shown.

Next I tried alternate ways to drill down through the over 900 stories from Kisumu…

Every word from all 900 Kisumu stories interconnected to adjacent words

And then I showed a more filtered word phrases map that one could better interpret…

The above version only includes words that have been used at least 5 times, along with adjacent words also used at least 5 times. By now it should be clear that major clusters are people, school, help, Kisumu, children, food, water, and education.

The difference between the bubbles and the word phrases is that the phrases of more universal categories that relate to many other sub-categories. Also – the Gephi software automatically organizes topics into clusters, which I have colored in manually for clarity.

A more common tool, the Wordle, gives similar information, but I feel it has less depth:

Both the wordle and the bubbles map miss the words education, because that is one word amplified by appearing frequently in connection with just a few other words in stories: secondary, primary, and free.

I then asked those present what specific topics they wanted to know more about. They answered:

  • Youth Development
  • Food Security, especially related to Fishing around Lake Victoria
  • Child abuse and protection
  • Disability

For “youth development” I opened up the online search engine and searched for that phrase. 17 results our of nearly 25,000.

“These are stories from ordinary community members,” I reminded them. “And they don’t write stories using the phrases you use in your grant proposals. What does ‘youth development’ actually mean?” I asked.

Black stares.

“You know, in English?”

Still, nobody could provide an intuitive answer.

“So why I don’t I search for ‘sports’ instead? Because every youth development organization I’ve met in kenya is using sports for social change.’

Now we had 218 stories about ‘sports’ and 548 stories with the word ‘sport’. If I wanted to go further, a more advanced search would include stories with any of these words and also football, tennis, rugby.

Since I was using the basic tools, here are a comparison of “sports” stories to those about “youth development”:

Youth Development (17)

Sports (218)

“What I see are two different perspectives in these stories. The word “sports” doesn’t even appear anywhere in the “youth development” stories.” I noted. Now that could either mean these stories are not about sports at all, or that those who are talking in these “youth development” stories are not speaking the same language.

So I dug deeper and later read every story. It turns out they’re not about sports but about various organizations that have the phrase “Youth Development” in their names:

  • UYDEL (Uganda Youth Development Link)
  • Youth Development Fund
  • Youth Development Programme from Barclays Bank (offers loans)
  • Youth Development Month Fund (National Youth Council)
  • MYSA (Mathare Youth Development Organization)
  • Kenya Red Cross Society (youth development group)

Here is one of the few of these stories that really described in any detail an actual youth doing some specific action:

One chilly morning, which was the 3rd day of August 2008. I looked at the ‘mtaro’ in front of our house with a lot of frustrations since it was our duty as a group to clean it. The duty was tiresome and boring.I went and founds my group bad started performing our duties as usuall.we had insufficient equipment of cleaning.Just in time ,there arrived two gentlemen .They were carring cleaning materials such as spades ,rakes and other materials .We looked the people in dis-belief  as they introduced themselves.They were from an organization called Mathare Youth Development Association (M.Y.S.A). They promised to give us more and more equipments  due to hard work .They also love us chicks and mash We appreciated their hospitality and thanks them in expanding our organization.

So the lesson is that if you search for “NGOese” jargon phrases like Youth Development, you’re probably going to get stories about organizations with those names. But if you want stories about various activities that can bring about “youth development” you need to put yourselves in the shoes of a storyteller.

Let’s look again at the wordle of all 548 ‘sport’ stories. What emerges?

Sport (548)

I noticed that transport, road, and government, only appear in this largest (548 story) set. School, activities, community, and time are common in “sports” and “sport” stories. Money and youth are in all three groups.

I wished I had a tool that could show the evolution of these wordles or word maps as we compare stories containing slightly different phrases.

My interpretation is “sports” vs. “youth development” is that the latter has a total lack of focus on youth activities, communities, and schools. Whereas “sports” is centered in a social or education setting, “youth development” focuses on jobs, work, and money.

Food Security and fishing around Lake Victoria

I started by searching for phrase “Food Security” and comparing it with a word I’d expect those most affected by a lack of food security to use in their own stories: “hungry.”

Food Security (67)

Hungry (115)

Hunger (204) – not shown

There is a clear difference between the jargon “food security” stories and the plain spoken “hungry” stories. “Food security” appears to be all about crops, farmers, water, maize, seeds, agriculture, harvest, production. I would be hard-pressed to find a more focused collection of words about agriculture and production. The NGOs mentioned in “food security” stories are:

  • Agricultural board of Kenya
  • Uganda National farmers Association
  • Shallow Wells International Movement
  • World Vision
  • Kenya Agricultural Research Institute (KARI)
  • Ssenyange Project

And for hungry stories:

  • World Food Program
  • Feed the Children
  • Unicef
  • Amref
  • Sadili Oval
  • Good Green Pastures
  • No one helping — 75 of 120 stories
  • Individual person –14 of 120 stories

(And all of these International NGOs account for only 18 of the 120 stories)

Stories about being hungry often mention children, schools, pupils, and Nairobi. No Agriculture topics at all. It would seem that real food security is an urban problem that starts with people have food and the money to buy it, or at least kids getting a square meal in school each day. This is a poverty-based “food security” rather than an agricultural-solutions one. And anyone dealing with this issue should be reading both kinds of stories to search for an understanding why a country with plentiful harvests has such a high rate of hungry people in Nairobi.

Once again, having a side-by-side comparison tool for this would help me, and I assume it would also help organizations that want to understand divergent patterns hidden in groups of stories. I guess I’ll get to work building it.

Examples of “hungry” stories:

In kibera many people are poor. The state of poverty is seen from the mode of dressing of the young boys and girls who put on tuttered clothes that are very old and dirty.some of these young children even lack what to eat and hence more into rich people dust bins to atleast scarage something that is edible to them to eat and even carry some home for their parents who are also tired and hungry to eat.

One evening as i was going home i found a lady at the bus stop who was also waiting for the bus.The lady asked me to help her if i had changed money.I gave her 10 thousand changed money.Close to her i heard her stomach gremble as if she was very hungry i thoughtfully checked in my bag and gave her a doughnut she thanked me and asked me how i knew she was very hungry.The bus came and we entered she bargained but the conductor refused to hear  saying she had to pay ten thousand i called her and added five thousand to her,she hugged & thanked me.From that time i got lessons that people have problems and they need help,from then i started helping people.

Fishing around Lake Victoria and Food Security

In spite of collecting all around the lake, only 47 stories mentioned “fishing” and virtually none also mentioned “food security” / “hunger” / “hungry”.

Judging by the lack of stories, I conclude we have not penetrated fishing communities around the lake.

Child abuse and protection

Found 228 stories with one of these phrases: “child abuse” “child protection” “child right” or “child labor.” For this data set, I wasn’t sure what the plain language version would be, so I analyzed all of these stories in patterns about who is talking about what from which kind of perspective using SenseMaker(R) from Cognitive-Edge.

All stories shown: Success vs Failure stories, Solution vs Problem or Need stories

Blue dots are stories from women, Red dots are stories from men.

When you slice this data into smaller pieces, looking at the AGE of the storyteller, you realize that most stories from young people (under 17) are from women, while most stories from 31-45 year olds are from men:

Under 17: Girls

Stories from people ages 31-45 are mostly men:

Another important question our scribes ask storytellers after they’ve written their story down is “what role did you play in this story?” Were you an actor, who helped make it happen? Or did you observe it happening? Or maybe you were affected by what happened. Again here, we see a striking pattern in child protection stories that should make local organizations sit up and re-think their issue engagement strategies:

Most child abuse / child labor / child protection stories from young people (under 17) are from girls who are actively involved in stopping it.

So I’ve parsed our 189 stories into just stories from people aged 0 to 16 (45 of 189), then into actors (29 of 189), and the combination leaves stories from all girls. It should also be alarming that one third of all stories from “actors” in child abuse stories are from youth and children.

So important lessons for organizations (my interpretation):

  1. Women are doing much more about child abuse than men.
  2. Young women are doing more than anybody else.
  3. Negative stories related to child abuse do not come out until after a scribe has been collecting stories for a while (see below).

Meta: How the number of stories a scribe has collected affects positive / negative aspect of story:

To gather these 25,000 stories, we work with over 3000 young people, whom we call scribes. We are always adding more. The only way you can manage a giant workforce with just 3 people is to use technology like SMS and meta analysis methods to see patterns in the stories.

As shown above, while both new and old scribes are getting “child abuse” stories, “failure” stories on this subject only came from scribes who’ve collected at least 100. This doesn’t mean we should work not with new scribes, but that we need to recognize some sensitive issues are very hard to gather data about. If it has taken 9 months of continuous listening to achieve this, imagine how much harder it would be if you did a one-time evaluation and left no feedback mechanism in place.

What can an organization learn this way?

These stories can be used to a help local organization learn about three things:

  1. Themselves (stories specifically naming that organization)
  2. Their community (studying all stories tied to a location)
  3. Some complex social problem they address (studying a concept)

The third thing is what I’ve done in this report. You can call it “conceptual evaluation” and I believe it is probably the most powerful use of large story collections. Stories from anywhere nationwide can provide valid insights.

We’re starting to see enough stories from a few towns to gauge community needs and priorities (thing #2). Stories about oneself would require a larger community-wide storytelling effort over a longer time to really work (perhaps 1 million stories across East Africa over 3 years). So far we’ve stories about over 1000 NGOs in Kenya and Uganda, but less than 50 have enough stories about them to really help an organization understand others’ impressions of itself.

Next post: Disability, Runaways, Step parents, and more.

This work has been funded by the Rockefeller Foundation and you can fund itself by giving to the Pulling for the Underdog Fund:

Give to support the storytelling project!

One thought on “Kisumu community feedback focuses on conceptual evaluation

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s