Visual aid for the story-based program evaluation method

One of the questions I get asked most about the GlobalGiving storytelling project is how will we use this to do program-level evaluations for ourselves and our partner organizations?

For two years we’ve tried a lot of tools for doing this. Here is one more that I built, and I think it has promise. These illustrations are computer generated maps of the overlap between how an organization describes itself (to donors) and how storytellers describe the organization or some relevant social problem.

Finally! Something that provides a lot of detail on the “WHAT” aspect of the 6 questions journalists care about (who, what, where, why, when, and how), and evaluators should also care about.

How storytellers describe ANPPCAN – a typical Ugandan local NGO

African Network for the Prevention and Protection against Child Abuse and Neglect (ANPPCAN) describes itself as a “pan African network that promotes child rights and child protection in Africa.”

This map is gray because I only have one source of information about them. ANPPCAN has not joined GlobalGiving, so I don’t have a track record of how the org describes itself to contrast with the 27 stories that we collected about them. From their website, I can quickly see that they have an inflated self-image, using the phrase “pan-African network” when all 27 stories about them came from Makono or Jinja, Uganda.

ADDING COLOR WITH AN ALGORITHM

In the maps below, each node (or word) has a color. Red words come from the organization’s mission, description, projects, and project updates on GlobalGiving. Blue words come entirely from related stories. Shades of purple overlap both groups to various degrees.

Vijana Amani Pamoja vs. Mrembo Project Stories

First, the wide-angle view.

I assume that good organizations communicate their purpose clearly. This map makes it quite clear that their donor communications have little in common with their project beneficiary stories. This org (VAP) went out and collected stories about their MREMBO project themselves, and were pleased to find out just exactly what adolescent girls were or were not learning after participation. Here’s more detail on that:

What stands out is that there are very few purple nodes. What overlaps here are the words aids, time, and program. The rest of the map shows that VAP can do more to describe their projects to donors from the perspective of the girls who are participating. There is still a lot more ‘drilling down’ one can do.

Since the goal of the mrembo program was to teach girls about female adolescent issues, we can now compare stories from mrembo girls to stories from all other storytellers that discuss these issues (early pregnancy, rape, HIV, and abuse):

Stories that mention ‘mrembo’ vs stories that include the words ‘rape’ and ‘prevent’:

NOTE: Mrembo stories are in blue. Rape + prevent stories from everywhere else are in red. Overlapping words are pink/purple.

So, in your opinion, is there enough overlap between these two sets of stories?

I would say yes. Many of the issues they intended to cover are mentioned in blue or purple, and most of the red words are not relevant. This does reveal that the Mrembo project could do more to talk about violence, and treatment (for rape or assault?). It also reveals that people outside of the mrembo project fail to make the association between HIV/AIDS and rape. Pregnancy and marriage are also program-focus ideas.

What about mrembo stories vs. rape, marriage, or unwanted pregnancy stories?

This map suffers from an order of magnitude difference in sample sizes (61 mrembo stories compared with 922 stories about rape, ‘early marriage’, or pregnancy). This sample size difference means that we can expect blue nodes to remain blue. Anything purple has really been emphasized by the mrembo stories.

What Nancy, the project manager, can do with this map is still get a clear picture of what related issues are lurking out there that can be addressed in future after-school life skills classes. Notice how child abuse emerges as a related issue.

And of course, there is another useful benchmark – another organization was running an anti-rape program targeting men. With hundreds of stories about Sita Kimya, you can do a head-to-head comparison:

Mrembo girls’ stories vs Sita Kimya guys’ stories

Because we have dozens of meta-data elements tied to these stories, we can apply any filter we want. In this case, I’ve only included stories told by girls that mention ‘mrembo’ (beautiful in swahili) project in red, compared with ‘Sita Kimya’ (I will not be silent!) stories from men in blue. As with all my other tools, the lack of connection between rape and HIV or marriage continues to be obvious among the hundreds of Sita Kimya stories from men.

Also note how men talk about ‘rape cases’ whereas girls talk about ‘rape’ in ‘school’. 

Trans-Nzoia Youth Sports Association (TYSA)

We have over 1000 stories about TYSA or Kitale (the town where they are based). This map contrasts TYSA (how the organization’s informs donors via GlobalGiving) with related local stories :

Without the word labels, it’s pretty clear that the overlap is spread out more evenly for TYSA than for VAP. When you add the labels, you can see that TYSA overlaps with many of the key local issues:

I’ve followed the work of TYSA for several years (I admit, I’m a donor.) and yet I just didn’t realize how focused TYSA has become on girls. I knew that education and youth were pillars of their mission, but they are very clearly targeting girls, family, parents via sports programs.

TYSA is the image of a true community based organization.

But wait! There’s more. On the periphery we can see distinct branches of red and blue, representing community efforts TYSA has tried and community focal points that TYSA has never tried to address:

In 2010 TYSA made an effort to start a community awareness conversation about the Kenyan constitution. Their education effort appears in red, but did not leave a lasting impression on the storytellers. In contrast, these storytellers are focused on HIV/AIDS, marriage, husband, wife, pregnancy, and dropping (out of school). Training is talked about in stories more than TYSA mentions on their project or reports to donors.

TYSA Youth Education branch

TYSA: mentions “teen mothers” and community mentions “pregnancy”. Community also talks about Wasichana Tunzaweza – a “Yes Girls We Can” campaign that resonated locally but was not described in the same ways to donors.

This part shows that education remains a project focus strongly aligned with community stories. Post election violence and peace are also a part of the picture. Clearly, TYSA tackles a wide variety of issues, and many of these are reflected in stories. These stories included all of the region, and not just those that mentioned TYSA, and yet the overlap is strong.

Nyaka Aids Orphans Project of Uganda

I admit I’m not familiar with this organization, but they were on the GlobalGiving homepage today, so I analyzed them against stories that mention them or aids or orphans (literally more than 7000 stories):

The number of overlapping (purple nodes) is less than for TYSA, but it is there.

From the center of this map, I conclude that Nyaka Aids Orphans Project focuses on secondary school, student needs, but also clean water. They do not address security, which appears to be a major concern for storytellers that mention HIV/AIDS or orphans. A more detailed analysis would break the reference stories down to just the town where this organization operates, and I would also meet with the org and encourage them to collect a bunch of stories. In that case, I’d want each storyteller to collect TWO stories. One can be about this project, but the other should be about anything else happening, so that a map like this would better reflect community priorities alongside the organization’s meaning to that community.

Why I like wordtree maps

1. They provide actionable information. In these examples, a local project leader can see what they have done that resonated, as well as what many people are talking about locally that they are not addressing. Allows for project evolution based on community feedback.
2. Instant, automated feedback. These maps are as easy to generate as using google search. (I still have to build a web-version, but it can be done.)
3. No training necessary to interpret them. They’re not quite ‘self-explanatory’ but they are much closer than anything in statistics.
4. They can be auto-generated by an algorithm that won’t create statistical errors. Statisticians talk about TYPE 1 (false positive) and TYPE 2 (false negative) errors. Staticians teach people how to understand and calculate data in ways that avoid these errors. This knowledge has never diffused to the world with quite the success of, say, google searching. (Yesterday I saw a TV survey inofgraphic with the words “No margin of error” written below it! Really?) I believe an algorithm can be ‘smart’ and help us avoid making these statistical errors, so that people can just focus on the data and not the method. Wordtree mapping can automatically filter out nodes that are too insignificant to bother with, but it doesn’t hide adjacent information lurking in the wings of every development question.
5. More quantitative than wordles. Word clouds were a past trend in simple text analysis, and I’d like to think this is more useful because you can also trace the relationships between words. As you get farther from the center of these graphs, the words become less important. And words with bigger nodes are used very often within that part of the graph.
6. Universal: works with ANY narrative data collection, period. There is no required survey template. As long as you want to compare two groups of narratives that are in the same language, and both are in files, it can be done. In our case, we have a lot of bonus meta-data tied to stories, so we can compare how men and women talk about things, or how girls under 16 describe ‘success’ in stories that mention starting a business, and who also describe a solution to a problem in their stories.
7. Everybody contributes to the picture. However, to be heard, each storyteller needs to tell a story that overlaps with elements in other stories. This means longer stories get heard more, and there is a bit of topic-voting involved in the emergence of themes – like democracy without a ballot.
8. Provides macro and micro lessons. Overall, the amount of purple instantly tells you how much overlap two sets of narratives have. Drilling down reveals the exact themes on overlapping and non-overlapping branches.
9. First decent tool for mapping the WHAT in narratives. For a long time aid visualizations have focused on the simplest quantitative aspects of projects: when did it take place, and where? The complexity lies in understand WHAT took place, and eventually to whom did it matter? We’re getting at that question, and when I refactor this tool to map the WHO question, we may be cooking. It fits nicely with the three elements I think any real-time continuous assessment reporting much contain to work in international development:

• Core Alignment – How does an organization’s mission align with the needs of the community where it works?
• Missed Opportunities – When you map everything organizations do, what is left that communities talk about and nobody is addressing? Somebody will need to step up. Finding the missed opportunities to make a bigger impact must be part of the community report.
• Finding Local Implementing Partners – to FLIP the system, you need to find out which organizations are doing what in your community, and reach out to them. A community reporting system can FLIP the system.