Crowdsourcing definitions in international development

I think international development struggles to describe what it means to people, as I’ve explained before here (dejargonifying) and here (whose perspective?) and here (fuzzy human rights?). This is a demonstration of how my wordtree tool can provide a unified view of very fuzzy and complex concepts. In this case I’m using it to define human rights, women’s empowerment, democracy, and to define the needs of children and animals — from the NGO perspective. But you could also use it to define your organization’s impact from the beneficiaries’ point of view too, if you were mapping stories instead of project documentation. (examples of this below)

Data: thousands of GlobalGiving projects and reports, grouped by theme.

By pulling from our database of literally thousands of project descriptions and project updates that match one of the 15 themes on our website, we can let hundreds of project leaders describe the meaning of such concepts as human rights, gender, womens’ empowerment, and the needs of children. In each example, I am contrasting words used in projects from two different themes, color coded by whether each word relates more to one group or the other. The shape, size, color, and locations of all words in these maps are determined by an algorithm.

Human rights projects (red, n=294) vs democracy projects (blue words, n=60)

Purple words are frequently used by both groups of projects.

In this blue branch, voter rights are relevant to democracy but absent from human rights projects.

Human rights projects talk about HIV, but democracy rights don’t.

What do they share in common? Advocacy, law, women, students, and schools are important to activities and project leaders working on human rigthts or democracy. Not much else overlaps. For an example of stronger overlap, look at 653 gender (women’s rights & empowerment) projects against 1110 children projects.

Gender (women’s rights & empowerment) projects (n=653) vs  Children projects (n=1,110)

And here is a little bit sparser version of the same map. Health, mothers, families, children, lives, social, home are words found in both gender/women and children of projects.

This is a much bigger sample of data, representing a broader perspective on what organizations believe women and children need. These two themes overlap quite a bit, as a lot of the words in the middle are purple.

What doesn’t overlap:  

Children projects: music, clean water, HIV, AIDS, malaria, sanitation, street children, orphans.

Gender projects: domestic violence and shelters for women, small business, human sex trafficking, and economic empowerment.

Animal projects (red, n=174) vs all GlobalGiving projects (blue, n=7200)

Animal projects are the most popular type on GlobalGiving, in terms of donors searching for something to support. They aren’t really similar to the other categories, so I compared them to all projects in this map. (note, a word needed to appear >110 times among GG projects to be included as blue node):

This gives a fair representation of the many kinds of animal projects (n=174). The blue sections are the rest of GlobalGiving’s projects, and represent a pretty good overall summary of the types of interventions that organizations do worldwide.

What’s the point?

Why should your interpretation of these maps be useful?

• This can be done with totally unstructured, qualitative information. But if you have a lot of data, the noise (individual differences) will cancel out, leaving a more unified perspective.
• This represents a huge number of perspectives. Instead of having a focus group of a dozen, you can have a crowd-perspective representing thousands of voices and yet all of them can be heard when they tjotjog. (Tjotjog is when many people are saying the same thing in slightly different ways, thus reinforcing some common principle without copying exactly the same idea. In sense, tjotjogging is when people are riffing off each other, not retweeting each other.)
• This can be more robust than even quantitative data. Without structure, there’s no way to guide users into saying what they think you want to hear. None of these participants were answering a survey – they were describing their project in the best way possible to attract individual donors.

Postscript: Community Level Evaluations

I said at the top that you could use this to look at the impact of one or a few organizations on a community. Since impact is hard to define (and I think impossible to measure in the causal sense), here is an operational definition of impact I will be using:

Social Impact:

Organizations positively affect society when they do what the community wants.

If this isn’t impact, then at least it is Democracy. But I think that Democracy is a pre-condition for social impact.

My assumptions:

• Communities talk about things that matter.
• They know what needs to be done.
• Organizations will have an impact on the community IF they do these things. (I don’t think this makes anyone instantly effective at doing them, but how effective can any organization be that doesn’t focus on what matters to the community? Doesn’t the community influence the effectiveness of projects meant to intervene and transform it?)
• Alignment: One expects to find the greatest social impact where organizations and communities share common goals and interests.

So this wordtree map allows one to see what organizations do focus on. The “evidence” comes from their communications via GlobalGiving to past and potential donors. This map of NGO ideas (red) is superimposed on a collection of local stories and themes from stories (blue).

Community Overlap: 18 GlobalGiving projects (red) in Kibera vs 3,150 stories (blue) from Kibera (Kenya). Pink are the overlapping words.

Below: same map, only zoomed in a little to make the center more readable.

Overlap: girls, youth, schools, election violence, school fees, skills, training, secondary school, teaching, learning, clinic (pink are areas of strong organization-community alignment)

Kibera Projects: leagues, film, tennis, teams (these are mis-alignments, where an organization discusses something that the community is silent about). Most of the red dots are names of organizations or people who work at organizations.

Kibera Stories: Drug abuse, drinking,  rape, toilets, water, clean environment, feeding program, HIV/AIDS (these are missed opportunities for organizations to make a bigger impact!)

Overlay: All Kenyan GlobalGiving projects (n=316) vs all stories from Kenya (n=30,741)

This zoom-out view should give you a feel for the total amount of overlap (pink) and the network of GlobalGiving partner organizations’ reach in Kenya. Compare it with the Ugandan version below.

This map is too big to digest (almost 2000 nodes!), but I put it up to give you a sense of how much data you can pull from for local alignment analysis.

Uganda GlobalGiving projects (red) (n=259) vs Uganda stories (blue) (n=15,796)

First, the really big picture – notice that GlobalGiving has less reach in Uganda, as fewer nodes are red or pink. Also, INCOME is the biggest topic in Uganda for both projects and stories.

And now, what’s that pink chewy center made of?

Overlap: income, HIV testing, counseling, HIV/AIDS, women’s health, orphans, education, community, school, latrine, hunger, water, sustainable development, microfinance, mosquito nets, hospital, grandmothers, grandchildren, police (pink are areas of strong organization-community alignment)

Uganda Projects: drama, skills, training, vaccine, sexual and domestic violence (these are mis-alignments, where an organization discusses something that the community is silent about). Some of the red dots are names of organizations or people who work at organizations.

Uganda Stories: rape, early marriage, pregnancy,  girls, malaria, scholarships, financial assistance, talents, drugs, farming, agriculture  (these are missed opportunities for organizations to make a bigger impact!)

Some of the red-blue mismatch is due to storytellers and organization staff using different words, but many of these words reflect totally different priorities. What amazes me about this algorithm is that nearly all of the words are relevant to decision-makers, yet none of these words have been pre-filtered. They appear solely because they emerge from many stories that use the same words together. (I do filter out common english words like “more” and “the”)

Next task: building a do-it-yourself tool so that any organization can get program level community feedback along these lines.