A Rosetta Stone is now available for NGO sentiment analysis

Until the Rosetta Stone was discovered in 1799, no one could read Ancient Egyptian Hieroglyphs. Because the inscription on this stone is identical in three languages, we were able to decode this ancient script.

rosetta_stone

By analogy, I am publishing a dictionary that allows us to understand what people on the receiving end of international aid really mean when they are given a chance to tell stories about how organizations have affected their lives. It works because the GlobalGiving Storytelling Project collected such a large sample of beneficiary feedback about every sort of community effort that we can reverse engineer what people mean in other contexts.

Building the word-tone dictionary

  1. Starting with the over 60,000 stories we’ve already collected from Kenyans and Ugandans about NGO work, I pulled a dictionary of 100,000 English words and queried the collection for stories that contained each word.
  2. Each story is associated with a series of mapping questions about what happened. Was it positive or negative? These outcome mapping questions allow me to associate specific words with specific outcomes on a range from positive to negative. For example, if everybody who tells a story about “measles” assigns the outcome to negative (the person wasn’t cured), the word “measles” would generally be a negative word in other NGO contexts.
  3. There are many kinds of positive and negative outcomes in the data already. We asked, “Who Benefited?” nobody? the wrong people? or the right people? Or “who did you feel about your story?” So even with just 100 stories that use a word, we often have several hundred data points averaged. As you would expect, most words are not strongly positive or negative.
  4. I then filtered out any word that wasn’t used in at least 100 stories, so that the remaining 1944 words (of 100,000) are pretty reliable as a reference dictionary. I will probably publish a larger dictionary with the remaining words if people ask for it in comments below.
  5. I then normalized the scores on a range from roughly -500 to +500, centered around zero by “turning up the gain” on the negative feedback. (Ask me how in comments in comments if you care). This step allows the data set to be used in other contexts, such as the import your own text analysis tool, where we don’t know whether stories had happy or sad ending.

This reference dictionary allows anyone to take a quick glance at the overall sentiment in any unstructured language from people who are affected by international development, based on how tens of thousands of people have used the same word previously.

Download your free word-tone dictionary

Click to download the 1944 word dictionary either as a CSV or a python pickled dictionary.

pickle

csv-xls

 

 

 

 

(Rename the files afterwards)

Results

If you plot all the words in excel and their sentiment scores, they look like this:

normalized word distribution from tone dictionary

If you turn that plot on its side, it will become a normal distribution, like this:

normdist

The corrected plot is centered around zero. That’s good. It means that these words are a good mix of positive and negative sentiments. I chose to adjust the raw positive-negative scores because there is such a huge positive-bias in all NGO feedback that it becomes ridiculous just how skewed the stories are, compared to how much peoples lives are really affected by these efforts.

If people really benefited as much as they say they are, we would have no poor people left.

Case in point: What word is outlier at the top of the chart?

full scale normalized word distribution from tone dictionary

Give up? This word has a positivity score of 10,125 after I corrected the data. The score of +10,125  is a measure of how consistently that word appears in positive success stories versus negative failure stories. A word with a score of zero is neutral, or used in stories with mixed positive-negative outcomes. And because only words used in at least 100 stories are used, these dots are not like to switch sides (or signs) if we repeated this experiment a different story collection from the aid world.

Still don’t know the mystery word?

Here is the answer:

annotated full scale normalized word distribution from tone dictionary

The outlier is the word ‘organization.’ People are very eager to tell positive stories about organizations. Literally thousands of times more likely to be positive in stories with the word ‘organization’ than in stories that contain the words that fall along the zero line of the chart.

Previously, my other means of measuring positive bias in stories concluded that people tell 10 to 30 positive stories for every negative story, across all 60,000 stories. By this measure, the positive sentiment in stories that include the word ‘organization’ is even higher still. Positive bias is a real problem. But using the word dictionary I’ve published, you can find the negative sentiments within a sea of rosy feedback.

My interpretations:

  • The most negative words were “came” and “time.” As in, “one time these people came to our village…” That meta pattern is quite alarming. I just finished reading Bill Easterly’s “The Tyranny of Experts” yesterday, which is all about getting the outsiders to leave people alone and instead focus on advocating for the rights of poor people. This pattern is consistent with the failure of outsiders to come into a place on a “one time” basis and make any sort of lasting positive change.The implications of these outlier words should be a wake up call to the aid sector.
  • People are more honest in Kibera and Uganda. Slum life in Kibera is poor, and people are ready to honestly talk about it. But across Uganda, people are almost as positive about everything as are people who talk about an “organization.”
  • Narratives are positively biased in the development world, but there is no reason to believe that numbers are somehow free of this bias. In any kind of survey, when some asks the citizen, “how much money do you make?” or “how many kids do you have?” they get back wrong answers, and always wrong in the direction of what a person knows the organization wants to hear. There are documented examples of women under-reporting the number of kids they have to Millennium Development Goals surveyors in Uganda because they knew the “smaller families” was the outcome measure outsiders were looking for. Likewise, people lie about income and say they are poorer when money might have handed out, or overestimate their income in surveys from micro-loan foundations that use this “success metric” as the basis for granting them larger loans.
  • You could throw up your hands and blame other people for lying, but I prefer to treat this as a symptom of the larger disease: Our programs are generally not making life better, and the only way to make life better is to play the game to get as much immediate aid as possible. No one has ever proven that putting money in a person’s pocket makes them poorer in the short-term. Yes, in the long term, they could become poorer, but poverty has a tendency to focus people on short term gains.
  • The difficulty is that there are two kinds of positive stories – the ones where things really turned out great, and the ones where they are saying good things but they’re still not happy about the outcomes. This one method alone doesn’t do enough to tease out these two kinds of positive, but when combined with other lines of evidence, other structural aspects in the narratives, it is possible to tell the difference between authentic praise and manufactured praise. For example, check out my first attempt at this.

Quick scan of the most positive / most negative words

Most negative words

came -3189
kibera -2540
time -2406
two -2244
kenya -2094
take -1755
come -1755
really -1478
place -1380
long -1375
decided -1368
years -1339
person -1292
lack -1265
mother -1208
green -1206
father -1182
able -1138
man -1116
ago -1092
brought -1081
certain -1040
things -1029
bad -979
told -943
water -923
belt -908
saw -907
going -907
hard -902
bring -867
see -848
young -848
girl -839

Most positive words

organisation 10128
uganda 4192
provides 2774
providing 2652
children 2501
standards 2286
done 2194
development 2110
poor 2050
helps 1901
living 1899
support 1882
community 1808
orphans 1707
giving 1595
improve 1548
district 1446
aids 1409
agriculture 1373
health 1362
education 1349
farmers 1312
gives 1309
counselling 1293
mbarara 1284
provided 1274
given 1272
helping 1250
association 1245
materials 1241
women 1219
vision 1202
world 1190
role 1169
town 1113
seeds 1113
scholastic 1052
care 1038
save 1025

Examples of organizations using Story-Centered Learning in their work

gg_storytelling_logo_explained

GlobalGiving has got a new crop of partner organizations trying out our storytelling method and adopting it to their local context. In every case, they try to get two stories from each person, and one of these stories can be on a narrower subject. The second story is very opened ended, about some community effort they know about. Here are examples of how each organization is adopting the storytelling method to their needs.

  • Center for Peacebuilding (Bosnia): We develop peacebuilding programs to foster peace and reconciliation among different ethnic and religious groups in Bosnia and Herzegovina. Our activities are designed to bring about comprehensive social change focusing on youth.

Their story prompt: Talk about a time when a person or organization tried to help someone of change something in your community. What happened?

Reflections from the organization’s peace ambassador (copied from her blog):

“I thought that many community organisations would not have the capacity to do so much work… particularly since we already have our own internal evaluation system. On the other hand…  this made me even more committed to find local capacity for CIM to fundraise on globalgiving… In this way, we can ensure that feedback collection will happen on the ground, and I can still handle the communications and analysis from anywhere in the world.

I see globalgiving raising the bar by raising the standards for local organisations in terms of programming. So indirectly and slowly, globalgiving could create a network of grassroots organisations that have a professional level in fundraising, evaluation, and programme development. The tools they need are easy to use. The points based system is somewhat competitive. The rewards they get are too good to move away from. Those who will be serious about development work, will have adapt, improve and sustain an impact on the ground in order to keep getting the benefits.”

  • Encompass – the Daniel Braden Reconciliation Trust (UK): Encompass works to bring together young people from different cultures and backgrounds, supporting them to become more understanding and tolerant of each other while giving them the skills and confidence to promote intercultural understanding in their communities. This storytelling project is carried forth by youth in the UK, US, and Gaza (Palestine).

Their story prompt: Please tell a story about a time when a conflict arose because you had to work with someone from a different background (religious, cultural, ethnic etc.) to yourself.

Their revised story prompt: Please tell a story about a time when a person changed someone else’s perception of them or challenged a prejudice or misunderstanding.

  • Guitars in the Classroom (USA): Since 1998, Guitars in the Classroom (GITC) has been inspiring, training and equipping classroom teachers to integrate music making across the academic curriculum through “song-based instruction” so students of all ages have educational, musical access & opportunity at school every day. Our work prepares educators to lead music, employing it as a dynamic tool for reaching all learners, teaching all subjects, and building character, creativity and community.Programs & materials are free.

Their story prompt:  We are excited to learn about how your experience with Guitars in the Classroom has affected you personally and, if you are an educator, professionally. We also hope to learn about other experiences you have had as a volunteer or participant with another charity. Thanks for participating!

  • La Reserva Forrest Foundation (Costa Rica): La Reserva Forest Foundation is a Costa Rican non-profit working to restore and preserve native tropical forests, dedicated to creating “tree bridges” linking isolated forest islands using volunteers and the local school communities, and fighting global warming through various carbon neutral projects.

Their story prompt: Please tell a story about a time when you had to choose between protecting the environment and maintaining a livelihood. Include if/how individuals or organizations were involved in the conflict.

  • Partnership for Every Child (Ukraine): Our vision is the world where every child grows up in a lovely and secure family. Mission. We professionally assist families, communities and governments in their work to ensure the rights of every child to live and develop in safe and secure family environments. Our main focus until 2015 is to prevent separation of children from families and placement in institutional care; support and strengthening parental capacities of vulnerable families; support to children leaving care.

Their story prompt: (They plan to use the standard story question to learn about youth needs in their program)

  • Tanzania Development Trust (Tanzania): The Trust Deed of 1975 says “The objects of the Trust shall be to relieve poverty and sickness among the people of Tanzania by means of the development of education, health and other social services, the improvement of water supplies and other communal facilities and the promotion of self- help activities.” Interpreting the Trust Deed for the needs of the 21st Century we add: “In making grants, the Trust tries to promote equal opportunities and projects which improve the environment”.

Their story prompt: Standard story prompt

  • Vacha Charitable Trust (India): Our mission is to focus on issues of women and girls through educational programmes, resource creation, research, training, campaigns, networking and advocacy. Our vision is of a world without exploitation, oppression, discrimination and injustice against women or any other section of society.

Their story prompt: Standard story prompt

  • Vijana Amani Pamoja (Kenya): VAP’s mission is to integrate social and economic values through football/soccer by creating a proactive health environment.

Their story prompt: Standard story prompt

  • London Youth organization helps thousands of teens in the city. They measure impact as improved self-confidence, educational attainment, and long-term community involvement. Their programs help young people get “back on track” and help them find fulfilling careers.
Their story prompt: Please tell a story about a time when a young person tried to change something in their area
  • An NGO in Botswana works in many communities to curb gender-based violence. Instead of asking about the issue directly, they are trying an indirect way to learn about underlying issues through storytelling.

Their story prompt: Please tell a story about a time when a person or an organization had a conflict or disagreement or problem with money.

  • In Japan, IsraAid is running a storytelling project to gather stories about the Japan Earthquake and Tsunami, and how areas are recovering.

Their story prompt: Standard story prompt

Globalgiving Storytelling Project-Bosnia Edition

Originally posted on almondsasdiamonds:

Over the past three years storytelling has become central to most of what I do. I never paid too much attention to it before, but since first coming to Bosnia I have begun to purposely acknowledge how both myself and others around me used it. I had positive experiences: listening to inspirational stories that in one way or another changed my life and the path I followed, and negative (but constructive) experiences: witnessing hopelessness, trauma, anxiety, anger, disillusionment.

From an academic point of view, this conflict transformation approach was ruined for me, and I surprised myself that despite my very negative experience with that specific module, I have not given up on it yet, on the contrary, I am eager to learn both in formal and informal ways.

I happen to be back in Bosnia. Together with the CIM directors, we agreed to apply for a grant from Globalgiving for a…

View original 971 more words

Understanding street children: What can you do with narratives?

Recently I was lucky enough to be invited to present findings about street children at the University of Manchester in UK. Eleanor Harrison (GlobalGiving UK CEO and former director of a street children program in Kenya) yielded her time to me, and I gave this presentation, virtually analyzing all the data on the spot.

Not being able to prepare in advance was a blessing. It helped me emphasize that we’ve come a long way in our quest to build simple, instant, intuitive tools for analyzing narratives and their meta data. These images are screen shots of the tools I’ve described previously at djotjog.com/search and djotjog.com/report.

What can you do with noisy, minimalist, or totally unstructured narratives?

A demonstration with stories about street children 

street children headerimg

First, it provides a top level visual summary

You don’t even need to know much English. Just look at the pictures and you’ll get an overall idea. Everybody talks about street children, but kids talk about it a bit more than the rest.

street children summary 1173 of 61000

Analyze the point of view:  “Actor in the story” versus “Affected by events”

When people tell their own stories, they turn out more negative. But young boys are more positive and young girls are more negative.

compare street children actor vs affected

A story’s point of view affects the depth, nuance, detail, and likely “data” that can be extracted:

Street children stories have more “I” stories that we would expect, pulling stories from our collection of 60,000+ at random, so the “I” is bigger.

street children affected pov map

A Wordtree is an unsupervised algorithm that works with any text

Djotjog.com/search generates these for ANY search result in seconds. Here are branches from hundreds of street children stories.

street children street kid run away ran away

Zooming in you can see how words and phrases connect to form ideas: An old man frequently interacting with a young girl, with money being part of that sentence in the story.

street children wordtree man did street children wordtree food life

Food and life also form two branches of the story.

We can look at how stories intersect with mentions of specific organizations.

jitegemee drill down

Contrasting stories with different endings can yield more program-level design insights. (Using stories about success or failure we map outcomes)

This is the most interesting snapshot of them all. Failure stories are more complex than success stories. Wordtrees look for associated words within sentences. Success stories have branches that fan outward because the concepts are simpler, less interwoven. Failure stories are a chaotic birdsnest of interwoven social issues.

In failure stories, each storyteller tends to describe the events using similar words but in a different order so the map is simply more complex, more interesting.

As organizations, too often we focus on learning from the success stories – but it is the failure stories that offer us the keys to understanding the problem.

street children success vs failure stories

Compare two story collections

Side by side: Stories from Pennsylvania (N= approx 60) and East Africa about street children / run aways are remarkably similar in the issues they address.penn state run away narratives N=118

And here is a map of about 50 blogs posts about street girls in Cairo form Nelly Ali’s blog:

nelly ali street girl map

You can do this yourself. Go to djotjog.com/search and type

"street children" or "street kids" or "run away" or "ran away"

in the search box, then hit “fetch stories.”

street children search box

Note: you can export the narratives and meta data as a CSV file, for further exploration in a tool such as SAS, SPSS, or BigML.com.

Postscript: What MORE can you do with narratives? Try BigML.com

I imported hundreds of stories that mention ‘child abuse,’ ‘child labor,’ or ‘child protection’ into BigML.com and used their version of a text-mining function to build trees that map outcomes in stories. The types of outcomes are how people felt about the story they told, such as ‘inspired’, ‘horrible’, or ‘happy:

Horrible stories

bigml child abuse inspired

Inspiring stories

Have mixed outcomes, and don’t mention your aunt. But more importantly, they are not from people who were “affected” by the events in the stories they share.

bigml child abuse inspiring

Important stories

Are commonly organization-centric narratives.

bigml child abuse important

Happy stories

Have a very long pattern of words in them. This example begins to demonstrate the true scope and nuance of “success” that lies in narratives, and which cannot be captured with a simple “did the right people benefit” question.

bigml child abuse happy

Changing your point of view to tell a more compelling story

For years I’ve wanted to write an algorithm that would predict whether a story is emotionally compelling or not. This would be a major breakthrough for natural language processing. It would also allow us to automatically rate most of the narrative content on the Internet.

While I am not there yet, I am making progress. Using the wisdom of James Pennebaker from The Secret Life of Pronouns I was able to write a story point of view detector that seems to finally work. Not only does it tell what the story’s point of view is, it can also assign a confidence score to its prediction, as well as reliably detect stories that lack a dominant point of view (result is “none”),  or share two alternating points of view (result is “mixed”).

That’s what goes into any good algorithm. If asked to decide between A or B (the simplest choice), there are actually four possible answers: A, B,  Both, or Neither.

Storytelling: Seven Points of View

After many rounds of testing, I discovered 7 points of view:

pov_chart

This may come as a surprise to anyone who was taught about only three point’s of view (POV). Based on the evidence that people respond differently to these different points of view, they are distinct.

Emotionally Compelling: Mixed or “I” stories

The most powerful point of view if you want to tell an emotionally compelling story, according to The Secret Life of Pronouns is the “mixed” perspective, followed by first person singular (“I”) stories. “Mixed” perspectives alternate between two points of view. And after you realize this, it’s obvious. If you want people to connect with you, and find your point of view credible, you need to spend a little bit of time telling the story from their point of view.

What 98,447 stories can teach us

Below is a chart showing what fraction of stories are told from each of these perspectives for three large bodies of narratives. GlobalGiving requires that every project leader report back to donors four times a year for every project. The report is supposed to be informal, conversational, emotionally engaging blog-type writing. And since 2010 we’ve been collecting stories in East Africa written by regular citizens about some specific community effort they witnessed — the Storytelling Project.

Lastly, for the last two years I’ve been getting a “story of the day” by email from a project of my favorite Artist, Jonathan Harris of I want you to want me” fame. His storytelling site, Cowbird.com, manually curates good stories from the thousands of submissions. Their 812 stories are a positive control group in this experiment, answering the question:

“From what point of view should an emotionally compelling story be told?”

It stands to reason that all of the 812 cowbird stories are good, and their point-of-view (pov) patterns are reflective of what makes for good storytelling as a rule. Let’s compare these three groups:

GlobalGiving Project Reports (N=35,689) East African Community Stories (N=61,946) Cowbird.com Story of the day (N=812)
fourth “this org” 0.35 fourth “this org” 0.29 first singular “I” 0.514
first plural “we” 0.268 third plural “they” 0.197 third singular “he” 0.112
third plural “we” 0.126 None (no pronouns) 0.18 fourth “it” 0.108
third singular “he” 0.098 third singular “he” 0.117 None (no pronouns) 0.078
second “you” 0.069 first plural “we” 0.084 first plural “we” 0.07
first singular “I” 0.046 first singular “I” 0.078 second “you” 0.057
None 0.04 mixed 0.049 third plural 0.033
mixed 0.003 second 0.007 mixed 0.028

As you can see from the table, there are dramatic differences. A graph of this makes the differences clearer:

pov chart project reports vs community stories vs cowbird

How POV affects story quality: Three major conclusions

oneFirst, 51% of Cowbird stories are first person singular (I, me, my, mine), compared to 4.6% of GG project reports and 7.8% of East African stories. If you want to reach people emotionally, only your own story will work. Instead of you telling his story, have him tell his own story with “I” pronouns.

two_2Second, Not enough (only 1 in 300) GlobalGiving project reports are told from a mixed point view. About 4.9% of East African stories and 2.8% of Cowbird SotDs have a more complex, mixed, alternating point of view. Had these reports been written to better reflect the beneficiary’s viewpoint, they could have raised 50% more money from donors (see below).

threeThird, too many GlobalGiving project leaders have a “fourth person” pov perspective. “Fourth person” is my name for stories that lack any pronouns at all, or contain a lot of definite articles (a, an, the). They tend to focus on objects over people and relationships. Fourth person (in my algorithm) also uses more organization only jaron (such as the words “ngo”,”cbo”, and “foundation”) than pronouns. All of these make for reports that read like cold blooded reports and not warm, personal, emotionally compelling stories. And if you read on, you’ll see these report raise 30% less money.

Since the point of communication is to affect each other’s lives, we should drop the old style reports in favor or just telling the truth and being authentic. But changing your pronouns won’t make your story better, if it was never your story to begin with. You need to actually help people tell their own stories, and be a steward of their words. For too long we’ve let organizations harvest the words of others to further their (organizational) objectives, and this algorithm will finally allow me to out the worst of the bunch and force them to shape up.

chart how point of view POV affects emotion in story

Your English teacher mistaught you; get over it.

When we want to inspire, engage, comfort, challenge and connect with each other, we use short, personal, evocative writing, with a good deal of “I” words. Yet from an early age we are exposed to bad writing, reflecting outdated “beliefs” about what makes writing good. The evidence here shows that good writing is less “professional.”

Which world do you want to build today?  “Professionalized” language gave us global poverty, a financial crisis, and broken politics?

Creative and informal language gave us The Muppets, Neil Degrasse Tyson, and Doctor Who.

Follow-up conclusions:

give_nowChanging your point of view really DOES affect your ability to raise money with a project report

I took those project reports from thousands of GlobalGiving partner organizations and compared the dominant point of view in each report with the amount of donations that came from people clicking on the GIVE BUTTON in those reports. The results were striking:

 Effectiveness of project reports in raising money None third plural (they) fourth (this org, it) first plural (we) third singular (he) first singular (I) second (you) mixed
Total $$ raised 78 220 267 292 302 329 421 567
Donations per report 0.9 2.5 2.8 3.1 3.5 3.8 4.8 6.5
Average $$ per donation 24.9 46.7 53.9 55.8 52.4 51.9 58.0 60.5
Number of reports (N) 611 2519 7413 5881 2184 1056 1449 98

Notes: N = 25,337 published reports. Data includes cases where nobody gave any money after reading reports (23% of total). While reports don’t generate a ton of revenue (50% of reports raised less than $100) $1,077,000 was raised between 2007 and 2014 in precisely this way. This data represents the best example of giving tied directly to feedback loops in international development that I know of.

The results show that the best POV ‘mixed’ is more than twice as effective as the most common POV ‘fourth':

Project reports with a “mixed” perspective raise 111% more money and get 160% more donations than reports with “fourth” org-centric point of view.

Some caveats: These are not true “controlled” experiments. Nobody forced these organizations to adopt a first or third person perspective. Nor did we randomize what donors saw, as a true researcher might do. It could be that people who are naturally better at raising money tend to choose to use pronouns differently from those who don’t. And it turns out that women write these reports 2:1 over men. And what people talk about has a big influence over how much money one can raise. Here’s an estimate of how project theme affects donor giving after they read a fresh report:

animals gender disaster children hunger finance health climate edu rights econ devt sport
total $$ 1707 1656 1194 1044 1079 944 809 806 757 700 375 578
Reports (N) 928 2820 1315 4542 226 544 3397 712 4502 618 1185 284

The smartest way to fix your point of view is to talk to others and share their stories, instead of only writing from your perspective. And Globalgiving has for years been helping organizations listen, act, learn better. In fact we’re giving away money to encourage organizations to do this.

donations-volume-size-vs-pov-projrept

The Gap

There is a huge gap between how most organizations speak and what donors respond to. The green line near the center shows what fraction of stories have each of 6 points of view. The blue and red lines represent more donations and more money raised from a “you”, “I” and “you and I” mixed perspective.

(2) Humans are not very good at determining a story’s point of view

In order to validate the accuracy of this algorithm, I ran 406 of the 813 Cowbird stories through an experiment on Crowdflower. Crowdflower is a distributed tasking site where you pay people a few pennies to do a bunch of simple tasks.

In my task, the person would read two Cowbird stories, select the point of view for each, and then choose which story was the more “emotionally compelling” one. The secret life of pronouns predicts that “mixed” perspectives and “I” stories are more compelling to readers than “you” | “we” | “he” | “they” stories. So I tested our data set and had three people do the test for each comparison. Inter-subject agreement is an important part of seeing whether this task is easy or hard for humans.

Now I know from reading Cowbird that most of the stories actually are “I” stories, and my algorithm predicted 51% of these stories to be first person singular as I expected to see. The “mixed” perspective was much lower – only about 2%. But these are very short stories, and switching perspective isn’t as easy in 100 words, so 2% sounded reasonable.

The results from 406 human story comparisons:

Q: Select the story’s point of view (POV) from these 6 choices:

“I” –FS 118 0.29 vs algorithm: 0.514
“we” –FP 100 0.25 vs algorithm: 0.07
“he” –TS 64 0.16 vs algorithm: 0.112
“they” –TP 79 0.19 vs algorithm: 0.033
“the org” or “it”–4th 35 0.09 vs algorithm: 0.108
“mixed” –mixed 10 0.02 vs algorithm: 0.028

  • The humans were 40% LESS likely to choose first person singular than the algorithm, and three times MORE likely to assign first person plural to stories.
  • Both humans and the algorithm agreed when assigning “mixed” and 4th person perspectives.
  • Humans tended to want to assign stories to each POV more equally than a computer. (If given 6  choices, we seem to think that the stories SHOULD match up with categories equally. Same bias is seen on standardized tests.)
  • These humans were not very reliable, because the humans only agreed with each other 11 out of 406 times. 2 out of 3 agreed 50% of the time on what the perspective was.

Q: Of these two, which story was more compelling?

Same result. They agreed with each other 36% of the time. If choosing randomly, they would agree with each other 33% of the time, so that confirms that these Crowdflower humans are really very random and not worth the $16 I paid to test this data set on them. Had I asked 5 interns to do this, I would have gotten more agreement, because they care about agreeing with each other more than the $0.05 I was paying these folks to do a simple (though enjoyable) task.

It also confirms that seeing a story’s point of view is not so easy. If it was trivial, they would have agreed with each other more. Agreeing on which of two stories is more emotionally compelling is much harder, and likely impossible for any algorithm well at predicting what humans like. Even “human algorithms” are terrible at doing it.

A good story is more a matter of taste than of process, but people DO give to projects more often when stories are told from the right point of view – the beneficiary’s.

Try it yourself!

I created a simple tool for anyone to use. Paste your text into the box and it will analyze your point of view.

At djotjog.com/c/report/.

screenshot-by-nimbus (21)

Practicing what I preach

Old habits die hard. I ran my own algorithm against this blog post and it predicted that I am writing from a “fourth person” perspective with an 80 percent confidence rating.

OUCH! I soooo suck as a writer. Or so my computer tells me.

So I went back into this and changed some of my “you” and “we” statements to “I” statements and ran it again.

The Result: “fourth person”, 92% sure, 108 pronouns,  6.3% of text is pronouns

Pronoun counts by POV type:
[('fourth', 40), ('first singular', 31), ('second', 17), ('first plural', 10), ('third plural', 7), ('third singular', 3)]

The reason why I failed? I used too many “its” and “these” and “those” and not enough “I”s in it.

Oh well. [I'm] Hitting the publishing button now.

:)

Who’s Who of Organizations Ranked by Website Traffick

Alexa.com ranks all internet websites in the world based on how much traffic they get. I pulled a list of 3600 organizations and looked at their rankings in Alexa. These are the top 70 sites:

(Lower is better. i.e. Face = 2 and Google = 1)

Alexa Rank Site
10008 kiva.org
29295 wikimediafoundation.org
29719 autismspeaks.org
30470 worldwildlife.org/
35952 crc.uri.edu
42261 unicefusa.org
43454 nationalmssociety.org
45993 donorschoose.org
48806 oxfam.org.uk
52434 tigweb.org
53778 worldvision.org
54869 nature.org/
57005 rotary.org/endpolio
60860 livestrong.org
63450 globalgiving.org
65156 carleton.edu
68667 stbaldricks.org
70790 habitat.org/default.aspx
70862 nwf.org/
71303 japan.ashoka.org
74814 feedingamerica.org
75675 doctorswithoutborders.org
76757 bhf.org.uk/
83023 savethechildren.org
85042 inotherwords.org
85268 defenders.org
88988 nols.edu/
93188 thetech.org
100190 bestfriends.org/
101997 laneta.apc.org/desmiac
105183 uopeople.org
106004 pathfinder.org
110170 care.org
111217 alzheimers.org.uk
111265 us.movember.com/
118500 mercycorps.org
123953 teachforindia.org
130336 cry.org/index.html
144080 cff.org
144795 ccfa.org
149263 iucn.org/
156040 isa.org/
163908 lls.org
170456 psoriasis.org
177934 princes-trust.org.uk/
177965 heifer.org
184671 ijm.org
199416 bbbs.org/memphis
200255 worldpulse.com
202038 wcs.org
211427 americanhumane.org
240543 path.org/
242017 internationalmedicalcorps.org
245309 oxfamamerica.org/
263635 teriin.org
271126 documentary.org
274407 girlswhocode.com
280964 ineesite.org
281032 sustrans.org.uk
283065 kipp.org/
285384 us.tzuchi.org
291488 notforsalecampaign.org
292427 roomtoread.org
294625 janegoodall.org
300621 unfoundation.org
300746 womenforwomen.org
320616 liverfoundation.org
322227 humanityhealing.org
338673 sfaf.org/
354992 cityyear.org
357158 mariecurie.org.uk

That list is a little different than the typical who’s who lists for international development organizations. You won’t find BRAC or CHEMONICS or a whole host of UN agencies, or basically any organization that depends primarily on government support. This is a who’s who list of organizations that depend on the public for support.

Five Holy Books in five images

Since it is Holy Week, here are some rather intriguing visuals of the Quran and three competing perspectives on Jesus (The Canonical Gospels, Paul’s attributions, and The (non-canonical) Gospel of Thomas):

The whole Holy Quran as a wordle

whole quran wordle

The Gospel of Thomas
gospel thomas wordle

The Gospel of John

gospel john wordle

All sayings attributed to Jesus in Paul’s Letters

pauls letters - all sayings attributed to jesus - wordle

The Gospel of Mark gospel mark wordle

A while back I wrote a simple python script that would perform differential wordles (like I used in these two rape-prevention programs) but I lost it. If I rewrite it, you would be able to see an adjusted view of what these different stories emphasize about God, Allah, Jesus, etc.

Source: http://www.utoronto.ca/religion/synopsis/meta-6gv.htm

Or you can read my series on how the Passion Narrative relates to international development:

One: Empire – and the hierarchy of aid power

Story-centered learning: Gather “big data” before hypothesis testing

Reblogged from my ThinkNPC guest post:

In the last half-century thousands of scientists have rigorously studied the causes and risk factors in heart disease, but a single longitudinal experiment has revealed more about this disease than any other approach.

In 1948, researchers began tracking health records from all participants in the town of Framingham, Massachusetts. This was an observational study; they did not formulate causal theories or test specific hypotheses, but simply let nature take its course and observed what happened.

In 1960, they found a link between smoking and heart disease. In 1961, they found a link with cholesterol. And in the coming decades, they also found correlations with obesity, exercise, high blood pressure, hypertension, stroke, diabetes—virtually everything that now matters to clinical treatment.

So why aren’t we in the philanthropy world copying this approach—observing what’s out there and looking for patterns over time?

As a neuroscientist, I have a confession to make. My type have been responsible for propagating a lie they still teach in schools, that scientists always devise a hypothesis and test it in controlled experiments. This is simply not true. The human genome project mapped 3 billion base pairs before understanding what variation in the genetic code meant. human-genome

The drugs you take were “discovered” in massive drug discovery libraries using a screening process that quickly conducts millions of tests, rather than hypothesizing. 

My point is that complex problems cannot be understood from a pre-defined framework; what matters emerges most efficiently from open-ended data collection that is later organised and then studied.

We already create more information every two days than existed in the first two millennia of human civilization, and this pace is accelerating. However, the rate with which we convert all this “information” into useful “knowledge” is slowing down.

all-story-topics-2011It was with this problem in mind that we started the GlobalGiving Storytelling project. We needed to dissociate two requirements: to collect rich information about development in a flexible, easily re-structurable way, and to turn these stories into data so we can interpret and contextualize what we see. We’ve come up with a survey design tool which you can use to do a custom evaluation and compare your results to stories told by others, with the overall aim of helping everyone share knowledge and improve project design. The  approach will save you time but it will also enable you to get more back than you could ever put in.

So why do we use storytelling, you wonder? It turns out that managing this process with metrics, indicators, spreadsheets, and a numbers-only mindset is far more difficult and time-consuming. Narratives and a few survey questions are sufficient to see common patterns emerge from many perspectives.

Continue reading on ThinkNPC

Marc Maxson is an innovation consultant with Globalgiving, where he manages their global storytelling project. Previously, he worked as a PhD Neuroscientist and did Fulbright research on the impact of the internet on rural education in West Africa. He writes about evolution and international development at chewychunks.wordpress.com

When toys tell stories

I first learned about GoldieBlox from their superbowl ad, where they aggressively combat the toy industry’s stupid assumptions about what girls like (It’s not just about making it pink and putting a pony tail on it).

They are on a mission:

Only 13% of engineers are women and they believe that women innovators are our greatest untapped resource. 

They have a theory of change:

We inspire girls during a critical period, between age 6 and 13, and allow them to realize for themselves that building, creating, and owning their own ideas is what it means to be a girl.

Their latest ad campaign continues their message more thoughtfully:

(Note that begins as a parody of a 1980s anti-drug commercial, and so their ads are also targeting parents)

How is GoldieBlox “for” girls? (From their website)

Our founder, Debbie, spent a year researching gender differences to develop a construction toy that went deeper than just “making it pink” to appeal to girls. She read countless articles on the female brain, cognitive development and children’s play patterns. She interviewed parents, educators, neuroscientists and STEM experts. Most importantly, she played with hundreds of kids. Her big “aha”? Girls have strong verbal skills. They love stories and characters. They aren’t as interested in building for the sake of building; they want to know why. GoldieBlox stories replace the 1-2-3 instruction manual and provide narrative-based building, centered around a role model character who solves problems by building machines. Goldie’s stories relate to girls’ lives, have a sense of humor and make engineering fun.

That was an “aha!” statement for me. “Finally, something I can sink my teeth into!” I thought. So building blocks can be thought of as a storytelling tool, like the magic cards I made earlier. I know about character driven stories, and putting conflict into scenes to move it along and draw in the audience.

And in a way, GoldieBlox is using a conflict narrative to draw in their audience – girls. What a brilliant way to get girls on board, by reminding them from age 6 onwards that playing with these toys is an act of defiance against gender stereotypes.

And another company, play-i, offers a complementary approach to the same goal, for a younger audience:

https://www.play-i.com/

I just wished they had similar toys for the teenage crowd? What will these Goldie girls do when they outgrow their blocks? Perhaps this?

Python_For_Kids-2

http://python4kids.wordpress.com/

 

goldieblox-logo

fry_shut_up_and_take_my_money.jgp

A good proxy indicator for organizational learning culture

A recent Huffington Post article brought an interesting tool to my colleague Nick’s attention. Collusion helps you spy on the companies that are colluding to spy on you as you surf the internet. For example, every time you check the weather all of these sites are informed about you:

the-weather-channel-collusion

A list of websites that receive information from weather.com are shown on the left. About half are red and crossed out because collusion (this chrome plugin) blocked their access.

As you browse, collusion creates a network map showing how the different sites you visit talk to each other. You can hover over any node in the network to see a site’s connections and automatically block the transmission of data to known tracking sites like Google ad services, Doubleclick.net, etc. As you sift through your browsing’s connections, it quickly becomes clear that not all sites are created equal when it comes to tracking your metadata.

Our insight was that this tool could serve another purpose. You see, Nick and I are responsible for building up GlobalGiving’s database on organizational behavior and curiousity. This is used to measure each organization’s performance in a real-time, comprehensive way. If we could sort all organizations in the world into “good” and “bad” groups based on their habits, such as being responsive to the community they serve, demonstrating a tendency to learn from mistakes and remember what they’ve tried before (knowledge management), or their making effective use of free performance tools in their daily work (agility), we could help more money reach better NGOs, and ultimately improve more lives with the same amount of resources.

This is the same as saying “we’re going to make the whole aid world more efficient,” but when we say it, we mean it – because we have a way to do what we say. In the “big data” era, information will be used to make thousands of little evidence-based decisions that will improve the system overall.

But on to specifics. What do organizations’ websites reveal about their agility? A lot.

Look at these organization websites:

Each of these have hundred-million-dollar budgets. So how much effort to they make to optimize learning about visitors to their homepages?

agile-world-vision

agile-care-orgagile-save-the-children-org

agile-brac

agile-msf-org

agile-united-way

agile-world-bank

agile-helen-keller

agile-oxfamagile-heifer-international

I see a correlation between how much the organization focuses on public donations (versus government or private support) and whether they use free analysis software, such as google analytics. Of the ten organizations shown above (which are close to a top ten list of worldwide organizations by size) only Save the Children, Care, and World Vision made a serious effort to learn from their website traffic. Five our of ten at least have some kind of basic (free) analytics (google-analytics and/or google tag manager).

For the other half that do not, it is telling. These organizations don’t really need public support to survive, and are also (in my opinion) less accountable to community feedback because they are “too big to fail” in the aid world:

  • World Bank
  • BRAC
  • MSF
  • United Way
  • Heifer International

Types of 3rd party data collection sites

Analysis (curiosity)

  • google-analytics.com
  • GoogleTagManager
  • kissmetrics
  • vmmpxl – quantcast web traffic demographics
  • mxpnl — mixpanel is like google analytics, but you pay for it and it offers more features

Visualization or dissemination

  • mapbox
  • uservoice.com
  • chartbeat.com
  • openlayers

Marketing

  • anything in red (advertising)
  • youtube

Faster web loading and cloud data 

  • amazonws
  • visualwebsiteoptimizer
  • rackcdn — rackspace cloud storage

Social Media Plugins

  • twimg — twitter
  • facebook

Design iteration and testing (curiosity)

  • optimizely
  • omniture

For comparison, I took snapshots of GlobalGiving and various other online giving marketplaces or organizations we partner with:

agile-globalgivingagile-donorschooseagile-kiva

agile-betterplace-org agile-razooagile-great-nonprofits agile-give-directly

agile-development-gateway

Clearly, all of these organizations take their web traffic seriously. Each of GlobalGiving, DonorsChoose, Kiva, BetterPlace, and Razoo uses at least one analytics tool, one cloud hosting tool to speed up website load times, and many use an iterative design and testing tool like optimizely.

The surprise here is that GiveDirectly (the recent darling of the aid world and the media world) does nothing to learn about their traffic. It makes me question how much of a learning focus their organization has internally.

And that is what this is all about. I believe that organizations stamp an imprint of their internal learning on their external websites.

Curious, learning, experimenting organizations use web-based tools that help them achieve their goals (and leave a trace for us to track).

Large bureaucratic “stick-in-the-mud” organizations do not use any of these tools, leave no trace of their learning, and thus are probably not focused on learning.

Web footprints for a few randomly chosen GlobalGiving partner orgs

These organizations are much smaller than the ones listed above, but they still use more learning tools than even the world bank or BRAC uses, ergo they are probably learning more with fewer resources in my assessment:

agile-wildlife-alliance-org

agile-mountains-of-hope-uganda

agile-ayni-education

agile-afghan-institute-of-learning

agile-outreach-uganda

agile-ouelessobougou

agile-vision-africa

agile-african-rainforest-conservancy

Five out of seven local GlobalGiving partner organizations use google analytics. 

That’s a small sample, but a larger fraction of the group are still using more tools to learn about web traffic than the million dollar orgs.

These are just screen shots to show that there is useful data out there. Once you realize that the tools exist to ask old questions in a new (and more efficient) way, you simply need to write a little code to gather all the information. This will be my take home message at the Georgetown University master’s program class I’m teaching this week:

Graduate School should help you learn how to ask better questions and to recognize when the status quo of information is insufficient to fix the problem.

We live in a world that clings to the “myth of evidence”‘: We think our leaders make decisions based on weighing evidence, but they do not. They never have. Throughout history they have made instead made experience-based decisions, limited by their own wisdom and prior failures. This is about to change.

Decisions used to be made using tiny scraps of information, because that is all that was available. But this decade is the turning point when evidence becomes cheaper to aggregate and interpret than the cost of making decisions without it. Some giants will fall and others will rise to take their places, all because they understand the new calculus of “big data.”

And when the dust clears, a new kind of democracy will be possible* where in the past is was merely theoretical: policy decisions will reflect all peoples’ opinions where choices are a matter of preference, or based on sound science and observing human behavior on a macro scale (like Isaac Asimov’s psychohistory idea) where policy depends on truth rather than preference.

(though this kind of democracy will be made possible, it will almost certainly be tried somewhere outside of North America or Europe first. My guess: somewhere in the middle east where people want real democracy)

Follow

Get every new post delivered to your Inbox.

Join 914 other followers