Survey feebleness

Nobody believes questionnaires are a truly quantitative way to gather information about people and society; we just use them a whole lot as if they were.

POP QUIZ!

Given a survey with two parts: (1) a space for people to tell you exactly what they mean in a story, and (2) a set of answers codified through multiple-choice questions, which one do you trust if the answers conflict?

Written your answer down yet? Good.

Here are some examples to spur your thinking on which one you should trust.

#1 World Food Program drought and famine relief effort in Kenya

(click title to read all stories)

The open ended question was, “Talk about a time when a person or organization tried to help someone or change something in your community.”

Of the people who chose WFP, and talked about a NEED, that was not met, because the wrong people / nobody benefitted and it was a bad idea, there were 5 “failure” stories:

(Please note: This plot was generated using SenseMaker(R) by Cognitive Edge. And there were over 100 generally positive stories about WFP, also shown in the plot but not circled.)

These are those five stories:

I’ve added emphasis to the parts relevant to whether they really are about success or failure.

Hunger is brought about by lack of food. Most of the parts of Northern Kenya have been worse hit by hunger in the recent past. Some parts of North eastern have been worse.There have been several reports of loss of lives due to higher and even cattle succumb to the situation. The United Nations through the World Food Organization have been on the  front line helping the affected people through food donations and other important commodities like water.Due to several cases of loss of lives and death of domestic animals, there is need to be met so that the issue can be solved fully. More organizations need to come in so that they may help tackle the hunger crisis.

Poverty in Kenya is a major issue and a normal thing to most of the citizens of Kenya living in rural areas. This is is because there are no major income generating activities that can make people rich.In Ugenya constituency most people a re living below the poverty line.There are people who only depend on others for their daily meals while others struggle to earn from hard work but  what they get is just too little for them to keep t hem going. Most of them depend on agriculture  yet the land i s not that productive. There is a program run by World food organization that provides to the people drought resistant crops but this not a major boost since there is no proper irrigation. Residents therefore always appeal for help from the government but what they get is too little.

World food program has helped many schools in kibera by providing food and shelter it has also helped in educating by providing books and also uniform. This organization was formed in the year 2009 may on the fifth day it has also improved high standard of people living in kibera. simply because people living in kibera do not have enough money to provide for their children for food. That is why world food program has brought free food to schools so that pupils could not be hungry when they are learning, Because food is important in our lives.

World Food Organisation which has been helping many children in the world, This days the children get food in the support of WFP. Those because people who are poor normally get their daily bread because they are being supported with this organisation. In every month W.F.P. Normally bring food to squaters who live in Rongai because they know that those squaters doesn’t have food and if they tell the Government they do nothing. That’s why World Food Programme Organisation intended to help them. That is why most of us thank the W.F.P for the support.

Joash was a man working with World Food Program Organization. He used to transfer food within Kibera schools pupils within kibera most of them their parents were as poor as a church mouse. So that pupils like  coming to school so that they may get food for the day. Others used to carry food taking to their parents. Joash managed to transfer food to their parts of Kenya. He used  to transfer food eastern part of Kenya.That is where he found  many pupils suffering from Marasmus.

I’d say that 4 of 5 of these speakly highly of WFP’s work, or of their employees. And in the one negative case, the storyteller explains exactly why WFP’s efforts haven’t succeeded. But if one was to just count these data points as evidence of failure without reading the stories, one might get the impression that failure was entirely WFP’s fault, which is was not.

In case you were wondering, here are other major relief / drought organizations mentioned:

Red Cross Kenya

Kenyan Government

#2 Education or school stories and Carolina for Kibera

Same analysis, different set of stories.

And the three most “negative” stories about Carolina for Kibera related to education:

I was a pupil in standard seven, i was doing well in my studies but not really, because most of the time i was being sent home to bring school fees. One day carolina came for kibera, came to our school they were a sponsor who was sponsoring children and thats why they came to our school.They were choosing the best pupil from class six upto seven in our school,and was among them who got the sponsorship.they were paying two thousand shillings for two terms but this year they increased three thousand for three terms.And now i am doing well because i am concentrating in my and i always thank the sponsorship for being kind to slum children and they helped many children apart from me.

Binti Pamoja is a sub organization that is under Carolina for Kibera. It mainly deals with girls as the name suggests; these are girls between the  age if ten to twenty five. It is a sub organization that has helped girls especially in kibera slum to realize their talents and also help their parents by offering scholarship to girls for secondary and collleges. The Binti pamoja sub-organization was established in order to help girls who did not know much about their status of life, this is because manyof the girls in klibera face many challenges in life like poverty. Girls now know what they want to achieve and remain focussed as they try and pursue their goals. This sub organization has really helped many girls in Kibera and they are able to cope with the situation around them and the girl child. rights are being proteced so no stigma and discrimination. This helping the young ladied have high self esteem and conqure the ups and downs of their lives with strength and self defense.

Two years ago I was with cousin called Isha Rose.we were going to Carolina to learn some activities like dancing,singing and narrating poems.As days went on, my cousin grew she sat for her national examination she passed and she taken to boarding where they paid for her the whole school fees with the organisation called (C F K)and thus she is now contuining with her studies at Ngara high school she is in formnfour (form 4) I give thanks to CAROLINA FOR KIBERA for helping my cousin in paying the fees.

World Vision

Negative World Vision story:

I was registered in World vision to be sponsored but i got nothing from world vision. Anyway they gave me some books and some scholastic materials for using but they done that when i was in primary two, when i reached primary three, i was discriminated from getting sponsorship and i started looking for any where or anybody else to process my education. Ihave reached senior six this year when world vision stopped even helping it’s people.

Conclusions

All three of the Carolina for Kibera stories were positive, though they are marked as negative in the combination of multiple choice questions. The one negative education story about World vision was clearly a complaint about the organization failing to continue it’s program.

(1) I interpret the inconsistency here as a lesson that checking boxes on survey questions is not sufficient to clearly codify some things, such as the author’s intent, tone, and what they would like to see change. I trust narratives to be more reliable sources of information even where multiple multiple choice questions indicate otherwise. If a person says through multiple choice questions, “This is about a NEED, and the WRONG PEOPLE benefited, and it was a BAD IDEA, I still want to read the story associated with these answers to be sure that they we are clear about what part of the story was a failure.

(2) You can’t codify everything, but you can explain everything. Narrative is fuzzy but powerful. Multiple choices questions are rigid and sometimes misleading.

 

Storytelling analysis tools and maps

Introducing some new interactive maps of the GlobalGiving Storytelling project in East Africa (all 2011 data is shown).

Who is talked about?

Purple dots: GlobalGiving projects & reports. Other dots: stories about organizations color-coded by type.

Success Stories

HIV stories, HIV treatment, and GlobalGiving projects overlay

Who-Where visualizer map

Gephi Map of everything talked about

From over 30,000 stories gathered in 2011. Words must have been used in a phrase at least 150 times to appear. So every phrase is significant.

 

Iowa Caucuses 2012: Rural Urban GOP divide

http://en.wikipedia.org/wiki/Iowa_Republican_caucuses,_2012

I couldn’t help noticing that the tie between Rick Santorum and Mitt Romney falls cleanly along the low/high population density divide. Nationally, Democrats dominate areas of the country with high population density (cities), and Republicans win rural areas.

You might think of Ron Paul as appealing to people in the suburbs, but that’s less clear.

I wonder how this affects the presidential contest.

What life is like in the Democratic Republic of Congo

The Democratic Republic of the Congo (DRC) has been in the news as two leaders dispute the election results and swear themselves in to run the country. Here is a source of information about what DRC communities think.

Mobile Accord was nice enough to share results of 12,750 text messages they collected in the Democratic Republic of Congo (DRC) in 2010 using their new GeoPoll application. Here are some visual snapshots of what they found:

What were the root causes of the conflict in DRC? (1073 responses)

(Responses gratuites: Quelles ont ete selon vous les causes profondes du conflit en RDC?)

Alternate visualization, using Gephi to map connected phrases from these texts:

Click on the overall map to browse. Here’s a zoom in on the heart of the answers: POWER

Variations on power was common in many answers: Tribalism and power, politicians fighting for power, a lack of power for the people, lust for power, competition for power, poor sharing of power, pursuit of power.

What are important changes your region requires? (1462 responses)

(Reponses gratuites: Quels seraient les changements les plus importants qui pourraient rendre votre region plus sure?)

Mapping the essential phrases:

The 1462 responses could spell out national priorities for the Government of DRC and the international community:  Infrastructure, job creation, employment, security, health, education, and a mentality of changing behavior. Wordle emphasizes building roads and providing electricity.

What role should the international community play?

(Qu’est-ce que la communaute internationale pourrait faire de plus pour soutenir la paix et la securite en RDC? Est-ce que sa presence aide ou fait du mal, et comment?)

Nothing clear emerges to this question, other than maintaining a presence to support peace.

You can click to explore further.

What should the top priorities be?

(Quelles sont les priorites pour ameliorer la vie dans votre region – par exemple, la securite, les infrastructures, l’education, la sante, l’emploi?)

This response mirrors the important changes question: Employment, Health, Education, Infrastructure, and Security.

What should happen to the perpetrators of violence? (936 responses)

(Responses gratuites: Que pensez-vous qu’il faudrait faire aux auteurs de violences en RDC?)

The answer seems painfully clear: The perpetrators of violence must be punished under international law to bring justice…

Why do young people turn to crime?

(Reponses gratuites: Pourquoi certains jeunes se tournent-ils vers une vie de crime ou de violence?)

And the word map of these texts shows a similar focus: Lack of education, lack of work lead to violence and poverty.

If it was safe to live and work in your area, how would life be different today?

(Reponses gratuites: S’il etait plus sur de vivre et de travailler dans votre region, comment votre vie serait differente aujourd’hui?)

This question did not really work. Most responses included the question rephrased as an answer.  One answer said, “Compared to that of today? The answer is: Life is.“

That may be more revealing than the rest. Do people imagine what life would have been like if circumstances were different in Eastern Congo? Perhaps not.

Prosperity and long-term trends

You’ve probably seen this Obama “Road to Recovery” jobs plot for 2008 to 2011:

And there’s a conservative version of the story on Carl Rove’s website as well:

Both are giving you a short term perspective of a long term problem. For the record, Carl Rove’s version is more misleading because of several plotting tricks:

• Arbitrary timeframe (if February 2009 is the starting point, then why is Obama judged to be 651,020 jobs in the hole on inauguration day? What’s the “zero time point” here?
• Arbitrary endpoint (Rove was posting this report monthly and suddenly stopped in December 2010. Why is that? Coincidentally, the next month starts the flip from negative to positive on Obama’s chart.

Side by side: Carl Rove chose the perfect time to start and stop reporting

The Missing Perspective

Looking back ten years helps me to see that job growth in 2011 wasn’t that different from most of the 2000s. But it doesn’t explain why unemployment remains high. I needed more context.

If the 2008-2010 recession was caused by banks, take a look at the banking industry. This chart took me a while to compile from several small charts because nobody looks at long term trends in banks.

From 2004 to 2010, the banking industry made a healthy profit. Even when the bottom was falling out of the housing market in 2008, requiring a record infusion of cash from the bank’s holding companies, shown here,

The banking industry did not collapse like it did in the past.

Going into the distant past, bank failures were much more common:

Why aren’t banks collapsing like they should?

Banking is not real capitalism. For contrast, airlines are a typical industry that follows a business (and regulation) cycle. Shown below are net profits for the airline industry from 1989 to 2009:

US Airline profits, 1989 to 2009

(I call the version that banks enjoy “crapitalism” – because the net effect of “stability” is less prosperity, more corruption, and ineffective governance.)

The long-term view of banks is totally different from the airline industry. Below you can see that profits and growth came to banks every year from 1904 to 2008 (1997-2004 is missing data).  I had to squeeze and scale lots of bank reports to show this, so sorry for the tiny text.

Going as far back as 1904, banks as an industry have made money every year even through depressions and booms. It would take the entire economy to collapse* in order to banks to feel it. (* so in the 4th quarter of 2008 looks like the economy collapsed, only the government flushed trillions of dollars into the chasm to seal it up.)

In the past (1870-1930), greed still managed to make a lot of banks fail, even though loans are a pretty safe way to make profits.

There’s another reason why banks do well during hard times. Governments borrow money and give it to banks, and this accounts for more and more of their assets.

Here you can see that over the last decade Japanese banks have loaned to the US government (by buying government bonds), and this is now most of their wealth:

And also over the long term, “banks” have become investment machines and less of what they do involves loaning money:

This is what bankers mean when they talk about innovation and “financial instruments”: Little of what they do with money is about loans, which is also why few banks needed to collapse simply because the economy did in 2008. This resilience is also evidence that the government is doing everything possible to prevent banks from experiencing what every other industry knows as bona fide “capitalism.”

(Note: The only other industries that have consistent profits and growth every year are US health care and the oil industry.)

Loan losses are huge, but banks can now make record profits simply by moving money around. In 2010, the largest US banks earned a profit every day of every quarter by taking US bailout money and returning it to the same government in the form of government bonds (US debt).

All of this sums up into a simple picture that I posted last week.

Real wealth versus fake wealth created by the global banking and finance system:

Development projects must be fun to work

A recent economist article talks about getting better results from development projects:

There is tremendous untapped capacity in unlikely places – Sudanese villages, public sector institutions in Kenya, and countless other developing world communities.

In Sierra Leone:

“Only 1,000 people used HIV/AIDS Voluntary Counseling and Testing (VCT) services over two years. Then, over 100 days, a small local team increased use of VCT services by 17,500.”

In fact, VAP in Kenya did that a few months ago. They had a “get tested” soccer tournament.  While they only got 1-2 thousand tested, it was a cheap, fast, and effective.

I like their idea:

“We can set a policy that out of each dollar spent on projects and programs, 10 cents are spent on unleashing this local performance capacity reserve. This would apply equally to “hard” infrastructure projects and to “soft” training or capacity-building projects. We contend that this step alone would improve these programs’ and projects’ impact by at least a factor of 10, or even 100.”

However I don’t think whether the money went to “hard”, “soft” or “capacity building” was what caused the 100-fold increase in people tested. Instead, I think these were the important factors:

• Local org.
• Local org decided VCT testing was it’s own priority.
• Local org used local people in smart, cost-effective ways (including local volunteers)
• Local org did not wait for outside help, outside directions, or outside financing
• Local org provided a FUN incentive to get tested.

Fun and Local

The authors overlooked the power of fun in their commentary. Without fun, people are generally unwilling to waste time doing things, even if those things are in their own best interest – like getting tested.

I’ve also written previously that funding for fun, local projects needs put more of the resources in the hands of local people, not governments, agencies, or even local authorities. Currently, the bigger the project budget, the smaller the percentage of money reaching local people.

A Better Project Budget Breakdown:

That 10 percent should go towards local organizations with fun ideas, and used to build systems that help mobilize local people. Organizations with a clear goal in mind who have had any previous social impact in their community – such as getting 2,000 people tested – should be allowed to tap this money.

I think GlobalGiving does this. Our due diligence process identifies competent organizations with a demonstrated social impact (even if small). We teach them how to mobilize people, donors, and articulate a clear vision for themselves. We’re building a SMS feedback loop system to allow these organizations to better keep in touch with their communities. And we try to to make it fun.

I hope that FUN gets written in as one of our core values. NaNoWriMo already has FUN as it’s #1 core value – and look at what a difference they make in the world of writing!

I think these other suggestions from the article were good:

• More coaching, less managing by outsiders.
• A space where local orgs can build a sense of ownership, commitment and confidence, and identity.

Enouce and Nancy of VAP running their VCT tournament