Survey feebleness

Nobody believes questionnaires are a truly quantitative way to gather information about people and society; we just use them a whole lot as if they were.


Given a survey with two parts: (1) a space for people to tell you exactly what they mean in a story, and (2) a set of answers codified through multiple-choice questions, which one do you trust if the answers conflict?

Written your answer down yet? Good.

Here are some examples to spur your thinking on which one you should trust.

#1 World Food Program drought and famine relief effort in Kenya

(click title to read all stories)

The open ended question was, “Talk about a time when a person or organization tried to help someone or change something in your community.”

Of the people who chose WFP, and talked about a NEED, that was not met, because the wrong people / nobody benefitted and it was a bad idea, there were 5 “failure” stories:

(Please note: This plot was generated using SenseMaker(R) by Cognitive Edge. And there were over 100 generally positive stories about WFP, also shown in the plot but not circled.)

These are those five stories:

I’ve added emphasis to the parts relevant to whether they really are about success or failure.

Hunger is brought about by lack of food. Most of the parts of Northern Kenya have been worse hit by hunger in the recent past. Some parts of North eastern have been worse.There have been several reports of loss of lives due to higher and even cattle succumb to the situation. The United Nations through the World Food Organization have been on the  front line helping the affected people through food donations and other important commodities like water.Due to several cases of loss of lives and death of domestic animals, there is need to be met so that the issue can be solved fully. More organizations need to come in so that they may help tackle the hunger crisis.

Poverty in Kenya is a major issue and a normal thing to most of the citizens of Kenya living in rural areas. This is is because there are no major income generating activities that can make people rich.In Ugenya constituency most people a re living below the poverty line.There are people who only depend on others for their daily meals while others struggle to earn from hard work but  what they get is just too little for them to keep t hem going. Most of them depend on agriculture  yet the land i s not that productive. There is a program run by World food organization that provides to the people drought resistant crops but this not a major boost since there is no proper irrigation. Residents therefore always appeal for help from the government but what they get is too little.

World food program has helped many schools in kibera by providing food and shelter it has also helped in educating by providing books and also uniform. This organization was formed in the year 2009 may on the fifth day it has also improved high standard of people living in kibera. simply because people living in kibera do not have enough money to provide for their children for food. That is why world food program has brought free food to schools so that pupils could not be hungry when they are learning, Because food is important in our lives.

World Food Organisation which has been helping many children in the world, This days the children get food in the support of WFP. Those because people who are poor normally get their daily bread because they are being supported with this organisation. In every month W.F.P. Normally bring food to squaters who live in Rongai because they know that those squaters doesn’t have food and if they tell the Government they do nothing. That’s why World Food Programme Organisation intended to help them. That is why most of us thank the W.F.P for the support.

Joash was a man working with World Food Program Organization. He used to transfer food within Kibera schools pupils within kibera most of them their parents were as poor as a church mouse. So that pupils like  coming to school so that they may get food for the day. Others used to carry food taking to their parents. Joash managed to transfer food to their parts of Kenya. He used  to transfer food eastern part of Kenya.That is where he found  many pupils suffering from Marasmus.

I’d say that 4 of 5 of these speakly highly of WFP’s work, or of their employees. And in the one negative case, the storyteller explains exactly why WFP’s efforts haven’t succeeded. But if one was to just count these data points as evidence of failure without reading the stories, one might get the impression that failure was entirely WFP’s fault, which is was not.

In case you were wondering, here are other major relief / drought organizations mentioned:

Red Cross Kenya

Kenyan Government

#2 Education or school stories and Carolina for Kibera

Same analysis, different set of stories.

And the three most “negative” stories about Carolina for Kibera related to education:

I was a pupil in standard seven, i was doing well in my studies but not really, because most of the time i was being sent home to bring school fees. One day carolina came for kibera, came to our school they were a sponsor who was sponsoring children and thats why they came to our school.They were choosing the best pupil from class six upto seven in our school,and was among them who got the sponsorship.they were paying two thousand shillings for two terms but this year they increased three thousand for three terms.And now i am doing well because i am concentrating in my and i always thank the sponsorship for being kind to slum children and they helped many children apart from me.

Binti Pamoja is a sub organization that is under Carolina for Kibera. It mainly deals with girls as the name suggests; these are girls between the  age if ten to twenty five. It is a sub organization that has helped girls especially in kibera slum to realize their talents and also help their parents by offering scholarship to girls for secondary and collleges. The Binti pamoja sub-organization was established in order to help girls who did not know much about their status of life, this is because manyof the girls in klibera face many challenges in life like poverty. Girls now know what they want to achieve and remain focussed as they try and pursue their goals. This sub organization has really helped many girls in Kibera and they are able to cope with the situation around them and the girl child. rights are being proteced so no stigma and discrimination. This helping the young ladied have high self esteem and conqure the ups and downs of their lives with strength and self defense.

Two years ago I was with cousin called Isha Rose.we were going to Carolina to learn some activities like dancing,singing and narrating poems.As days went on, my cousin grew she sat for her national examination she passed and she taken to boarding where they paid for her the whole school fees with the organisation called (C F K)and thus she is now contuining with her studies at Ngara high school she is in formnfour (form 4) I give thanks to CAROLINA FOR KIBERA for helping my cousin in paying the fees.

World Vision

Negative World Vision story:

I was registered in World vision to be sponsored but i got nothing from world vision. Anyway they gave me some books and some scholastic materials for using but they done that when i was in primary two, when i reached primary three, i was discriminated from getting sponsorship and i started looking for any where or anybody else to process my education. Ihave reached senior six this year when world vision stopped even helping it’s people.


All three of the Carolina for Kibera stories were positive, though they are marked as negative in the combination of multiple choice questions. The one negative education story about World vision was clearly a complaint about the organization failing to continue it’s program.

(1) I interpret the inconsistency here as a lesson that checking boxes on survey questions is not sufficient to clearly codify some things, such as the author’s intent, tone, and what they would like to see change. I trust narratives to be more reliable sources of information even where multiple multiple choice questions indicate otherwise. If a person says through multiple choice questions, “This is about a NEED, and the WRONG PEOPLE benefited, and it was a BAD IDEA, I still want to read the story associated with these answers to be sure that they we are clear about what part of the story was a failure.

(2) You can’t codify everything, but you can explain everything. Narrative is fuzzy but powerful. Multiple choices questions are rigid and sometimes misleading.


One thought on “Survey feebleness

  1. Is this blog post really about how check boxes are inaccurate or about how no one understands triangles?

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s