How to conduct a storytelling evaluation (Part 1 of 2: analyze stories)

Introduction

If you work at an organization and have done an evaluation before, you probably followed these steps:

  1. Define the problem
  2. Design the survey
  3. Collect data
  4. Analyze data
  5. Form a conclusion
  6. Change project direction, apply for new funding, etc.

Our process has pretty much the same steps, only in the opposite order:

  1. Analyze existing data
  2. Form a hypothesis
  3. Design a different kind of survey
  4. Collect more data
  5. Compare your new data against the existing data
  6. Form a conclusion
  7. Define the problem
  8. Change project direction, apply for funding, etc.
  9. Go to step 3; repeat.

This “different kind of survey” uses qualitative stories in a quantitative way by combining text analyiss with questions about the story in the follow-up survey. These questions help define the ambiguity hiding in our preconceptions of the problem, mapping out power relationships in communities, conflict aggressors and victims, problems, solutions, innovation, and understanding what could have happened if things had worked perfectly. We provide a distributed data collection method that ensures many voices are heard. Every response begins with a story – a brief personal narrative – and the phrases in the story are an essential part of forming a hypothesis (step 2).

  • Aggregate power: With tens of thousands of stories in the reference collection, most organizations can find hundreds of stories related to their mission among the >58,000 we already collected in Kenya and Uganda from 2010-2012.
  • Filtering: Stories “at scale” makes it possible to validate the numbers and filter out misleading information.
  • Exploration: Broadness of the narrative prompting question and the scope of indivual perspectives we’ve already heard from allows the analysis tools to reveal unexpected connections between issues, peoples, locations, and organizations.
  • Benchmarking: Another major advantage is that our survey data can be combined with results from surveys conducted by others, as well as with your previous survey data. The analysis tools are built for pooling data and comparing subsets to each other, so that data trends become more reliable over time.
  • Durable, perenial data: Traditional evaluations usually create a new survey each time or lack the technology to merge new results with old ones done by others. Our story form builder is a compromise between design flexibility and instant benchmaring. And because our design makes it easy for you to change your questionnaire while adding to your growing data collection, you can form conclusions based on all of the data (past and present!).

Our process as a series of 9 steps. This tutorial covers the first two steps:
(1)    Analyze existing data — use the story search and story phrase bubbles tools
(2)    Form a hypothesis — based on what you see
(3)    Design a different kind of survey — story form builder
(4)    Collect more data — attract a group of young people, train them, given to papers, and send them out
(5)    Compare your new data against the existing data — in the same analysis tools as step (1)
(6)    Form a conclusion
(7)    Define the problem — informed by a new perspective
(8)    Change project direction, apply for funding, etc.
(9)    Go to step 3; repeat.

Explore the story search tool at djotjog.com/search/

wpid4265-media_1374868383219.png

Go to djotjog.com/search/ and either type in a word or phrase (in quotes, like “school fee”) to see stories that contain those words. Or you can click on one of the suggested search links below.

Build a good topical story collection with at least 200 stories.

wpid4266-media_1374869022412.png

I entered some words and phrases that relate to women entrepreneurs. But note that the word “entrepreneurs” was only found in ~50 stories (out of over 58,000). Instead, words like “micro loan” and “start business” are better choices. You can add several phrases and it will combine results for all stories that contain any one of those phrases.

Example: “micro loan” “start business” “started business” entrepreneur= 159 stories with any one of these phrases
Example: ugali bread “grow food” maize rice = 1485 stories with any one of these grain crop food words
Example: women and group and business = 290 stories with all of these words

Note that there is no way to combine words with AND and OR at this time, but you can usually find hundreds of stories with the existing search.

Understanding the visual summary icons

wpid4267-media_1374869238166.png

When hundreds of stories match, the search results include a visual summary of who told these stories and what they talked about. The icons of girls, boys, women and men will appear different sizes and colors depending on how prevalent that demographic group is in the group of matching stories, compared to the number of people of the same age/sex in all 58,000 stories.

A larger icon means that the demographic group is overrepresented.
A green icon means stories from this group are more positive in tone than the overall sample.
Red means more negative.
Yellow means about the same as the whole set of stories.

Note that you can mouse over these icons and see absolute numbers for the percent of stories that came from each demographic group, or relate to one of the ten story themes.

When is a trend in the icons important?

wpid4274-media_1375125428905.png

That is a difficult question to answer because it depends on the context. What are YOU trying to learn? If you have no questions about what your community thinks and feels about a subject, and no doubts about the work you are doing, this tool is not for you. But for me, speaking as a scientist, I am constantly asking myself if I am doing the right thing for the community. This reality check helps.

In this example, the overall sample of stories that mentioned “election” (n=724) was negative. But those about recent events in the last six months (from when the story was collected) from people who felt they played a role in the story were a mix of positive and negative. By selecting a more recent timeframe I was able to see that the trend is towards more examples of respect growing (green, positive icon), yet actual freedom remains negative.

If you were wondering how I was able to reduce the 724 stories to just 24 interesting ones, I was using power search, explained next.

Use Power Search to filter stories

wpid4268-media_1374871465342.png

Check boxes next to the things you want to see in your stories

wpid4269-media_1374871612372.png

By default, the checkbox for the words you searched for will be selected. Uncheck it if you want to lookat all the stories.
Check other boxes to narrow stories by outcome, demographic group, point of view, or related topics, and git [Generate a custom report now]

Note that within a set of answers to the same question, such as type of story, you can check more than one box and it will return MORE stories than checking just one box, because it will match ANY of the answers you select.

Power search lets you filter by any criteria in the questionnaire and export as CSV.

wpid4270-media_1374872003791.png

In the example shown, 1485 stories related to types of crops people grow in Kenya have been reduced to 55 stories using filters. The POV (point of view) filter excludes stories that sound like they are from the organization’s perspective (and not a personal one). The story needs to be focused on a problem (not a solution) and about economic opportunity (not social relations or physical well being).

A quick scan of the visual summary reveals that older men tell more possible “success” stories and women share more negative stories. The themes appear to focus on food and shelter, and to a lesser extent, on self esteem. Oddly, several categories are far less represented or absent: freedom, creativity, knowledge, respect, and family.

Export as CSV

wpid4271-media_1374874022827.png

If you want to analyze these stories in any other way, feel free to grab them.

What the export contains: Everything!

wpid4272-media_1374877136551.png

age_of_story_when_told, city, country, date_transcribed, felt, group_in_story, id, latitude, longitude, need_prob_soln, net_heirarchy_topic_score, num_stories_told, org, org_type, original_organization_name, other_information, outcome, pov, pov_avg, project_focused, quality_score, revised_organization_name, sex, soc_phys_econ, story, story_char_length, story_connection, storyteller_age, storyteller_contact_sms, title, topic_creativity, topic_family, topic_food, topic_freedom, topic_fun, topic_knowledge, topic_physical_needs, topic_respect, topic_security, topic_self_esteem, translated, url, who_benefited,

You can compare a story to other stories told by the same person

wpid4273-media_1374877402221.png

This is a “within-subjects” comparison – a quick way to find out if you are deceiving yourself about how much diversity is in the viewpoints you are listening to. If a person only talks about one issue, and nothing else, they are probably not giving the full story, and possibly not the most honest one.

Summary of hypothesis generating tools

wpid4275-media_1375127124517.png

You explore a large collection of anecdotes in order to get a feel for many perspectives on around an issue. Here are some strategies, with case studies to follow:
(1) Pick a general search phrase (e.g. “street child”) and then change the filter options around demographics and point of view. Save screen shots for each perspective and look at them side by side.

(2) Build a set of stories exactly related to your work, then broaden your search and look at how the trends change as you adjust your search words.

wpid4276-media_1375127625824.png

As you can see, choosing words that are more likely to be used by regular people will lead to stories that have less positive bias. Stories from farmers and that use the NGO phrase “food security” are more positive than those about “grow food” and bread, ugali, etc. — yet stories about these foods are true “food security” stories.

(3) Compare stories related to your work with any other topics that matter to these same people

wpid4277-media_1375128434560.png

In this example – over 5000 stories related to farming and agriculture and “food security” there is a reference group of 1106 stories NOT about these subjects but shared by people who gave these stories. The ability to do a “within-subjects” comparison on the fly is unique to this approach. We will be building more tools that make this easier to visualize. For now, you can begin by browsing these stories.

Bubbles: A more abstract comparison tool

wpid4278-media_1375128634704.png

This tool attempts to merge and parse all of the narratives in a single view. Words that are common and interesting appear in bubbles. The location of the bubble above or below a line is determined by the meta-data for each story. So far, the types of comparisons this tool allows are:
(1) is the story related to a known NGO or is it some unmet need, (or work by people not NGOs?)
(2) comparing success vs failure stories
(3) how does the number of stories a person shares relate to what is said?

Part 2: Build your own story form on GlobalGiving

Advertisements

One thought on “How to conduct a storytelling evaluation (Part 1 of 2: analyze stories)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s