About the RealBook (manual for story-based monitoring & evaluation)

storytelling-logo-transparent

(Summarized here)

Storytelling-RealBook-May-23-2012

Just as improv musicians rely on the RealBook, a loosely structured  guide to playing hundreds of standard songs, I think evaluators should have their own RealBook – so I wrote one. It incorporates many of the philosophies that guided the GlobalGiving Storytelling Project, but is not so rigid that others cannot adopt it to their own needs.

Here are some of the cool things you can read about in the RealBook:

  • Be Agile: Agile is a software development philosophy that emphasizes iterative and incremental development, where requirements and solutions evolve through collaboration between self-organizingcross-functional teams. Evaluations, and indeed all international development needs to be Agile to work.
  • Be community-centered: Evaluations are often done by outsiders, for outsiders – and few of the local leaders & implementers are influenced by the findings. Often they are never even told about the results. Evaluations should be done by local people for local people – with a minimum of training necessary.
  • Be practical: Technology is the future, and all evaluations should ultimately find themselves into electronic records. But don’t try some “breakthrough technology” that requires people living on less than $2 a day to own a $125 smart phone. We provide solutions that work on basic SMS, cost practically nothing, and provide much of the benefits of electronic data collection without laptops and smart phones.
  • Qualitative narratives can be quantitative data: This requires collecting a LOT of stories from a diverse group of people. Certain topics and patterns will emerge with sufficient sampling, and these will be the data. Some of these data will be “robust” – meaning the same pattern emerges in many places and consistently over time.
  • The right Incentives allow you to scale up collection: We found that every storyteller wants to be paid a little something for their effort. In the future, other incentives might be possible, such as reputation, recognition, or more power as a community liaison.
  • Analysis: make it digestible and actionable.
  • Benchmarking: Apply the 2-story rule so that you are constantly collecting a reliable external reference frame from which to judge the authenticity of what the analysis reveals.
  • Self-bias: All evaluations have a self-bias problem. Even 3rd party evaluators have to get a list of “beneficiaries” they can go interview. By changing the target audience from “beneficiaries” to “communities” we get much less bias, and we can tell which organizations are over-influencing the stories we collect.
  • Diversity sampling: Random sampling is just not a feasible strategy, and there’s no test to confirm the sample was random after you did it. In contrast, diversity sampling at least tells you whether you have sufficient diversity. (“sufficient” is some arbitrary standard to choose, but it is a reproducible target)
  • TjotjogA powerful coherence method. Story tjotjog is not two people saying the same thing; rather, story tjotjog is when many people tell slightly different stories that reinforce some common theme among all of them – a theme that is hard to predict beforehand and may not even be conscious in the minds of the individual storytellers.
  • Extending the method:
    • The self-organizing survey design
    • Computer-algorithm-aided analyses
    • Person-permanence – tracking the same people over time by giving them an incentive to identify themselves each time.
    • Ask everyone for 2 stories (within-subjects control group / benchmark)
    • Tell us a story to get a story SMS tool
    • Complete open storytelling with thematic emergence/convergence – using the SMS !tag! tool
    • Rigor comes from emergence, not survey redundancy (like the Meyers-Briggs test)
  • Add your own chapter: I’d like more people to be submitting their ideas into the RealBook for circulation. I have a side file (24 half-baked ideas to improve decision-making within and among organizations) that will slowly get incorporated in future versions. Don’t you have something valuable to teach others?

Since 2010, these organizations have expressed interest (to GlobalGiving) in learning more about story-based evaluation methods:

  1. DFID
  2. Keystone Accountability
  3. Development Gateway / AIDdata
  4. Search for Common Ground
  5. Models of Unity
  6. KCDF Kenya
  7. AIDinfo UK
  8. AMREF USA
  9. Case Foundation
  10. Daktari Bush School
  11. UN Global Pulse
  12. CHF International
  13. IDF Uganda
  14. USAID
  15. Americans for the Arts
  16. PLA Uganda
  17. RETRAK
  18. PEDN
  19. Vijana Amani Pamoja
  20. TYSA
  21. Western Water & Sanitation Forum
  22. Community Transformers
  23. Mathari Children’s Fund
  24. Hot Sun Foundation
  25. Sustainable Harvest
  26. Kituvo Mobile
  27. World Pulse Media
  28. A24 media
  29. Murdock University, Perth Australia
  30. Philanthropiece
  31. WMI
  32. IDEA – Irish Development Education Association
  33. Nominet Trust
  34. IREX
  35. IMAGINE: The Global Initiative for the Empowerment of Women
  36. Kabissa
  37. International Program for Development Evaluation Training (IPDET)
  38. Boma Project
  39. Kakenya’s Dream

These institutions have adopted and are testing a variation of this method:

  1. University of Limpopo, South Africa – Chris Burman
  2. USAID
  3. IFC
  4. PACT Kenya

These institutions help us do it:

  1. Rockefeller Foundation – funds our research into methods and tools for community feedback
  2. Cognitive-Edge – Licenses the SenseMaker suite (R)
  3. Development Seed – makers of the free mapbox tool
  4. Envaya – free SMS gateway technology

These peoples’ ideas have contributed to this project:

  1. Irene Guijt – LearningByDesign.com
  2. Dave Snowden – Cognitive Edge
  3. Nancy McPherson – Rockefeller Foundation
  4. Jonathan Harris – Cowbird.com
  5. Eric Berlow – Simplicity from Complexity (see TED Talk)
  6. Cynthia Kurtz  – Narrative Evaluations Expert
  7. Dennis Whittle – GlobalGiving
  8. Bill Easterly – White Man’s Burden:  Why the West’s Efforts to Aid the Rest Have Done So Much Ill and So Little Good
  9. Dan Ariely – Predictably Irrational
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s