Lean Startup Philosophy is about iterating on an idea in order to not waste time building something that nobody wants. Here I illustrate many iterations along the journey towards an analysis tool anyone can use with storytelling data. It starts in January 2012, 18 months ago. Eventually you reach simplicity beyond the intermediate stages of evolving complexity, where most projects flounder.
My design goals for the story analysis tool:
- Simple and intuitive user interface – We wanted it to be so easy to understand that people who had no training in statistics or analysis could use it.
- Flexible – able to provide an answer to thousands of questions anyone could pose to a large qualitative data set.
- Online interface built on free/open source code – because our users are spread around the globe and won’t want to download a desktop app.
- Low bandwidth friendly – many of our users get about 200 kbps download speed (about 1% the speed of American Internet!)
- Actionable – aims to provide answers that an organization can use to take action, rather than the “vanity metrics” that Eric Ries complains about in his book. A sample distinction would be knowing how many beneficiaries talked about you (less actionable information) vs discovering what kind of people talk about the problems your organization addresses (more actionable insights).
Complete design iteration map
Every project should make one of these in order to see what’s worked and what hasn’t. Details of each iteration follow.
Or see a previous iteration of this design map.
Detailed Timeline of design iterations
Each of these iterations took about a month (only some of my time was spent iterating on this).
Version 1.0 (a very basic search box that could only return stories that contained the exact phrase you entered.)
Luckily our GlobalGiving tech team was able to put every story online from day one with its own unique URL and meta data. All I had to do was link to the pages and show a summary of the story itself.
Search with more options
User feedback: We need more search options, especially the ability to drill down to stories with specific phrases, mentioning specific organizations, and from particular location.
Design changes: I hired a freelancer to build something more sophisticated, but the php/mysql design was not very easy to maintain because a drop-down list of orgs and locations pulled from hard coded information. Yet our data set was constantly out-growing the drop down list.
I also thought seeing a list of what others had searched for would help. It didn’t work within this design. I am planning to re-introduce this feature this month in a more usable form (the 100 top story “topics” shown below as search-result links instead of just the last 10 searches people performed on this page).
Visual who-what-where visualizer
I realized that what I believed people using search wanted as a quick vizual summary of all the ideas in a set of stories. So Fabio (my freelance developer) built a bubble chart that would make bubbles change size if they were more common. It elevated the word-bubbles when they were coming from a larger scope of stories (more scribes, more NGOs, more locations) and spread the words along the X-axis alphabetically to make it more readable.
This gave interesting results, but it wasn’t obvious what the visual pattern meant in many cases. It also could only drill down to a point. Then, if you wanted to go deeper into a set of stories you couldn’t do any more than just read them. And some common words would flood the view (like “help” and “child”).
Still, it was amazingly helpful in seeing a “self-report bias” pattern present in stories from one organization and not in another:
Organizations that provide “too much helpful guidance” to storytellers on what stories to write generate a set of stories with no “noise” at the bottom of this chart. So it is useful, but doesn’t actually give a good summary of what the stories contain.
At this point, I was beginning to think that these bubble-word maps could show much more information to our organization users if words were connected to each other in a map. So I used python and Gephi -a free network mapping tool – to build maps.
This one maps the adjacent words in sentences to build a meta-phrases map
I used this to provide feedback to some organizations. For example, all the stories that mention ANPPCAN are merged into one phrases map here:
Eventually, iteration on the mapping concept could yield some pretty powerful analyses. Here, two rape prevention programs in Nairobi are contrasted by comparing the words in both stories in red vs blue.
But even iterating on the design to map words in stories in more elegant ways did not capture the idea of what was in stories very intuitively.
Problems with maps:
- Not intuitive: It required us to educate our users, which did not meet our design goals.
- Not online: Gephi is desktop software. we built on online version but it more difficult to interpret. There is potential here to make a “community map” of organizations in a location and the topics they address (and visualize what the community wants that is not being addressed) – but this too diverged from our design goals.
- Not flexible if offline: We could never provide the user with a dynamic tool if it was not web-based.
- Lukewarm user response: I did a few tests with organization users. Demand was there, but not enough to make developing this a top priority.
The word cloud technique provides an effective way to contrast two sets of stories. So I built a prototype.
Example: These two word clouds are the product of a calculation. In the first, words found more often in stories about the Mrembo Project are shown. Words found more often in Sita Kimya stories are omitted from the word cloud:
Here is the flip-side calcuation: words found more often in Sita Kimya stories than in Mrembo stories:
After you stare at these two word clouds side by side, if may become clear to you that Sita Kimya stories lack any mention of HIV/AIDS. Mrembo stories appear to say less about rape.
Differential Wordles looked promising, but had some problems:
- Analysis was not easy or quick. It took time for users to detect what was different. It required a lot of stories and a computational correction for unequal-sample-size. It was ultimately a fuzzy way to compare things.
- Sometimes this gives users misleading picture. HIV/AIDS was almost totally absent from the Sita Kimya Project stories, but rape was definitely part of both projects. This visualization made it appear that rape was not emphasized enough in Mrembo, which was misleading.
- Not open-source. Using wordle makes it hard to “wrap” the visual output with any other features that add flexibility. Flexibility in analysis is one of our core design goals.
It was helpful in that diversity of language around any topic could be analyzed. The idea is not dead; it will just become one part of a larger set of tools.
A year later, I combined the differential wordles idea with the word bubbles plot from the who-what-where visualizer to make a tool for a Somali storytelling project:
SMS (mobile phone text message) story search
As our Rockefeller grant in the beginning of 2012 was focused on giving community members direct feedback, and mobile phones were the most ubiquitous technology available, I tried to copy the original search page as a SMS mobile service. (It was not an “app” because dumb phones cannot run software).
This took 2 months. I hired a full time Kenyan developer, who built a custom gateway program that would archive and manage conversations with phone people.
With the infrastructure in place, I wrote python programs that would respond to incoming text messages that contained the magic keyword “!search!” – it would search the stories and return the story text that best matched the incoming SMS, similar to google.
Texted In: “Green Belt movement came society, they showed us how to plant trees and how to take care of them. lack of warmth method failed succeed. !search!”
SMS Out: “Since the green belt movement came to our society, we now know the importance of trees. They showed us how to do planting of small seedling to keep our country green, But because of lack of water their method did not succeed.”
While this tool relied on appropriate technology, it did not meet a demand for our target group. Thinking that maybe NGO staff would use it more than storytellers, we then built a more complex question and answer prompting version that would allow an NGO to receive a personalized community feedback report. We gave out instructions on 3X5″ index cards at a community feedback meeting in Kibera (May 2012):
Only 5 of the 30 people made an effort to try the tool, and only 3 of them answered all the questions. Though the technology (phone texting) was a bit clumsy for filling out forms, I believe the low 18% response rate was a sign of low demand for this, or confusion about what “this” was. What is a “community report” anyway? And what does “community feedback” and “needs assessment” mean to a local NGO? Why and how, they wondered, could this information help them raise more money?
This failure was a useful moment in learning. And in the language of Eric Ries’s Lean Startup Philosophy, it later was “validated learning” because it helped us pivot and focus more on a version of our idea that our target user (an NGO staff person) would intuitively understand. (That product was still to come 5 iterations and 12 months in the future, but our low baseline “response rate” of 18% was a floor from which we could measure products.)
6 months later we also used our SMS conversation system to get feedback from storytellers and scribes (local story collectors). An interesting lesson from this SMS experiment was that 64% of scribes but only 2% of storytellers responded. That day we got important information about who our “customer” for SMS info was – it wasn’t the storytellers who’d participated before, even though we had more than 10,000 of them opt-in (give permission) to being contacted again. It might still be the ‘scribes’ because the response rate was very high. If (and this is a BIG IF) we could devise a product that would allow our scribes to serve as community informants (or town criers in our scribe analogy), then we clearly had a group of people who were likely to use such a tool.
But what would their motivation be to inform their community? We don’t know. But that currently is the question one of our Feedback Labs interns is asking this summer.
The step-by-step community report builder
I interpreted our poor NGO user response to SMS-based report building as a signal that we ought to build a more sophisticated and easier-to-use report building tool on the web. This wasn’t a simple idea, and my mistake cramming too many features into the concept of the tool instead of building a minimum viable product.
The working concept was for NGOs to build their own community report and send it in with their next funding proposal. It would help them interpret stories and build a community map that reveals three kinds of actionable information:
And the PDF report it would generate would start with a “org summary” based on GlobalGiving’s online NGO database. It would provide a summary of the balance of success and failure in related stories.
And I was tinkering with how you show a difference-in-difference data analysis model to non-statisticians. This is, after all, the power of internal and external benchmarks in data analysis – one of the strengths of our storytelling data.
And my developer built a complete “walk-through” for users to come and build the report:
And we even built an interactive story pattern exploration tool, inspired by XKCD’s value of everything chart…
These tools were getting pretty sophisticated. Here is an example of one iteration of the story “dashboard“:
But there were a lot of problems with the story walk-through design:
- We didn’t built a “minimum viable product.” There were too many features to test at once. We would find out that these features depended on each other and when one needed to be tweaked, so did all the others. This slowed down our building process.
- It was buggy because it was too complex. We were building something with a lot of moving parts. We eventually had errors that we could not trace to their core, and some of the bugs that we did trace required rewriting a lot of code to fix – and we hadn’t even tested the basic version yet.
- We stopped iterating. User feedback was very limited because too many of the steps were designed to be automated. A year later when I read “The Lean Startup,” I realize what we needed to try was a “concierge minimum viable product” whereby a person acts like the computer system and manually creates the data visualizations in order to test our basic assumptions (what is the product, exactly? and who is the customer?) In retrospect I tried the concierge MVP approach but I wasn’t listening to the message: what we were building wasn’t making life easier for our users. It was just as cumbersome as our previous attempts. Making it big and complex and fully featured just prolonged the time it took us to learn this truth.
- Modular components of the project stopped working as modules. This might sound technical, but the basic concept applies to many projects. We wanted the grid plotting tool to work elsewhere if we plopped it out and plugged it into a different project. Only, it wasn’t. We wanted the keyword bubblizer to plug into this walk through tool, but it wasn’t compatible either. Basically our big complex project was impossible to maintain or constrain, and no part of it could be salvaged or repurposed.
Eventually this iteration imploded. There were errors we never resolved, so I had to kill the project 9 months after we’d started. I’d overrun my budget and my allotted time, and had little to show for it beyond “unvalidated learning.”
Relevance matching algorithms
Another huge problem within my walk-through tool was writing an algorithm that would select the “relevant” set of stories to any organization:
The basic concept was sound – using our large collection of content tied to specific organizations as the input to search for stories with matching words. But this is more challenging than it sounds. I spent at least 3 months iterating on better algorithms for this (keyword matching, TF-IDF, genetic algorithms, hybrids), ultimately finding and implementing the lda-gibbs-neural-net method. This better matching was essential for building the story search drill down tool that I am currently testing with NGO users.
While this walk through / gridplot porject was in its final gasp of life, I went to a talk about corruption and built a simple hueristic auditing tool. This seemed totally unrelated to my storytelling project but the process gave me one valuable insight:
The most likely people to use a fraud detection tool are the victims of fraud themselves. These people tend to be less educated and unempowered, but they can understand basic dashboard icons.
So in a “duh” moment, I started remaking the original story search page from the first iteration but with a very simple web-form interface where people could check boxes on a list in order to parse stories in dozens of ways.
Once I had the “topic model” and search parser working well, the rest of the features were easy to build and maintain. It might not be obvious to the user, but the back end system to this drill down tool is doing a lot of good stuff in simple, efficient fashion. It runs some quasi-statistical tests while searching for stories and hides parts of the results if the statistics don’t merit showing it. Here are some of the hidden features in this drill down process:
We are starting to test it this month with real users.
Why it feels right to me:
- Interface is simple, yet it does a lot for the user in the background.
- First tool with extremely flexible options to parse stories along the axes that organization staff care about.
- Icon-based dashboard: Seems to be a breakthrough when you can display 4 dimensional data results with just a simple picture that has variable size and color. People who have tested it so far intuitively understand how to read the results.
But is it actionable? Only structured user-testing will be able to answer that question. Testing this is another Feedback Labs summer intern project. But it meets four out of the five design goals so far. And as a bonus, the icon-font dashboard will work with smart phone apps with minimal changes required.