Running a focus group on less than $5 for evidence-based decision making

By the end of this decade there will be a new kind of actor in board rooms and executive team meetings. The role of the “quant” is changing from the guy who runs reports and gives you a write up before the meeting to the guy/gal that sits in the meeting, fact checking assertions from leaders before they can finish a sentence.

Today I got to be that guy.

If you think this sounds like a “gotcha”1 think again. This is what evidence-based decision-making looks like in the digital age.

Every other week I attend a LabStorm at FeedbackLabs.org, a consortium of like-minded organizations working to establish the practice of listening to constituents as the only smart, feasible, right way of working for change. During our discussion, one of us speculated that a crowd of ordinary untrained people would be able to go to the typical organization’s website, find their articulated theory of change, read it, understand it, and then comment intelligently on whether that organization’s work was aligned with its own theory. If people could do this, he reasoned, we could weed out the well-run organizations from the poorly-run ones.

This is totally unrealistic.

Instead of trying to dissuade the group of pursuing this idea with logic and reason, which would derail the conversation and draw more attention to a suggestion that did not deserve it, I powered up a new human intelligence task (HIT) on Amazon’s Mechanical Turk. This platform lets you publish simple jobs for pennies a person. Most tasks take less than 5 minutes to do and require no special skill. It’s perfect for rapid opinion polling from “average people” similar to running a focus group, but without costing much time or money.

Today, rapid real-time polling is the cure for vapid speculation in board meetings.

So to recap, someone claimed that anyone could go to our own organization’s web site, find our theory of change, and define our organization’s impact based on this published data. So I paid $4.50 for 30 of these turkers to do exactly that. I asked them to describe what our purpose was, and what assumptions we were making about the organizations we work with in pursuit of our mission. This was timely, as we were planning a longer meeting to unpack our assumptions later this month.

Later that day, I had some quick answers.

amazon-turk-boss-focus-group-results

It took the average person over 12 minutes to complete the task. For comparison, I posted a separate batch of tasks for other people to do the identical thing with the website I manage – FeedbackCommons.org. It took those people half that time, or 6.6 minutes, to do the same task. If it takes someone 12 minutes (on average) to understand our purpose (and we’re paying them), and the average attention span of the people we really need to reach is 30 seconds – we have a problem.

Buy-in

Later, when a colleague of mine heard about this experiment, that person quipped, “I don’t think I could answer that question, and I work here!” Even if most of our team was already convinced of something, doing this experiment demonstrated the issue in a way that would convince everyone. This is “buy in” – the most difficult part of organizational change. Often people won’t buy in to new ideas until they agree on first-principles. I’m essentially challenging our internal assumptions with these questions:

  • How do we know whether we are on track in what we’re doing?
  • Does what we believe match reality?

The Data

The turk answers were very diverse. Here are a few of their descriptions of our Theory of Change:

  • If you give people power they will speak for themselves.
  • Is saying you gave a voice if you you used our feed back and surveys.
  • Adapt and help us improve.
  • To create social value what the customer satisfaction industry does for consumer-facing businesses.
  • If we listen, we will hear what people want.
  • If we manage the organization’s performance, then we can improve our ability to bring about social change.
  • Their theory is the “ask them.” If they ask people’s opinions about plans and based on that they analise if it will work or not, also is a way to improve the said plan.

These themes are not completely different from what our staff might say. Earlier this year we did this exercise ourselves and got a wide range of staff answers. While there’s some understanding here, I don’t think I’d like my work to be evaluated based on such different interpretations of our mission. This supports my original point, but luckily I didn’t stop there.

A few days later I raw three more rapid fire focus groups over Amazon Turk, each focusing on a different part of the products we promote. None cost more than $5 for 30 responses. Their collective insights gave me the evidence I needed to argue for a shift in our priorities, during our monthly agile sprint planning meeting. Rapid, diverse, cheap feedback gets us to focus on doing what matters most, first.

I asked similar questions about the new product tour on the Feedback Commons. People we already know really like the tour, but 30 strangers had a lot of additional suggestions to improve it. I also asked them at what point in the training they lost interest – which was super helpful.  Next round I plan to “focus group” different ways of explain these concepts, until I land on the clearest explanations.

In each test I also asked them to lay out what assumptions our organization was making about our customer. These were helpful in building a more complete theory of change. Instead of using focus groups to validate our assumptions, it might be smarter to use them to expose new overlooked assumptions in our theories of change – the real gotchas1 I worry about.

What I mean by a theory of change

scientific-theory-of-change-pattern

What a logical framework helps you do

logic-model-pattern

And how they relate

logic-model-vs-theory-of-change

Good management is doing things right, and good leadership doing the right things. Technology has improved management, but not leadership. Perhaps we’re on the cusp of seeing mechanisms that improve our leadership – how we use evidence to make smarter decisions. That’s why I joined Keystone Accountability, a not-for-profit consultancy that helps others make evidence-based decisions – through feedback loops.

Planet money has a great podcast about the people who do Mechanical Turk tasks.

See our product-tour.

(1) Note: A “gotcha” is slang used to express satisfaction at having captured or defeated someone or uncovered their faults.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s