Are you measuring the right thing?

This is real data from an organization. Each dot represents a different thing they measure. The connecting lines between them are strong statistical correlations between the things they measure.

are we measuring the right thing

The details of what these dots represent don’t matter, because all organizations do this: They set out to measure their “impact” and select the pink dot as an indicator. Then along the way, with funder pressure and for other compliance reasons, they spend most of their time measuring the cluster of stuff in the center of this network. However, in order to really understand your “impact” you need to measure several things that “co-relate” to your indicator (things that change when your indicator scores go up or down).

Luckily, offers a new tool that can produce this associations map from any excel spreadsheet in about 4 clicks. (The Association Discovery Tool) The map shows that only 3 of the many things this organization tracks are actually connected to the indicator that ultimately matters to their success. Eric Berlow explains why this matters in a 3 minute TED talk about the American Counter-insurgency strategy in Afghanistan:

The BigML tool provides any organization with an easy way to prove it is improving, by doing this:

  1. csv-xlsExport your organization’s data to CSV, and generate this map in (free for small data sets)
  2. associationsSave that map.
  3. keystone_logoGo out and try one of Keystone’s radically simple surveys, such as our Delighted (net promoter) tool, or a one-question survey. If you don’t understand this, look at this blog series, Be More Effective, for real world case studies from GlobalGiving.
  4. Merge your feedback from people (0-to-10 scores from people you serve) with other measurements you have from these same people. (Oh, is this HARD? Well luckily I built a data merge tool that makes this easy. It a core part of the feedback commons — being able to merge your own data with your other data. Contact me via the commons for help.)
  5. Go and re-run the BigML analysis, now including the feedback data from a radically simple one-question survey. If I am right, you should see that feedback from people you serve is more closely tied to whatever your fancy schmancy indicator of “impact” is.
  6. Now, use that feedback data to refine your program, and ask for it regularly.

This is the simplest, cleanest, most direct process by which all organizations can get empirically better. A lot of people complicate this whole data thing, but it’s as simple as that.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s