If they give you lined paper, write the other way

Sometimes the rules have been true for so long, we forget that they are rules we thought up ourselves They are not defined by nature.

This post by Knowledge Management on a dollar a day expands a debate between qualitative (stories) and quantitative (numbers) research – particularly in International Development. I don’t think there’s any point to either side of this debate, because both sides are missing the point: the world has changed.

Today you can gather about 10,000 times more  information for the same amount of effort as you could in 1995. Computers, SMS, and electronic traces of human behavior are everywhere. We aren’t recognizing what this means:

There is a scale that stretches continuously from qualitative to quantitative. Where data falls on this scale is defined by the diversity of the sources from which it is derived.

Numbers can be just as wrong as stories – especially when they are all coming from a single source that wants the numbers to tell a single story. Likewise, stories can melt into a single number that magically measures impact — but not really — when the same problem occurs: many words, but few unique sources.

You need a lot of stories before anything consistent emerges.

Just to be clear about what I mean by “a lot”…

• GlobalGiving storytelling project collected 40,000 stories in one year from over 10,000 people.
• Facebook collects 900 MILLION rather meaningless bits of data EACH DAY.
• The Framingham longitudinal health study — responsible for over 50 peer reviewed papers about QUANTITATIVE health outcomes — covers a pathetically paltry 5,209 people over 40 years.

Qualitative data can EMERGE into quantitative data when two things happen – you start working with tens of thousands of degrees of freedom and QUANTITATIVELY detect and correct self-reporting bias. I say “degrees of freedom” and not stories because I don’t assume every story represents a different person’s perspective and knowledge – this is the sort of stuff you need to be constantly testing and challenging against other reference data sets. Also, even if the people involved are socially isolated from each other, a lot of stuff we talk about and “believe” is not true from our own lived experience, but is almost entirely what others and the news media tells us. Our method teases these different perspectives and examines them separately.

It’s also worth noting in these three examples above that cost is inversely correlated to sample output, yet fun is directly proportional to the volume of data generated. Why are we not talking more about making the information gathering process fun? I won’t be happy until at least a million stories about development are being gathered in Africa each year, and people are having fun doing it.

When we lack diversity of viewpoints and perspectives in any kind of data, the problem is not about quant-vs-qual. The problem is that we aren’t working with any data, period. As defined by Shannon’s information theory:

Where there is nothing unexpected revealed, there is no information.

Returning to today’s photo theme. “Qualitative” messages can always be checked for internal validity when we have a lot of them.

“If they give you lined paper, write the other way” means to start thinking about what the tools can do, beyond conventional rules that merely constrain our thinking about how to use those tools.

See also:

Shannon Information Theory as a model for noise-filtering in international development