Searching for the equivalent of Impact Factors, citations, and the h-index in development

As I sloshed through the snow last week, I thought about the way science “measures impact” compared to international development. They are quite different. The international development approach is exemplified by this world bank explanation:

When a project is completed … the World Bank and the borrower government document the results achieved; the problems encountered; the lessons learned; and the knowledge gained from carrying out the project.

It continues…

The International Development Association (the World Bank’s fund for the world’s poorest countries) tracks aggregated results. Its Results Measurement System (RMS) is designed to strengthen the focus of IDA’s activities on development outcomes and keep donors aware of IDA’s effectiveness. The system measures results on two levels:

1. Aggregate country outcomes including:
Growth and poverty reduction,
Governance and investment climate,
Infrastructure for development,
Human development.
2. IDA’s contribution to country outcomes, or “agency effectiveness.”

Wow. That’s very structured. But does it work?

In contrast, nobody asks a scientist to count the lives saved by her research. Instead, they ask “who found this scientist’s work valuable?” They count the number of other scientists who reference the first. Influential documents are the currency of impact in science.

The influence of particular efforts can be quantified, and most of the scientific community accepts the premise behind the journal Impact Factor, which is calculated by a 3rd party (Thompson-Reuters) and has a correction built into the formula so that any two journals from unrelated fields of science can be compared to each other.

And while I personally feel this focus on knowledge rather than results has led some scientists to avoid research that would benefit the lives of the most people (because these challenges are usually hard, often fruitless, well-traveled research areas), science has done far more to advance society than international development.

Why is that?

The grant award process is fairer and less politicized. You can assign one number to a person’s total contribution to the field. This is the h-index, or the the number of papers a scientist has that each have as many references from papers written by colleagues.This h-index is detailed by J.E. Hirsh in PNAS:

It boils down to some really basic results:

With this one number, you can tell whether a scientist is in the company of nobel laureates. Thus, top research grants go to people who have an h-index of at least 20 (they authored 20 papers that each have been cited as useful by 20 other people).

Nobody compared the actual types of work to each other in this case, yet everyone agrees that it is a reliable measurement for comparing that work. Amazing.  The impact factor and the h-index are based only on social networks of scientists citing each other. And everyone in science is required to contribute to the reputation system. That’s also important. Those who work in isolation are largely irrelevant to the progress of science.

For decades the world bank, the UN, and all the rest have been trying to compare the accomplishments of their projects (largely in isolation), while ignoring the social connections among their practitioners. But with social media, there has to be some international development equivalent to a “citation” by one’s peers. But what is that?

If we measured this, we could attribute value to the work of practitioners in the field, and thus reward the best with continued support.

5 thoughts on “Searching for the equivalent of Impact Factors, citations, and the h-index in development

  1. Need to blog on this stuff too:

    Any intelligent fool can make things bigger and more complex… It takes a touch of genius – and a lot of courage to move in the opposite direction.

    We’re hell bent on creating convoluted indexes and formulas to calculate and measure the fuzzy stuff like influence, affinity, or loyalty. As if somehow putting an algebraic formula to it will make it legitimate. Are there simpler ways we can be approaching these seemingly complex problems from a more human level? Is it ever enough to just say “this feels like the right thing to do”, even if we don’t have a spreadsheet upon which to demonstrate the results?

  2. Based on my read of the *Double helix* (a wonderful British tabloid account of how science is done) I’d expect Watson to have had an `h-index` of about 5 when he published the paper that won the nobel prize. Crick on the other hand was higher, but lower than 35.. but later went on to establish himself in the typical 40+ range for career `h-index`.

    So was Watson a hack? no. Just super lucky.

    So we need an `L-index` which is your `h-index` adjusted for quantified luck in your career.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s