Quantifying Impact: Why NFL’s quarterback rating (QBR) is smarter than any development index

The NFL’s QBR succeeds where most NGO performance indicators fall short: It captures the context and complexity of real world situations, and gives us a blueprint for how we fix Impact Measurement.

[Background]

International aid organizations like the USAID, DFID, and the World Bank use performance indicators to measure the impact of their programs. Quality of life is one example:

Quality of Life Index is a composite of 9 other factors, which are composites of 47 other measurements

OECD’s interactive version is much better, but still based on the same approach of counting and averaging a bunch of stuff:

Better, because YOU define what's important and it recalculates.

Counting things and more things and putting them all together can give false impressions, because sometimes the numbers don’t reflect the context. A number might mean something else if you adjusted for the situations in which these numbers appear. And yet we make big decisions, allocating millions of dollars every day on the basis of these imperfect indicators.

Total Quarterback Rating (QBR)

This is why the QBR is way better. Whereas the old QB rating reflected the skill of the quarterback (answering: “how good is this player?“), the new QBR reflects the player’s contribution to the team’s overall success (answering: “how essential was this player to winning the game?“). It’s the difference between quantifying outputs and quantifying outcomes.

In order to reflect outcomes, and not outputs, the QBR breaks down each of the player’s actions and gives them a positive or negative score, adjusted by a “clutch play” factor. The “clutch play” factor is the brilliant part, because situational context now determines the score. If your team is tied and the quarterback throws a pass for a touchdown to give the team a lead, that action has more value than the identical action taken when you team is already ahead by 14 points. If the pass is 40 yards and the receiver runs for 10, then the quarterback is awarded more points than if he dumps the ball off for 5 yards and the receiver advances it 45 yards himself.

Situational variables affect the rating and give a realistic view of the quarterback’s contribution to the outcome (winning or losing). Given all the data (captured on 4 cameras), it would be crazy if we could not mathematically define the quarterback’s Impact in a game.

“Clutch” moves by smart NGOs

If we desire it, international development indicators could just as easily incorporate “clutch factors” too! A women’s giving circle that risks all to start an agro-business during a drought and succeeds should be given a much higher performance and outcomes rating than a large business that invests 5% of it’s profits in a risky venture. A local NGO with minimal budget run by volunteers that manages to “solve” a tricky problem like getting loitering kids back into school should be awarded much higher performance points for a clutch move, given their lack of capacity to intervene. Governments that test multiple interventions to improve education and disseminate findings about the one that succeeds should have a better credit rating than a government that fails to invest and experiment – because honestly, much more of the taxpayers’ money is likely to be wasted complacent leadership. Recognition for “Clutch play” and “Clutch moves” by NGOs is lacking in international development, as is the concept that situations affect the appearance of success.

People still resist conclusions about outcomes

I’m shocked at all the fan negativity about Total QBR, but perhaps that’s precisely because it challenges our comfortable notion that a “good try” is as good as actually winning the game. To illustrate, here are some quarterbacks compared in the new and old rating systems: (2011 rating source) (2011 QBR source) (full season 2011 QBR statistics)

QBR statistics From Week 3:

QB	New QBR rating	QB	Old rating	Change (Old -> New)
Tom Brady	85.3	Aaron Rodgers QB, GB	120.9	+1
Ryan Fitzpatrick	83.7	Tom Brady QB, NE	113.8	+4
Aaron Rodgers	79.1	Matthew Stafford QB, DET	110.7	-3
Matt Hasselbeck	77.8	Drew Brees QB, NO	109.7	+3
Drew Brees	75.8	Eli Manning QB, NYG	104.3	-1
Matt Schaub	74.7	Ryan Fitzpatrick QB, BUF	103.5	+2
Tony Romo	73.9	Matt Hasselbeck QB, TEN	102.2	+2
Matthew Stafford	70.9	Matt Schaub QB, HOU	101.9	-5
Jason Campbell	67.5	Tony Romo QB, DAL	95.8	+1
Eli Manning	64.9	Jason Campbell QB, OAK	93.8	-5
Joe Flacco	64.2	Kevin Kolb QB, ARI	93.8	+1
Chad Henne	62	Joe Flacco QB, BAL	91.9	+7
Josh Freeman	55.3	Alex Smith QB, SF	91.3	+14
Ben Roethlisberger	52.9	Mark Sanchez QB, NYJ	90.9	+3
Philip Rivers	51.3	Michael Vick QB, PHI	87.7	+6
Alex Smith	49	Rex Grossman QB, WAS	86.4	-4
Cam Newton	47.8	Ben Roethlisberger QB, PIT	85.5	+1
Rex Grossman	47.5	Cam Newton QB, CAR	85.1	-2
Kyle Orton	47.2	Chad Henne QB, MIA	82.4	+5
Kevin Kolb	44.9	Jay Cutler QB, CHI	82.4	-9
Colt McCoy	43.5	Philip Rivers QB, SD	82.1	+4
Donovan McNabb	42.8	Andy Dalton QB, CIN	82.1	+4
Michael Vick	42.6	Matt Ryan QB, ATL	81.5	-8
Matt Ryan	40.5	Kyle Orton QB, DEN	79.1	-1
Sam Bradford	33.7	Colt McCoy QB, CLE	78.4	+4
Tarvaris Jackson	32.8	Donovan McNabb QB, MIN	78.1	+2
Mark Sanchez	32.3	Josh Freeman QB, TB	76.1	-13
Jay Cutler	29.4	Tarvaris Jackson QB, SEA	73.7	-8
Andy Dalton	26.5	Sam Bradford QB, STL	73.3	-7
Matt Cassel	22.3	Kerry Collins QB, IND	65.9	+1
Kerry Collins	13.6	Matt Cassel QB, KC	65.5	-1

[Note: the Change column refers to the first quarterback named in a row (i.e. Ryan Fitzpatrick = +4)]

If you run down the list, you’ll notice that highly rated QBR quarterbacks are on teams with a better record. Four weeks ago nobody would have consider Ryan Fitzpatrick (a 7th round draft pick and the only Harvard Graduate to start as a quarterback in the NFL in decades!) to be a superstar, but each week Fitzpatrick has carried his team from 14, 17, and 20 points down back to victory in the 4th quarter. He is the poster child for a clutch player.

Likewise, Josh Freeman of the Tampa Bay Buccaneers doesn’t put up numbers but consistently wins close games (leading the NFL in 4th quarter come-backs last year). [Update: By the end of the season, both of these trends had reversed, and their QBRs were down to 18th and 23rd place for Fitzpatrick and Freeman respectively.]

However, the QBR isn’t useful for Fantasy Football, where you can start awesome players who rack up monster stats on teams that often lose games. The quarterback on my current fantasy team is a perfect case in point: Mark Sanchez is the most overrated QB in the NFL (13 spots higher based on his stats, compared with his contribution to winning games for the Jets), and yet he makes perfect sense to start in a Fantasy League because the rules never penalize players for losing games. Fantasy Football is all about individual outputs, not outcomes. So I can win in the fake world by focusing on intermediate factors that don’t matter to winning the game.

There is an obvious real-world analogy here with certain aid agencies that focus on outputs instead of striving to measure outcomes: They are merely playing “fantasy football” in international development, and not necessary winning games. What’s the solution? Devise indicators that adopt complexity-based thinking and adjust parameters situationally, until the indicators are really incorporating as many factors as are needed to understand the real world.

I admit this is hard, but not impossible. It simply requires another system to know which models are improvements over the older ideas. Head-to-head competition between models has been used in both molecular Neuron modeling and nuclear retaliation protocols, and certain models actually DO perform better than all the rest. (Sadly, the best model for neuron interactions only predicts a quarter of the real neuron behavior, and the best nuclear retaliation game theory still destroys the world) – so not every problem can be fully modeled (or fully “indexed”).

2011 Complete Season QBR ratings

Rank	Name	QBR
1	Aaron Rodgers, GB	85.2
2	Drew Brees, NO	84
3	Tom Brady, NE	74.2
4	Tony Romo, DAL	70.1
5	Matt Ryan, ATL	67.5
6	Matt Schaub, HOU	66.7
7	Matthew Stafford, DET	65.1
8	Philip Rivers, SD	64.3
9	Ben Roethlisberger, PIT	63.3
10	Michael Vick, PHI	63.2
11	Carson Palmer, OAK	62.3
12	Matt Hasselbeck, TEN	61.6
13	Eli Manning, NYG	61
14	Jay Cutler, CHI	59.5
15	Joe Flacco, BAL	57.9
16	Cam Newton, CAR	56.6
17	Matt Moore, MIA	54
18	Ryan Fitzpatrick, BUF	51.2
19	Kyle Orton, DEN/KC	50.1
20	Matt Cassel, KC	48.8
21	Andy Dalton, CIN	47.3
22	Alex Smith, SF	46.4
23	Josh Freeman, TB	43.3
24	Rex Grossman, WSH	42.2
25	Colt McCoy, CLE	39.8
26	John Skelton, ARI	39.5
27	Tarvaris Jackson, SEA	37.4
28	Christian Ponder, MIN	35.9
29	Kevin Kolb, ARI	34.2
30	Mark Sanchez, NYJ	33.6
31	Sam Bradford, STL	27.3
32	Tim Tebow, DEN	27.2
33	Curtis Painter, IND	23.4
34	Blaine Gabbert, JAC	20.5