Quantifying Impact: Why NFL’s quarterback rating (QBR) is smarter than any development index

The NFL’s QBR succeeds where most NGO performance indicators fall short: It captures the context and complexity of real world situations, and gives us a blueprint for how we fix Impact Measurement.

[Background]

International aid organizations like the USAID, DFID, and the World Bank use performance indicators to measure the impact of their programs. Quality of life is one example:

Quality of Life Index is a composite of 9 other factors, which are composites of 47 other measurements

OECD’s interactive version is much better, but still based on the same approach of counting and averaging a bunch of stuff:

Better, because YOU define what's important and it recalculates.

Counting things and more things and putting them all together can give false impressions, because sometimes the numbers don’t reflect the context. A number might mean something else if you adjusted for the situations in which these numbers appear. And yet we make big decisions, allocating millions of dollars every day on the basis of these imperfect indicators.

Total Quarterback Rating (QBR)

This is why the QBR is way better. Whereas the old QB rating reflected the skill of the quarterback (answering: “how good is this player?“), the new QBR reflects the player’s contribution to the team’s overall success (answering: “how essential was this player to winning the game?“). It’s the difference between quantifying outputs and quantifying outcomes.

In order to reflect outcomes, and not outputs, the QBR breaks down each of the player’s actions and gives them a positive or negative score, adjusted by a “clutch play” factor. The “clutch play” factor is the brilliant part, because situational context now determines the score. If your team is tied and the quarterback throws a pass for a touchdown to give the team a lead, that action has more value than the identical action taken when you team is already ahead by 14 points. If the pass is 40 yards and the receiver runs for 10, then the quarterback is awarded more points than if he dumps the ball off for 5 yards and the receiver advances it 45 yards himself.

Situational variables affect the rating and give a realistic view of the quarterback’s contribution to the outcome (winning or losing). Given all the data (captured on 4 cameras), it would be crazy if we could not mathematically define the quarterback’s Impact in a game.

“Clutch” moves by smart NGOs

If we desire it, international development indicators could just as easily incorporate “clutch factors” too! A women’s giving circle that risks all to start an agro-business during a drought and succeeds should be given a much higher performance and outcomes rating than a large business that invests 5% of it’s profits in a risky venture. A local NGO with minimal budget run by volunteers that manages to “solve” a tricky problem like getting loitering kids back into school should be awarded much higher performance points for a clutch move, given their lack of capacity to intervene. Governments that test multiple interventions to improve education and disseminate findings about the one that succeeds should have a better credit rating than a government that fails to invest and experiment – because honestly, much more of the taxpayers’ money is likely to be wasted complacent leadership. Recognition for “Clutch play” and “Clutch moves” by NGOs is lacking in international development, as is the concept that situations affect the appearance of success.

People still resist conclusions about outcomes

I’m shocked at all the fan negativity about Total QBR, but perhaps that’s precisely because it challenges our comfortable notion that a “good try” is as good as actually winning the game. To illustrate, here are some quarterbacks compared in the new and old rating systems: (2011 rating source) (2011 QBR source) (full season 2011 QBR statistics)

QBR statistics From Week 3:

QB New QBR rating

 

QB Old rating Change (Old -> New)
Tom Brady 85.3  Aaron Rodgers QB, GB 120.9 +1
Ryan Fitzpatrick 83.7 Tom Brady QB, NE 113.8 +4
Aaron Rodgers 79.1 Matthew Stafford QB, DET 110.7 -3
Matt Hasselbeck 77.8 Drew Brees QB, NO 109.7 +3
Drew Brees 75.8 Eli Manning QB, NYG 104.3 -1
Matt Schaub 74.7 Ryan Fitzpatrick QB, BUF 103.5 +2
Tony Romo 73.9 Matt Hasselbeck QB, TEN 102.2 +2
Matthew Stafford 70.9 Matt Schaub QB, HOU 101.9 -5
Jason Campbell 67.5 Tony Romo QB, DAL 95.8 +1
Eli Manning 64.9 Jason Campbell QB, OAK 93.8 -5
Joe Flacco 64.2 Kevin Kolb QB, ARI 93.8 +1
Chad Henne 62 Joe Flacco QB, BAL 91.9 +7
Josh Freeman 55.3 Alex Smith QB, SF 91.3 +14
Ben Roethlisberger 52.9 Mark Sanchez QB, NYJ 90.9 +3
Philip Rivers 51.3 Michael Vick QB, PHI 87.7 +6
Alex Smith 49 Rex Grossman QB, WAS 86.4 -4
Cam Newton 47.8 Ben Roethlisberger QB, PIT 85.5 +1
Rex Grossman 47.5 Cam Newton QB, CAR 85.1 -2
Kyle Orton 47.2 Chad Henne QB, MIA 82.4 +5
Kevin Kolb 44.9 Jay Cutler QB, CHI 82.4 -9
Colt McCoy 43.5 Philip Rivers QB, SD 82.1 +4
Donovan McNabb 42.8 Andy Dalton QB, CIN 82.1 +4
Michael Vick 42.6 Matt Ryan QB, ATL 81.5 -8
Matt Ryan 40.5 Kyle Orton QB, DEN 79.1 -1
Sam Bradford 33.7 Colt McCoy QB, CLE 78.4 +4
Tarvaris Jackson 32.8 Donovan McNabb QB, MIN 78.1 +2
Mark Sanchez 32.3 Josh Freeman QB, TB 76.1 -13
Jay Cutler 29.4 Tarvaris Jackson QB, SEA 73.7 -8
Andy Dalton 26.5 Sam Bradford QB, STL 73.3 -7
Matt Cassel 22.3 Kerry Collins QB, IND 65.9 +1
Kerry Collins 13.6 Matt Cassel QB, KC 65.5 -1

 

[Note: the Change column refers to the first quarterback named in a row (i.e. Ryan Fitzpatrick = +4)]

If you run down the list, you’ll notice that highly rated QBR quarterbacks are on teams with a better record. Four weeks ago nobody would have consider Ryan Fitzpatrick (a 7th round draft pick and the only Harvard Graduate to start as a quarterback in the NFL in decades!) to be a superstar, but each week Fitzpatrick has carried his team from 14, 17, and 20 points down back to victory in the 4th quarter. He is the poster child for a clutch player.

Likewise, Josh Freeman of the Tampa Bay Buccaneers doesn’t put up numbers but consistently wins close games (leading the NFL in 4th quarter come-backs last year). [Update: By the end of the season, both of these trends had reversed, and their QBRs were down to 18th and 23rd place for Fitzpatrick and Freeman respectively.]

However, the QBR isn’t useful for Fantasy Football, where you can start awesome players who rack up monster stats on teams that often lose games. The quarterback on my current fantasy team is a perfect case in point: Mark Sanchez is the most overrated QB in the NFL (13 spots higher based on his stats, compared with his contribution to winning games for the Jets), and yet he makes perfect sense to start in a Fantasy League because the rules never penalize players for losing games. Fantasy Football is all about individual outputs, not outcomes. So I can win in the fake world by focusing on intermediate factors that don’t matter to winning the game.

There is an obvious real-world analogy here with certain aid agencies that focus on outputs instead of striving to measure outcomes: They are merely playing “fantasy football” in international development, and not necessary winning games. What’s the solution? Devise indicators that adopt complexity-based thinking and adjust parameters situationally, until the indicators are really incorporating as many factors as are needed to understand the real world.

I admit this is hard, but not impossible. It simply requires another system to know which models are improvements over the older ideas. Head-to-head competition between models has been used in both molecular Neuron modeling and nuclear retaliation protocols, and certain models actually DO perform better than all the rest. (Sadly, the best model for neuron interactions only predicts a quarter of the real neuron behavior, and the best nuclear retaliation game theory still destroys the world) – so not every problem can be fully modeled (or fully “indexed”).

2011 Complete Season QBR ratings

Rank Name QBR
1 Aaron Rodgers, GB 85.2
2 Drew Brees, NO 84
3 Tom Brady, NE 74.2
4 Tony Romo, DAL 70.1
5 Matt Ryan, ATL 67.5
6 Matt Schaub, HOU 66.7
7 Matthew Stafford, DET 65.1
8 Philip Rivers, SD 64.3
9 Ben Roethlisberger, PIT 63.3
10 Michael Vick, PHI 63.2
11 Carson Palmer, OAK 62.3
12 Matt Hasselbeck, TEN 61.6
13 Eli Manning, NYG 61
14 Jay Cutler, CHI 59.5
15 Joe Flacco, BAL 57.9
16 Cam Newton, CAR 56.6
17 Matt Moore, MIA 54
18 Ryan Fitzpatrick, BUF 51.2
19 Kyle Orton, DEN/KC 50.1
20 Matt Cassel, KC 48.8
21 Andy Dalton, CIN 47.3
22 Alex Smith, SF 46.4
23 Josh Freeman, TB 43.3
24 Rex Grossman, WSH 42.2
25 Colt McCoy, CLE 39.8
26 John Skelton, ARI 39.5
27 Tarvaris Jackson, SEA 37.4
28 Christian Ponder, MIN 35.9
29 Kevin Kolb, ARI 34.2
30 Mark Sanchez, NYJ 33.6
31 Sam Bradford, STL 27.3
32 Tim Tebow, DEN 27.2
33 Curtis Painter, IND 23.4
34 Blaine Gabbert, JAC 20.5
Advertisements

One thought on “Quantifying Impact: Why NFL’s quarterback rating (QBR) is smarter than any development index

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s