The NFL’s QBR succeeds where most NGO performance indicators fall short: It captures the context and complexity of real world situations, and gives us a blueprint for how we fix Impact Measurement.
International aid organizations like the USAID, DFID, and the World Bank use performance indicators to measure the impact of their programs. Quality of life is one example:
OECD’s interactive version is much better, but still based on the same approach of counting and averaging a bunch of stuff:
Counting things and more things and putting them all together can give false impressions, because sometimes the numbers don’t reflect the context. A number might mean something else if you adjusted for the situations in which these numbers appear. And yet we make big decisions, allocating millions of dollars every day on the basis of these imperfect indicators.
Total Quarterback Rating (QBR)
This is why the QBR is way better. Whereas the old QB rating reflected the skill of the quarterback (answering: “how good is this player?“), the new QBR reflects the player’s contribution to the team’s overall success (answering: “how essential was this player to winning the game?“). It’s the difference between quantifying outputs and quantifying outcomes.
In order to reflect outcomes, and not outputs, the QBR breaks down each of the player’s actions and gives them a positive or negative score, adjusted by a “clutch play” factor. The “clutch play” factor is the brilliant part, because situational context now determines the score. If your team is tied and the quarterback throws a pass for a touchdown to give the team a lead, that action has more value than the identical action taken when you team is already ahead by 14 points. If the pass is 40 yards and the receiver runs for 10, then the quarterback is awarded more points than if he dumps the ball off for 5 yards and the receiver advances it 45 yards himself.
Situational variables affect the rating and give a realistic view of the quarterback’s contribution to the outcome (winning or losing). Given all the data (captured on 4 cameras), it would be crazy if we could not mathematically define the quarterback’s Impact in a game.
“Clutch” moves by smart NGOs
If we desire it, international development indicators could just as easily incorporate “clutch factors” too! A women’s giving circle that risks all to start an agro-business during a drought and succeeds should be given a much higher performance and outcomes rating than a large business that invests 5% of it’s profits in a risky venture. A local NGO with minimal budget run by volunteers that manages to “solve” a tricky problem like getting loitering kids back into school should be awarded much higher performance points for a clutch move, given their lack of capacity to intervene. Governments that test multiple interventions to improve education and disseminate findings about the one that succeeds should have a better credit rating than a government that fails to invest and experiment – because honestly, much more of the taxpayers’ money is likely to be wasted complacent leadership. Recognition for “Clutch play” and “Clutch moves” by NGOs is lacking in international development, as is the concept that situations affect the appearance of success.
People still resist conclusions about outcomes
I’m shocked at all the fan negativity about Total QBR, but perhaps that’s precisely because it challenges our comfortable notion that a “good try” is as good as actually winning the game. To illustrate, here are some quarterbacks compared in the new and old rating systems: (2011 rating source) (2011 QBR source) (full season 2011 QBR statistics)
QBR statistics From Week 3:
|QB||New QBR rating
|QB||Old rating||Change (Old -> New)|
|Tom Brady||85.3||Aaron Rodgers QB, GB||120.9||+1|
|Ryan Fitzpatrick||83.7||Tom Brady QB, NE||113.8||+4|
|Aaron Rodgers||79.1||Matthew Stafford QB, DET||110.7||-3|
|Matt Hasselbeck||77.8||Drew Brees QB, NO||109.7||+3|
|Drew Brees||75.8||Eli Manning QB, NYG||104.3||-1|
|Matt Schaub||74.7||Ryan Fitzpatrick QB, BUF||103.5||+2|
|Tony Romo||73.9||Matt Hasselbeck QB, TEN||102.2||+2|
|Matthew Stafford||70.9||Matt Schaub QB, HOU||101.9||-5|
|Jason Campbell||67.5||Tony Romo QB, DAL||95.8||+1|
|Eli Manning||64.9||Jason Campbell QB, OAK||93.8||-5|
|Joe Flacco||64.2||Kevin Kolb QB, ARI||93.8||+1|
|Chad Henne||62||Joe Flacco QB, BAL||91.9||+7|
|Josh Freeman||55.3||Alex Smith QB, SF||91.3||+14|
|Ben Roethlisberger||52.9||Mark Sanchez QB, NYJ||90.9||+3|
|Philip Rivers||51.3||Michael Vick QB, PHI||87.7||+6|
|Alex Smith||49||Rex Grossman QB, WAS||86.4||-4|
|Cam Newton||47.8||Ben Roethlisberger QB, PIT||85.5||+1|
|Rex Grossman||47.5||Cam Newton QB, CAR||85.1||-2|
|Kyle Orton||47.2||Chad Henne QB, MIA||82.4||+5|
|Kevin Kolb||44.9||Jay Cutler QB, CHI||82.4||-9|
|Colt McCoy||43.5||Philip Rivers QB, SD||82.1||+4|
|Donovan McNabb||42.8||Andy Dalton QB, CIN||82.1||+4|
|Michael Vick||42.6||Matt Ryan QB, ATL||81.5||-8|
|Matt Ryan||40.5||Kyle Orton QB, DEN||79.1||-1|
|Sam Bradford||33.7||Colt McCoy QB, CLE||78.4||+4|
|Tarvaris Jackson||32.8||Donovan McNabb QB, MIN||78.1||+2|
|Mark Sanchez||32.3||Josh Freeman QB, TB||76.1||-13|
|Jay Cutler||29.4||Tarvaris Jackson QB, SEA||73.7||-8|
|Andy Dalton||26.5||Sam Bradford QB, STL||73.3||-7|
|Matt Cassel||22.3||Kerry Collins QB, IND||65.9||+1|
|Kerry Collins||13.6||Matt Cassel QB, KC||65.5||-1
[Note: the Change column refers to the first quarterback named in a row (i.e. Ryan Fitzpatrick = +4)]
If you run down the list, you’ll notice that highly rated QBR quarterbacks are on teams with a better record. Four weeks ago nobody would have consider Ryan Fitzpatrick (a 7th round draft pick and the only Harvard Graduate to start as a quarterback in the NFL in decades!) to be a superstar, but each week Fitzpatrick has carried his team from 14, 17, and 20 points down back to victory in the 4th quarter. He is the poster child for a clutch player.
Likewise, Josh Freeman of the Tampa Bay Buccaneers doesn’t put up numbers but consistently wins close games (leading the NFL in 4th quarter come-backs last year). [Update: By the end of the season, both of these trends had reversed, and their QBRs were down to 18th and 23rd place for Fitzpatrick and Freeman respectively.]
However, the QBR isn’t useful for Fantasy Football, where you can start awesome players who rack up monster stats on teams that often lose games. The quarterback on my current fantasy team is a perfect case in point: Mark Sanchez is the most overrated QB in the NFL (13 spots higher based on his stats, compared with his contribution to winning games for the Jets), and yet he makes perfect sense to start in a Fantasy League because the rules never penalize players for losing games. Fantasy Football is all about individual outputs, not outcomes. So I can win in the fake world by focusing on intermediate factors that don’t matter to winning the game.
There is an obvious real-world analogy here with certain aid agencies that focus on outputs instead of striving to measure outcomes: They are merely playing “fantasy football” in international development, and not necessary winning games. What’s the solution? Devise indicators that adopt complexity-based thinking and adjust parameters situationally, until the indicators are really incorporating as many factors as are needed to understand the real world.
I admit this is hard, but not impossible. It simply requires another system to know which models are improvements over the older ideas. Head-to-head competition between models has been used in both molecular Neuron modeling and nuclear retaliation protocols, and certain models actually DO perform better than all the rest. (Sadly, the best model for neuron interactions only predicts a quarter of the real neuron behavior, and the best nuclear retaliation game theory still destroys the world) – so not every problem can be fully modeled (or fully “indexed”).
2011 Complete Season QBR ratings
|1||Aaron Rodgers, GB||85.2|
|2||Drew Brees, NO||84|
|3||Tom Brady, NE||74.2|
|4||Tony Romo, DAL||70.1|
|5||Matt Ryan, ATL||67.5|
|6||Matt Schaub, HOU||66.7|
|7||Matthew Stafford, DET||65.1|
|8||Philip Rivers, SD||64.3|
|9||Ben Roethlisberger, PIT||63.3|
|10||Michael Vick, PHI||63.2|
|11||Carson Palmer, OAK||62.3|
|12||Matt Hasselbeck, TEN||61.6|
|13||Eli Manning, NYG||61|
|14||Jay Cutler, CHI||59.5|
|15||Joe Flacco, BAL||57.9|
|16||Cam Newton, CAR||56.6|
|17||Matt Moore, MIA||54|
|18||Ryan Fitzpatrick, BUF||51.2|
|19||Kyle Orton, DEN/KC||50.1|
|20||Matt Cassel, KC||48.8|
|21||Andy Dalton, CIN||47.3|
|22||Alex Smith, SF||46.4|
|23||Josh Freeman, TB||43.3|
|24||Rex Grossman, WSH||42.2|
|25||Colt McCoy, CLE||39.8|
|26||John Skelton, ARI||39.5|
|27||Tarvaris Jackson, SEA||37.4|
|28||Christian Ponder, MIN||35.9|
|29||Kevin Kolb, ARI||34.2|
|30||Mark Sanchez, NYJ||33.6|
|31||Sam Bradford, STL||27.3|
|32||Tim Tebow, DEN||27.2|
|33||Curtis Painter, IND||23.4|
|34||Blaine Gabbert, JAC||20.5|
One thought on “Quantifying Impact: Why NFL’s quarterback rating (QBR) is smarter than any development index”
Hi, the whole thing is going nicely here aand ofcourese ebery onne is
sharing facts, that’s actually fine, keep up writing.