In this post I'll recap how my QB rating is determined, and I'll add a new dimension--adjustments for opposing pass defenses.
QB Rating Recap
The QB rating I post often is different from most, including the NFL passer rating. The stat I use to compare QB performance is +WP16 or wins added per 16 games. +WP16 estimates the number of wins above average a quarterback's performance would mean to his team over a full regular season.
The +WP16 stat estimates the wins added based on a regression that analyzes what makes teams win. The QB stats included are passing yards per attempt, rushing yards per attempt, sack yards per attempt, fumble rates, and interception rates. +WP16 isolates QB performance by assuming an average rushing game and an average defense.
First downs and touchdowns are not included despite their apparent importance because they are intermediate outcomes between actual QB performance and the outcome of interest--team wins.
Perhaps the most unique aspect of +WP16 is that it does not include passing yards gained by receivers after the catch (YAC). Instead it is based on air yards--the yards the ball travels through the air forward of the line of scrimmage. The percent of passing yards made up of YAC varies widely from QB to QB. Brodie Croyle gets 68% of "his" yards from YAC while Ben Roethlisberger gets only 34% of his yards from YAC.
There is some debate over whether or how much YAC should be credited to QBs. In my mind, the answer is much closer to none than to all.
There is statistical evidence that YAC belongs to the receiver. Year-to-year correlations of YAC are far stronger for receivers than for QBs. Accuracy rates and QBs' abilities to read the defense and "see the field" (measured by interception rates) do not contribute to YAC. It appears that YAC is determined by receiver ability far more than QB ability.
The types of passes thrown by a QB also help determine YAC. Screen passes and check downs to RBs, which are not difficult passes, tend to produce more YAC relative to air yards. Deep out passes, which are more difficult and require more QB skill, tend to generate more air yards and often no YAC at all. The other type of pass that can create a lot of YAC is when a WR beats the defense deep by simply out-running them. Although it certainly helps if a QB can get the ball to a receiver who has gotten behind his coverage, those types of plays are rare and are really made possible by the WR, at least in my mind.
For the reasons above, I believe the truer measure of a QB's performance is one that does not include YAC. Here is the ranking of QBs based on performance so far this season:
The biggest surprise at the top of the list is David Garrard. He's ranked so high primarily because of his phenomenally low turnover rate, having thrown only three interceptions all year. His air yards per attempt (AY/A) is impressive too. He ranks among the elite QBs in the league with 4.8 yds/att. In total, Garrard ranks second behind Brady with +2.21 wins added.
Adjusting for Opponents' Pass Defense
When comparing team performance, opponent strength matters a great deal. Schedules in the NFL are not balanced. During the regular season a team only plays 13 of the 31 other teams in the league, 3 of them twice. If those 3 teams a team's own division are among the toughest, and the 18 teams it never plays comprise some of the weakest opponents, then the team would appear to be much worse than it would be if it had an average schedule.
Early in the season opponent adjustments are most important, because as the season goes on and each team faces more different types of opponents, each team's strength of schedule tends to even out. But not completely.
It stands to reason that QB performance ratings would also be affected by opponent strength, specifically opponent pass defense. To confirm this effect, each team's strength of pass defense was computed. That includes average pass yards per attempt allowed, sack yards per pass play, and interception rate. Each stat was weighted according to how important they are towards winning based on the same regression model that determine the weights of the QB rating. This produced a unit-less pass defense score.
Then each QB's average opponent pass defense strength was determined. QBs were counted as having played a team if they had more than 10 attempts in a game.
To see how opposing pass defenses affect QB performance, the correlation coefficient between each QB's rating and his opponents' average pass defense strength was calculated. The correlation was 0.098, which is weak but what we'd expect. About 9% of a QBs season-long performance can be attributed to his opponent's pass defense. Or strictly speaking, one standard deviation (SD) increase in opposition strength results in one SD reduction in QB performance.
The last step in the process was to adjust the QB rating (+WP16) accordingly. Each QB's rating was adjusted by 0.098 SDs for each SD of opponent strength. The stronger the average opponents' pass defense, the higher the adjustment for the QB.
The table below lists each QB's unadjusted rating, his average opponent's pass defense strength, and the adjusted rating. To make the numbers a little easier to understand, pass defense strength is converted into GWP (generic win probability), or the probability a pass defense would win a game against a league-average opponent given an average run defense and offense.
|Rank||Name||+WP16||Opp Pass D Str||Adj +WP16|
At this point in the season (going into week 16), we wouldn't expect there to be much change in the ratings by accounting for opponent strength. Most teams have faced some good pass defenses and some weak ones. And indeed there is relatively little change in the rankings.
There are some notable exceptions, however. The QB who faced the weakest pass defenses was Steve McNair who played against CIN, ARI, CLE, and SF, all of which had below average pass defenses. McNair still managed to turn in a miserable performance in 2007 due to age, injury, or probably both.
The other notable exception is David Garrard. Garrard faced the toughest pass defenses of any QB in 2007. If not for an injury that kept him out of three games against TB, NO, and TEN in weeks 8,9, and 10, he would have an even tougher year. Despite his uphill battle, Garrard is playing extremely well this season. Before adjusting for opponent strength he ranked second among all QBs (not even counting for his week 17 demolition of OAK.)
In the year of Tom Brady and the Patriots, it may be David Garrard who is the best QB this season. After factoring in opponent strength, his +WP16 rating jumps from +2.2 to +2.5, edging Brady for the top spot. That's remarkable considering Garrard wasn't even his team's starter in training camp. Jaguars coach, Jack Del Rio, looks like a genius for cutting Leftwhich and sticking with Garrard.