Should Rookie QBs Start?

Wages of Wins author and loyal Detroit Lions fan Dave Berri recently asked me about research on whether rookie quarterbacks are better off standing on the sideline all season. One of the big questions in Detroit this year will certainly be whether overall number one pick Matthew Stafford should start at QB. The central issue is whether starting a rookie QB somehow harms his long-term development. Does a year holding a clipboard allow rookies to adapt to the NFL and boost their prospects for a successful career?

Names like Boller, Harrington, Couch, Shuler, and Carr underscore the danger of starting rookie passers. But there are also names such as Manning, Roethlisberger, Marino, Elway, and Aikman that indicate that starting as a rookie may not be so damaging to a QB's development.

This is a question I get often but haven’t looked into it because of complications associated with the issue. First, there aren’t that many top QBs to analyze—the sample size is fairly small. Any inference we make needs to keep in mind the small sample. Second, there is a problem of bias in the data. The better QBs would be the ones to earn starting jobs their rookie year, and would also likely tend to be the ones to enjoy successful careers.

The trick would be to properly account for underlying QB potential, which would be quite a trick. If we knew that, that’s all we’d ever really need to know about a QB. There's no perfect way to measure that, but in the end, I think the most reasonable variable to indicate potential is overall draft pick number. It’s something that is established prior to any decision to start or not as a rookie. Also, although it is often an unreliable predictor of career performance for individual QBs, it correlates very well for QBs as a group. In other words, the number one QB might not pan out or even be better than the second QB taken in any particular draft year. But as a whole, the top picks reliably tend to outperform subsequent picks.

Data

I looked at first and second round QBs drafted between 1980 and 2004. I chose 1980 because it is roughly the dawn of the modern NFL passing rules. I chose 2004 to allow at least four years to asses each QB’s performance. Data are from Pro-Football-Reference.com. To measure career success, I used Adjusted Yards per Attempt (Adj YPA). This is passing yards minus 45 yards for every interception, per pass attempt. For QBs with very low pass attempts and spurious Adj YPA stats, I replaced their actual Adj YPA with 3.0 YPA, generally the floor for QBs with a reasonable sample size of attempts. (There is no reason to expect 1989 Chief’s 2nd-round pick Mike Elkins to lose 20 yards for every pass based on only 2 attempts.)

Methodology

The first step is to account for potential using overall draft pick number. The graph below plots career Adj YPA by pick number. The scatterplot is pretty random for any individual QB, but as a whole there is a predictable trend. Top draft picks tend to end up as NFL top passers. I estimated the expected Adj YPA based on pick number. A QB from the top of the first round would be expected to average 5.0 Adj YPA, while a QB from the bottom of the second round would be expected to average 4.1 Adj YPA.



We can see a clear trend. Unsurprisingly, top picks would be expected to end up with better career passing stats than later picks. A linear regression estimates what the baseline expectation should be for each slot in the draft.

Next I calculated the 'Adj YPA above expected' for each QB. Now we can compare QBs who started their first year to those that didn’t, while holding “potential” equal.

But how do we define “started their first year?” How many starts qualifies--4, 5, 11? I don’t know, so let’s start by looking at the whole picture.


Results

We can use that baseline to compare each QB's career Adj YPA. Some QBs did better than expected given their draft slots and some did worse. We can now test if there is a connection between better than expected performance and the number of rookie year starts.



As it turns out, it appears that QBs with more rookie starts tend to enjoy greater career success, even accounting for draft order.

Some of you might have noticed that what I've really done here is a crude multivariate regression. Holding for draft order, I estimated the effect of games started. What if we just do the regression directly?

As expected we get a small negative effect with draft order. (The higher the pick number, the worse the expected stats.) The Games Started variable is positive and significant at p=0.03. The model as a whole has an r-squared of 0.15--small in absolute terms, but considerable given the highly random individual variance in QB careers.

But this is only one way of looking at whether a QB was a starter or not. What if we draw a line at say, 5 rookies starts--below 5 starts he's not a rookie starter and above it he is. The group of QBs with 5 or less starts averages -0.01 Adj YPA above expected, and the group with 6 or more starts averages +0.3 Adj YPA above expected. If we define it at zero starts, those with no rookie starts averaged -0.03 Adj YPA above expected, while those with at least one start averaged +0.02 starts above expected. In fact, no matter where I chose the endpoints of the groups, from 3 to 13 starts, the group with more starts outperforms the group with fewer starts by about 0.4 Adj YPA.

Conclusion

Does this mean teams should rush their rookies out to face the onslaught of NFL defenses to somehow make them better? I really doubt it. If I had to bet, I'd say that we simply haven't fully accounted for QB "potential" using draft order alone. I think the better QBs, those with the best chances of career success, often gain and maintain starting positions earlier.

But at the very least, we can say this: Given this analysis, there is no reason for a coach to arbitrarily keep a rookie QB on the bench. He should start his best QB, rookie or not, and not worry about incubating him under a ballcap on the sidelines. In the end, it should be the coach's qualitative judgment on the readiness of the player.

Here are the QBs and their stats I used for this article. (It's interesting just to see who the QBs are who exceeded expectations. There are some surprising names--Pennington and Batch at the top, for example. And Is Eli Manning really worse than David Carr? Wow.) You can sort the table by clicking on the column headers.















































































Year Rnd Pick NameTeamAdj YPAExp Adj YPAYr1 GSAdjYPA Abv Exp
2001232Drew Brees 6.04.601.4
1981233Neil Lomax 5.94.671.4
199811Peyton Manning 6.45.0161.4
1998260Charlie Batch 5.54.1121.4
200414Philip Rivers 6.45.001.4
2004111Ben Roethlisberger 6.24.9131.3
1983127Dan Marino6.04.691.3
2000118Chad Pennington 6.14.801.3
1999111Daunte Culpepper 6.14.901.3
1984238Boomer Esiason 5.74.541.2
1985237Randall Cunningham 5.64.541.1
1983124Ken O'Brien 5.74.701.1
1995245Todd Collins 5.34.411.0
1991233Brett Favre 5.54.601.0
1983114Jim Kelly 5.84.800.9
199912Donovan McNabb 5.95.060.9
1983115Tony Eason 5.74.840.8
200311Carson Palmer 5.85.000.8
1987126Jim Harbaugh 5.44.700.7
199513Steve McNair 5.75.020.7
1996242Tony Banks 5.14.4130.7
198311John Elway 5.75.0100.7
199011Jeff George 5.75.0120.6
1997242Jake Plummer 5.14.490.6
2001253Quincy Carter 4.94.380.6
198911Troy Aikman5.65.0110.6
200317Byron Leftwich 5.54.9130.6
198215Jim McMahon 5.55.070.5
1995260Kordell Stewart 4.74.120.5
198613Jim Everett 5.55.050.5
2002132Patrick Ramsey 5.04.650.4
1999250Shaun King 4.74.350.4
1989251Billy Joe Tolliver 4.64.350.3
200111Michael Vick 5.35.020.3
2004122J.P. Losman 5.04.700.3
1987113Chris Miller 5.14.920.2
199311Drew Bledsoe 5.35.0120.2
199515Kerry Collins 5.25.0130.2
198711Vinny Testaverde 5.15.040.1
2003122Rex Grossman 4.84.730.1
1992125Tommy Maddox 4.74.740.0
1986247Jack Trudeau 4.34.3110.0
200211David Carr 5.05.016-0.1
200411Eli Manning 4.95.07-0.1
1991124Todd Marinovich 4.64.71-0.1
1980115Marc Wilson 4.74.80-0.1
199017Andre Ware 4.74.91-0.3
2003119Kyle Boller 4.54.89-0.3
199911Tim Couch 4.75.014-0.3
199416Trent Dilfer 4.65.02-0.3
1999112Cade McNown 4.44.96-0.5
1982244Oliver Luck 3.94.40-0.5
1980128Mark Malone 4.04.60-0.7
200213Joey Harrington 4.35.012-0.7
1986112Chuck Long 4.14.92-0.8
199216David Klingler 4.25.04-0.8
199312Rick Mirer 4.25.016-0.8
198317Todd Blackledge 4.14.90-0.9
1991234Browning Nagle 3.64.50-0.9
1982248Matt Kofler 3.34.30-1.1
199413Heath Shuler 3.75.08-1.3
1992246Tony Sacca 3.04.40-1.4
1992240Matt Blundin 3.04.40-1.4
199913Akili Smith 3.55.04-1.5
1980237Gene Bradley 3.04.5













-1.5
2001259Marques Tuiasosopo 2.74.20-1.5
1989232Mike Elkins 3.04.60-1.6
198716Kelly Stouffer 3.45.00-1.6
1991116Dan McGwire 3.24.81-1.6
1997126Jim Druckenmiller 3.04.71-1.7
199812Ryan Leaf 3.15.09-1.9
198116Rich Campbell 3.05.00-2.0
198214Art Schlichter 3.05.00-2.0


  • Spread The Love
  • Digg This Post
  • Tweet This Post
  • Stumble This Post
  • Submit This Post To Delicious
  • Submit This Post To Reddit
  • Submit This Post To Mixx

19 Responses to “Should Rookie QBs Start?”

  1. Anonymous says:

    Looks like the effect is still mostly selection bias. Since teams likely change their evaluation of rookies significantly between draft day and the middle of the regular season, accounting for draft order doesn't eliminate the bias. High draft picks that look great in preseason/practice are more likely to succeed and get more starts than high draft picks that struggle.

  2. Chase says:

    Good work, as usual, Brian. I actually started working on this right after the draft for the same reasons you did, but trashed the project after seeing pretty similar results. I essentially got a small sign that starting early is better, but like you, I am hesitant to decide which way the causation arrow points.

    One general question -- I assume you are not putting in actual draft order into your regression? Since draft potential, as measured by draft order, is not linear, you wouldn't want to do that. It's the exact reason I came up with this post (http://www.pro-football-reference.com/blog/?p=527). So whenever I'm doing a draft analysis like this, I replace actual draft order with the numbers in those posts.

    It's a very difficult question to answer. Even accounting for draft position, coaches get to see a full training camp out of a guy. Those who aren't very good will have no chance of starting; those who look good will have a chance. So we already have some data (albeit one with lots of noise in it) on a QB besides his draft position by game 1 -- was he an opening day starter?

    In the end, I suspect there's no career difference between a guy starting and not starting as a rookie. And certainly, we haven't seen any evidence to the contrary.

  3. Brian Burke says:

    Thanks, Chase. I should have been more clear on the regression. The dependent variable was Adj YPA (not 'Adj YPA above expected'), and the predictors were draft order and rookie games started.

    I agree with you on the non-linearity of potential, but strictly for Adj YPA, it does look linear, at least for the first 2 rounds. (See First graph above). Maybe Adj YPA doesn't really capture how truly 'good' a QB is. Or, the utility function of YPA isn't linear. In other words having a 5.0 YPA QB is more than 25% better than having a 4.0 YPA QB.

    I don't think this proves anything, just that using this methodology we can't really detect any advantage to waiting. And even if the methodology doesn't capture every little nuance and bias, the true effect must be small enough to be engulfed by other effects, and is therefore probably not big enough to worry about.

    If I have time today, I might rerun it with your AV numbers instead of raw draft order, and see how it turns out.

  4. Brian Burke says:

    Full regression results:
    Dep var: Adj YPA
    const: 4.67
    Rookie GS: coeff 0.051, p=0.03
    Pick#: coeff -0.007, p=0.30
    Model F-stat:3.64, p=0.03
    R-squared: 0.15

  5. Mike says:

    Hi Brian,

    Your site is fast becoming one of my personal favorites. I will add your site to my blogroll at TheSportsDick.com.

    If you disregard draft position and look at what really matters - the adj YPA, four of the top six and 10 of the top 15 performers started four games or less during their rookie season. Conversely if you look at the worst performers, only three of the bottom 20 started more than four games. From that data alone you can conclude that starting a rookie slightly improves their likelihood of a successful career.

    The problem with coming to any conclusing by analyzing this data is that it's largely irrelevant. The factors that determine a successful quarterback include the talent around the rookie going in, the quality of coaching he receives, whether he earns the job or inherits it due to injury, etc. How many of the top QB's would still be at the top of the list if they had been drafted by Detroit? I submit that not all of them would have enjoyed the same success as they did.

  6. Brian Burke says:

    Mike- I largely have to agree with your take. There are any number of factors that influence QB success. If we could model all of them realistically it would be quite the breakthrough.

    However, the hope is that with a large enough data set, the factors not in the model (coaching, team talent, injury) will largely even out. Unfortunately, this data set is not particularly large.

    But in the end, if you put a gun to my head, I'd bet that if there is some scarring effect of starting as a rookie, we would have seen it here.

  7. Mike says:

    Brian,

    It'd be extremely difficult to quantify the intangibles. I pose this question. Is it safe to assume that the fact that a team drafting a QB in the first two rounds indicates that the franchise is coming off of a bad year? No one drafts a QB to "take the best player remaining". Generally a quarterback taken on one day one is to fill a need. A team that needs a QB usually isn't in good shape.

    I know this is a bit of a stretch. To support this, consider that of the 51 QB's taken in the first round, only 10 of them were taken in with one of the last 10 picks of the round - the picks owned by presumably strong teams (I know there are anomalies caused by trades and such).

    Of the 22 2nd rd QB's taken, only five were among the last 10 of the round.

    It would appear that the median number for avg YPA is around 4.8 or so. Of the 36 QB's below average, only eight of them were drafted with the last 10 picks of either round. Of those eight, none of them started more than five games their rookie year.

    Looking at the number of games started the rookie year of all the players on the list, the median number is just less than 4. 13 of the 36 teams above that threshold were below the 4.8 adj YPA metric. Of those 13 'busts', seven were taken at the end of their respective round.

    To clarify, of 15 late rounders, seven were busts that started four games or more.

    If 4.8 is the threshold, than only one #1 overall pick failed to meet that standard. Of those 11 #1's, only three of them failed to start seven games or more. Interesting enough, of the eight players selected # 2 or 3, five of them were busts.

    Of all of the busts, almost none of them were drafted by teams that had ever won a Super Bowl.

    I think that's about the only conclusion you can come to. That bad teams draft badly. The performance of their rookie quarterbacks is nothing more than a reflection of the ineptitude of the team that selected him.

  8. Brian Burke says:

    In case anyone was curious, I reran the estimates using Chase's Approximate Value (AV) numbers in place of the overall draft pick number. The results are very similar.

    Rookie Year Games Started is still positive and significant. The coefficient for Rookie GS drops a little from 0.05 to 0.04.

  9. Brian Burke says:

    I ran a few more versions. I factored in sacks--"Adj Net YPA above expected." The result is the same if not more pronounced. Rookie GS is positive and significant.

    I also used Chases AV numbers in conjunction with the sack-added data, and the result is again positive and significant for rookie GS.

    Lastly, I reran the regression using only 1st Round players. Same result with slightly less significance, which would be expected due to the smaller sample size.

  10. JulianW says:

    It would be interesting to see the results if you compared players who started, say, 10+ games to those who started 0 games in their rookie year. This would (mostly) filter out rookie QBs who were benched after a few bad starts, presumably because they were performing poorly. With these benched rookie QBs in your dataset, it's not surprising that players with more rookie-year starts appear to have better careers: they performed better early in the season because they were simply better players (at least on average) to begin with!

  11. Ryan says:

    Brian,
    Really interesting study here. As a Jets fan, I've been wondering about this recently: historically, it seems to me most quarterbacks drafted in the first round will start early if they have inexperienced QBs on the team already (Ryan, Flacco), but will sit and learn for a while if there are decent, veteran QBs on the roster (Leinart/Warner, Rivers/Brees...). I agree that - using the Jets as an example - if Sanchez turns out to be better than Clemens, et al (with 8 career starts), he should start right off the bat and will benefit from a good team around him and the experience gained in the long run. On the flipside, if they happened to have a journeyman QB like Jeff Garcia, showing him the ropes, I wouldn't be surprised if sitting for a year and learning the system & the playbook helps a young QB over time.

  12. parker says:

    Brian,
    I suspect your results may change if you account for the quality of the other quarterbacks on the roster of the rookie quarterback.

  13. Brian Burke says:

    I'm not sure I follow. I can see how that would affect number of rookie games games started, but I don't see how that affects the career prospects of the rookie. In other words, it would affect the independent variable, but not the dependent variable I'm interested in.

  14. Anonymous says:

    My hat is off to you Brian, you did an amazing job with this study. My only concern is that the primary measure of success, yards per attempt, is influenced by the style of offense a quarterback plays in. For example, a system that runs on first down, runs on second down, and then asks the passer to make a play on third and long (i.e. Eli Manning's Giants) would artificially hurt a ranking, while an offense that regularly passes on first down - when most teams play to stop the run - would artificially help it (i.e. Daunte Culpepper's years in Minnesota). But this is a small flaw that would not significantly improve variance prediction, and would be hard to quantify.

    As a Lions fan, the question of how soon to play a rookie QB is pretty pressing right now. We screwed up Joey Harrington by turning him into a check-down machine while running for his life, and the general thought is that he started before he was ready. On the other hand, Chuck Long sat for the better part of three years before he saw the field and just couldn't do it on the field by then. It was like he forgot how to play in a game.

    So, with Matthew Stafford and still not a very good offensive line, this data suggests that the Lions should go ahead and start him as soon as the coaching staff deems he's ready. There are successful QBs and busts that didn't start as a rookie, or that started full seasons. He'll either be good or not based on his talent and preparation. A long incubation doesn't help.

  15. Anonymous says:

    Dear Brian,

    I just came across your blog today coming from the NYT and I'm really happy I did. It is great to see what you are doing.

    I would like to suggest a couple of things for this particular analysis. You can approach it as using Analysis of Covariance (ANCOVA) to account for the draft order.

    This approach would be more powerful because I'm assuming that the first model you use to define your response variable (above or below average) is not very strong to begin with.

    As other people suggested, I'm not sure that YPA is the best choice because it depends on the type of offense. Maybe QB ratings could work better? Maybe adjusting for number of games played.

    Also the choice of the variable number of games started as a rookie is the best. You found that those QB's who started more games are a little better than those who started a smaller numer of games. That is something to be expected, because if the QB had sucess in the first games he plays, that means the coach would be more likely to start him again.

    This is a great blog and I'll be following your work. I hope this will encourage more people to learn and enjoy math.

  16. Brian Burke says:

    Thanks for the great suggestion. I'm definitely going to do that.

  17. TheBard says:

    I've looked over QBs drafted just in the first round (and Drew Brees since he's basically a first rounder) since Manning-Leaf up till Young-Lineart. Rookie first rounders who got the starting job before game four or five tend to bomb except for Manning, Big Ben and Leftwhich. Of those three, Big Ben and Leftwhich were playing for quality teams. The only QB I'm not sure about is Akili Smith since I haven't been able to find what games he did start in his rookie season.

  18. Brian Burke says:

    So you're looking at when in the rookie year they got their starts? Interesting.

  19. Anonymous says:

    I'm currently using your data for my Junior Research Project based on the thesis that Young QB's take longer to develop than runningbacks, due to the new offense,recieving cores, and defenses. Can you think of any other factors that can make or break a rookie quarterback?

Leave a Reply

Note: Only a member of this blog may post a comment.