Comments on Advanced NFL Stats: Should Rookie QBs Start?

I'm currently using your data for my Junior Re...

2010-03-01T22:08:41.623-05:00

I'm currently using your data for my Junior Research Project based on the thesis that Young QB's take longer to develop than runningbacks, due to the new offense,recieving cores, and defenses. Can you think of any other factors that can make or break a rookie quarterback?

So you're looking at when in the rookie...

2009-07-31T13:26:16.618-04:00

So you're looking at when in the rookie year they got their starts? Interesting.

I've looked over QBs drafted just in the first...

2009-07-31T13:20:14.613-04:00

I've looked over QBs drafted just in the first round (and Drew Brees since he's basically a first rounder) since Manning-Leaf up till Young-Lineart. Rookie first rounders who got the starting job before game four or five tend to bomb except for Manning, Big Ben and Leftwhich. Of those three, Big Ben and Leftwhich were playing for quality teams. The only QB I'm not sure about is Akili Smith since I haven't been able to find what games he did start in his rookie season.

Thanks for the great suggestion. I'm definitel...

2009-07-28T18:02:11.983-04:00

Thanks for the great suggestion. I'm definitely going to do that.

Dear Brian, I just came across your blog today co...

2009-07-28T10:17:47.476-04:00

Dear Brian,

I just came across your blog today coming from the NYT and I'm really happy I did. It is great to see what you are doing.

I would like to suggest a couple of things for this particular analysis. You can approach it as using Analysis of Covariance (ANCOVA) to account for the draft order.

This approach would be more powerful because I'm assuming that the first model you use to define your response variable (above or below average) is not very strong to begin with.

As other people suggested, I'm not sure that YPA is the best choice because it depends on the type of offense. Maybe QB ratings could work better? Maybe adjusting for number of games played.

Also the choice of the variable number of games started as a rookie is the best. You found that those QB's who started more games are a little better than those who started a smaller numer of games. That is something to be expected, because if the QB had sucess in the first games he plays, that means the coach would be more likely to start him again.

This is a great blog and I'll be following your work. I hope this will encourage more people to learn and enjoy math.

My hat is off to you Brian, you did an amazing job...

2009-05-27T14:56:33.317-04:00

My hat is off to you Brian, you did an amazing job with this study. My only concern is that the primary measure of success, yards per attempt, is influenced by the style of offense a quarterback plays in. For example, a system that runs on first down, runs on second down, and then asks the passer to make a play on third and long (i.e. Eli Manning's Giants) would artificially hurt a ranking, while an offense that regularly passes on first down - when most teams play to stop the run - would artificially help it (i.e. Daunte Culpepper's years in Minnesota). But this is a small flaw that would not significantly improve variance prediction, and would be hard to quantify.

As a Lions fan, the question of how soon to play a rookie QB is pretty pressing right now. We screwed up Joey Harrington by turning him into a check-down machine while running for his life, and the general thought is that he started before he was ready. On the other hand, Chuck Long sat for the better part of three years before he saw the field and just couldn't do it on the field by then. It was like he forgot how to play in a game.

So, with Matthew Stafford and still not a very good offensive line, this data suggests that the Lions should go ahead and start him as soon as the coaching staff deems he's ready. There are successful QBs and busts that didn't start as a rookie, or that started full seasons. He'll either be good or not based on his talent and preparation. A long incubation doesn't help.

I'm not sure I follow. I can see how that would af...

2009-05-21T08:18:58.585-04:00

I'm not sure I follow. I can see how that would affect number of rookie games games started, but I don't see how that affects the career prospects of the rookie. In other words, it would affect the independent variable, but not the dependent variable I'm interested in.

Brian, I suspect your results may change if you ac...

2009-05-21T07:52:08.018-04:00

Brian,
I suspect your results may change if you account for the quality of the other quarterbacks on the roster of the rookie quarterback.

Brian, Really interesting study here. As a Jets f...

2009-05-20T02:59:59.469-04:00

Brian,
Really interesting study here. As a Jets fan, I've been wondering about this recently: historically, it seems to me most quarterbacks drafted in the first round will start early if they have inexperienced QBs on the team already (Ryan, Flacco), but will sit and learn for a while if there are decent, veteran QBs on the roster (Leinart/Warner, Rivers/Brees...). I agree that - using the Jets as an example - if Sanchez turns out to be better than Clemens, et al (with 8 career starts), he should start right off the bat and will benefit from a good team around him and the experience gained in the long run. On the flipside, if they happened to have a journeyman QB like Jeff Garcia, showing him the ropes, I wouldn't be surprised if sitting for a year and learning the system & the playbook helps a young QB over time.

It would be interesting to see the results if you ...

2009-05-19T13:41:00.000-04:00

It would be interesting to see the results if you compared players who started, say, 10+ games to those who started 0 games in their rookie year. This would (mostly) filter out rookie QBs who were benched after a few bad starts, presumably because they were performing poorly. With these benched rookie QBs in your dataset, it's not surprising that players with more rookie-year starts appear to have better careers: they performed better early in the season because they were simply better players (at least on average) to begin with!

I ran a few more versions. I factored in sacks--"A...

2009-05-18T21:54:00.000-04:00

I ran a few more versions. I factored in sacks--"Adj Net YPA above expected." The result is the same if not more pronounced. Rookie GS is positive and significant.

I also used Chases AV numbers in conjunction with the sack-added data, and the result is again positive and significant for rookie GS.

Lastly, I reran the regression using only 1st Round players. Same result with slightly less significance, which would be expected due to the smaller sample size.

In case anyone was curious, I reran the estimates ...

2009-05-18T17:49:00.000-04:00

In case anyone was curious, I reran the estimates using Chase's Approximate Value (AV) numbers in place of the overall draft pick number. The results are very similar.

Rookie Year Games Started is still positive and significant. The coefficient for Rookie GS drops a little from 0.05 to 0.04.

Brian, It'd be extremely difficult to quantify th...

2009-05-18T16:55:00.000-04:00

Brian,

It'd be extremely difficult to quantify the intangibles. I pose this question. Is it safe to assume that the fact that a team drafting a QB in the first two rounds indicates that the franchise is coming off of a bad year? No one drafts a QB to "take the best player remaining". Generally a quarterback taken on one day one is to fill a need. A team that needs a QB usually isn't in good shape.

I know this is a bit of a stretch. To support this, consider that of the 51 QB's taken in the first round, only 10 of them were taken in with one of the last 10 picks of the round - the picks owned by presumably strong teams (I know there are anomalies caused by trades and such).

Of the 22 2nd rd QB's taken, only five were among the last 10 of the round.

It would appear that the median number for avg YPA is around 4.8 or so. Of the 36 QB's below average, only eight of them were drafted with the last 10 picks of either round. Of those eight, none of them started more than five games their rookie year.

Looking at the number of games started the rookie year of all the players on the list, the median number is just less than 4. 13 of the 36 teams above that threshold were below the 4.8 adj YPA metric. Of those 13 'busts', seven were taken at the end of their respective round.

To clarify, of 15 late rounders, seven were busts that started four games or more.

If 4.8 is the threshold, than only one #1 overall pick failed to meet that standard. Of those 11 #1's, only three of them failed to start seven games or more. Interesting enough, of the eight players selected # 2 or 3, five of them were busts.

Of all of the busts, almost none of them were drafted by teams that had ever won a Super Bowl.

I think that's about the only conclusion you can come to. That bad teams draft badly. The performance of their rookie quarterbacks is nothing more than a reflection of the ineptitude of the team that selected him.

Mike- I largely have to agree with your take. Ther...

2009-05-18T13:31:00.000-04:00

Mike- I largely have to agree with your take. There are any number of factors that influence QB success. If we could model all of them realistically it would be quite the breakthrough.

However, the hope is that with a large enough data set, the factors not in the model (coaching, team talent, injury) will largely even out. Unfortunately, this data set is not particularly large.

But in the end, if you put a gun to my head, I'd bet that if there is some scarring effect of starting as a rookie, we would have seen it here.

Hi Brian, Your site is fast becoming one of my pe...

2009-05-18T13:14:00.000-04:00

Hi Brian,

Your site is fast becoming one of my personal favorites. I will add your site to my blogroll at TheSportsDick.com.

If you disregard draft position and look at what really matters - the adj YPA, four of the top six and 10 of the top 15 performers started four games or less during their rookie season. Conversely if you look at the worst performers, only three of the bottom 20 started more than four games. From that data alone you can conclude that starting a rookie slightly improves their likelihood of a successful career.

The problem with coming to any conclusing by analyzing this data is that it's largely irrelevant. The factors that determine a successful quarterback include the talent around the rookie going in, the quality of coaching he receives, whether he earns the job or inherits it due to injury, etc. How many of the top QB's would still be at the top of the list if they had been drafted by Detroit? I submit that not all of them would have enjoyed the same success as they did.

Full regression results: Dep var: Adj YPA const: 4...

2009-05-18T11:37:00.000-04:00

Full regression results:
Dep var: Adj YPA
const: 4.67
Rookie GS: coeff 0.051, p=0.03
Pick#: coeff -0.007, p=0.30
Model F-stat:3.64, p=0.03
R-squared: 0.15

Thanks, Chase. I should have been more clear on th...

2009-05-18T11:00:00.001-04:00

Thanks, Chase. I should have been more clear on the regression. The dependent variable was Adj YPA (not 'Adj YPA above expected'), and the predictors were draft order and rookie games started.

I agree with you on the non-linearity of potential, but strictly for Adj YPA, it does look linear, at least for the first 2 rounds. (See First graph above). Maybe Adj YPA doesn't really capture how truly 'good' a QB is. Or, the utility function of YPA isn't linear. In other words having a 5.0 YPA QB is more than 25% better than having a 4.0 YPA QB.

I don't think this proves anything, just that using this methodology we can't really detect any advantage to waiting. And even if the methodology doesn't capture every little nuance and bias, the true effect must be small enough to be engulfed by other effects, and is therefore probably not big enough to worry about.

If I have time today, I might rerun it with your AV numbers instead of raw draft order, and see how it turns out.

Good work, as usual, Brian. I actually started wo...

2009-05-18T10:07:00.000-04:00

Good work, as usual, Brian. I actually started working on this right after the draft for the same reasons you did, but trashed the project after seeing pretty similar results. I essentially got a small sign that starting early is better, but like you, I am hesitant to decide which way the causation arrow points.

One general question -- I assume you are not putting in actual draft order into your regression? Since draft potential, as measured by draft order, is not linear, you wouldn't want to do that. It's the exact reason I came up with this post (http://www.pro-football-reference.com/blog/?p=527). So whenever I'm doing a draft analysis like this, I replace actual draft order with the numbers in those posts.

It's a very difficult question to answer. Even accounting for draft position, coaches get to see a full training camp out of a guy. Those who aren't very good will have no chance of starting; those who look good will have a chance. So we already have some data (albeit one with lots of noise in it) on a QB besides his draft position by game 1 -- was he an opening day starter?

In the end, I suspect there's no career difference between a guy starting and not starting as a rookie. And certainly, we haven't seen any evidence to the contrary.

Looks like the effect is still mostly selection bi...

2009-05-18T08:20:00.000-04:00

Looks like the effect is still mostly selection bias. Since teams likely change their evaluation of rookies significantly between draft day and the middle of the regular season, accounting for draft order doesn't eliminate the bias. High draft picks that look great in preseason/practice are more likely to succeed and get more starts than high draft picks that struggle.