Comments on Advanced NFL Stats: Irrational Play Calling

I gotcha. Your theory is that teams with better ki...

2009-10-13T23:04:00.272-04:00

I gotcha. Your theory is that teams with better kickers will forgo going for it more often on 4th and short, when other teams typically would attempt the conversion. This would have the effect of increasing the FG:punt ratio at short to-go distances.

What if there is simply a negative correlation bet...

2009-10-12T21:42:39.400-04:00

What if there is simply a negative correlation between FG% and 4th-down-conversion rate? What if, for reasons of salary cap or other, teams with good kickers tend to have poor short games or offenses generally? What if punting ability varies relatively little, either inherently or because of punters' relative non-impact on others' salaries?

I would expect a low correlation between to-go dis...

2009-10-12T20:45:03.447-04:00

I would expect a low correlation between to-go distance and yd line, because the vast majority of your sample is 4+ yards to go (at shorter yardage, most teams run or pass, so they aren't in your sample). Can you tell us the average yd line for 4th-1, 4th-2, 4th-3, and 4+ yards to go?
I don't think yard line explains all of this, but some of it.

"we know there can't be a correlation between kicker ability and to-go distance."

Why not? If two teams both face 4-and-3 at the 33, doesn't the correct choice depend in part on the likelihood of making a FGA? So teams with good kickers should choose the FG more than teams with bad kickers. And the weak kicker team then drops out of your sample.

Brian, you keep going back to talking about all 4th down plays. But what matters is the subset you are looking at -- the kicks. That's totally different -- at 4th and 1, you're only looking at about 20% of the plays. You can only assume these smaller samples are all similar if they are independent from the values of the FG and the punt -- but they are not.

There doesn't need to be a correlation between...

2009-10-12T18:53:43.136-04:00

There doesn't need to be a correlation between kicker ability and to-go distance (or between to-go distance and weather or between to-go distance and anything besides the expected value of going for it). Let x be the average expected value of kicking a FG in the cases where a FG has a higher EV than a punt, and let y be the average expected value of a punt in the cases where a punt has a higher EV than a FG. All we need is for x to be greater than y, and we should get the pattern found in the data where going for it replaces punts more readily than it replaces FGs.

Vince-We know the conditions in the dome games are...

2009-10-12T18:15:23.012-04:00

Vince-We know the conditions in the dome games are equal. Plus, we know there can't be a correlation between kicker ability and to-go distance.

Ok. Ran a few graphs. No conclusions here-I'll...

2009-10-12T18:13:04.415-04:00

Ok. Ran a few graphs. No conclusions here-I'll just throw out what I found.

Only dome games: Same trend, not as pronounced. Punt:FG ratio goes from about 5:95 at 1 yd to go to about 15:85 at 10 yds to go.

Only 1st half, >3 min left--same result as all 4 qtrs.

There are small correlations between to-go dist and yd line. For dome-only games, it is 0.05 and for all venues it's 0.08. In other words, to go distances tend to very slightly increase with yd line within the 30-35 range.

Graphing all 3 options, it is clear that FGs and going for it "rob from each other." Their plots appear to be mirror images. What would that suggest the decision tree looks like?

Why dome games? By "good FG conditions"...

2009-10-12T17:54:20.679-04:00

Why dome games? By "good FG conditions" I just mean circumstances where the team has a relatively high probability of making the field goal, which depends on how good your kicker is and the line of scrimmage, as well as weather/stadium issues. I'd expect to see the same pattern in dome games, probably with a higher overall percentage of field goals.

Vince-I understand. That easy enough to check. I&#...

2009-10-12T17:36:18.243-04:00

Vince-I understand. That easy enough to check. I'll see whether there is the same tendency in dome games.

I thought my 2nd comment explained why this is rat...

2009-10-12T16:31:30.776-04:00

I thought my 2nd comment explained why this is rational: FGs (in good FG conditions) are more valuable than punts (in bad FG conditions where punting is a better option than a FG), so coaches should be more willing to go for it in bad FG conditions where going for it has a lower threshold to cross in order to be the best option. When going for it is an option (when the to-go distance is not too long), going for it should replace punts more readily than it replaces field goals, so a higher proportion of kicks should be FGs.

Or we can do it with numbers. Let's say that you have a completely rational coach who makes decisions by estimating the expected value of each option (Punt, FG, Go For It) and choosing the option with the highest EV (no restrictive decision tree for him). To simplify the numbers, we'll say that the expected value of a punt is always 0. The expected value of a FG varies a lot depending on conditions, but to keep it simple we'll say that 2/3 of the time (when conditions are good) it equals 1 and 1/3 of the time (when conditions are bad) it equals -1. I won't give numbers for the expected value of going for it, but what's important is that the EV varies with to-go distance (it's smaller for longer distances) and it also varies for other reasons like the quality of the offense & defense (so at each distance you can think of it as having a distribution rather than equaling a single number).

So what would we see for this coach if we graphed kick type by distance to go? Starting at the far right of the graph, on 4th and very long, Go For It will always have negative EV so the coach will never go for it: 2/3 of the time he'll choose FG and 1/3 he'll choose Punt. As to-go distance gets shorter, he'll start to run into some cases where the EV of going for it is greater than 0 (but still less than 1) so Go For It will sometimes replace Punt as the chosen option when FG conditions are bad, but Field Goal will always be the option when FG conditions are good. That means that FGs will be more than 2/3 of the kicks, since they're 2/3 of all plays and punts are less than 1/3 of all plays. As to-go distance gets shorter, the EV of going for it increases and is greater than 0 more and more often and occasionally even greater than 1, so Go For It replaces more and more punts and even starts to replace some FGs. With many fewer punts and a few less FGs, FGs will be an even larger proportion of kicks (even though they're less than 2/3 of all plays). At some point, as to-go distance gets even closer, Go For It might start replacing FG as quickly as it replaces Punt, so the proportion of kicks that are FGs could stabilize. But as long as teams are sometimes kicking, FG should be more than 2/3 of the kicks because the EV of Go For It will be less than 1 more often than it is less than 0.

In other words, the graph for the rational coach would look a lot like the graph for actual coaches. We know from other data that coaches aren't perfectly rational (they don't go for it often enough), but I don't see any reason to think that they're being irrational about choosing between FGs and Punts because this interaction with to-go distance is exactly what you'd expect if they were rational.

Brian: you mention above that field position is ...

2009-10-12T16:19:02.907-04:00

Brian: you mention above that field position is identical (32.6) for all yards-to-go. And that makes sense for all 4th down plays. But it can't be true for your sample of kick plays (I don't think), since teams go for it much less frequently from the 30 than from the 34 or 35. Your sample (punt or kick) should have an average field position well below 32.6, probably 32 or less. So I'm thinking you perhaps looked at all 4th downs, not just the kick plays?

If so, you should check to see if distance to goal is in fact shorter at 4th-and-short than it is at 4th-and-long. It seems possible that length of prospective FG plays a bigger role in determining whether the team goes for it when yardage to first is short. For example, a team might choose FGA at 4th-and-2 from the 30, go for it at 4th-and 2 from 35, and yet choose to kick at either distance on 4th-and-8. That would result in shorter FG distances for your 4th-and-short plays.

Interesting stuff Brian. I believe you do understa...

2009-10-12T15:01:22.289-04:00

Interesting stuff Brian. I believe you do understand me correctly. I understand the point you are making with the alternate decision tree and why that would lead to the opposite effect which we do not observe. I completely agree that coaches should weigh all three options at once. Easier said than done.

Also if I had to guess, I would say that the real coaching decision tree is probably a little more complicated than I said at first. On fourth and short situations, the coach weighs the FG vs going for it (with punting a distant third option). On fourth and long, the coach weighs the FG vs punting (with going for it a distant third option). This seems reasonable. The FG is always a viable option while punting slowly replaces going for it as a second viable option.

And I do think that the percentage of field goal attempts is not quite steady but it probably goes down a bit as you get down to 4th and 2 and 4th and 1. It is replaced not by punting but by going for the first down. It does not make sense, nor do I believe that it is the case, that the percentage of FG attempts drops as the to-go distance increases. It's just that punting increases.

Here's what I think an interesting next step would be. Do coaches have a chart for this? Could you make a chart? What does WPA say? It would be cool to have a little chart of opponent's yard line vs to-go distance, in this 30-40 yard area, with the three options presented in order of value.

Another wrinkle would be to suppose that we had some extra information about the field goal kicking game. If we think our kicker can hit 75% of the time from 49 yards we will play it a lot differently than if he can only hit 30%. Ultimately, if you aggregated all of the situations, playing rationally and optimally at each step of the way, I think you would observe results that roughly mimic the effect seen in graph at the top.

I think Dave and Vince have demonstrated quite wel...

2009-10-12T14:15:41.570-04:00

I think Dave and Vince have demonstrated quite well how a coach's mind works. These explanations do not follow a logical decision making process, and that is probably exactly what the NFL coaches are doing. I still think there is some room for selection bias. First, the dumber coach's are less likely go for it on 4th and short, and are therefore more likely to make irrational decisions, so maybe this pattern doesn't hold across the board.

I also wonder how much taking out 3 minutes helps the problem of teams needing FGs being able to rush into 4th and short position. For example, if the game is expected to be low scoring, say 3-6, a team might just try to force there way into fg range from the start. Perhaps in low scoring games there are very few 4th and longs from in this range.

Dave- I think I understand what you're saying....

2009-10-12T13:21:10.049-04:00

Dave- I think I understand what you're saying. According to your model, a coach's decision tree would have to be:

1. FG or other?
2. If other, go for it or punt?

Going for it and punts would be in the same family of "non-field goals." But how is that still rational? (genuine question, not being argumentative)

Your model would also suggest that coaches are ignoring to-go distance in their initial decision. A FG rate would be fixed at 60% (or whatever) regardless of to-go distance.

What about an alternate decision tree: What if going for it and FGs are in the same family of "needing points?" And if coaches need points, they should first choose between trying for points and punting. If they don't punt, they then decide whether to kick or go for it.

But that would make the ratio of FGs:punts lower at shorter to-go distances, the opposite of what we see.

Ultimately the only truly rational method is to evaluate the value of all three options in a single stage and choose the one with the highest value.

Instead, there seems to be a single-elimination tournament going on in coaches' heads, as if they can't weigh more than two options simultaneously. They only consider 2 options at once, and eliminate one at each stage yielding the choice. And if I understand Dave right, it suggests coaches are ignoring to-go distance at the first stage.

(This all assumes no large systematic connection between to go distances and the values of kicks.)

No no no! Field goal values have NOTHING to do wi...

2009-10-12T12:19:39.359-04:00

No no no! Field goal values have NOTHING to do with to-go distances! By removing "going for it" you have created an illusion.

Since we are missing the "going for it" aspect of the graph, I'll make up some fake data. NOTE: if you were to take out the "Going for it" part of the data and only show the field goal vs punt, this data would match the actual graph above (roughly).

F = Field Goal, G = Going for it, P = Punt, (F v P)= implied percentage of field goals vs punts, ignoring going for it, as in the graph above. Note that this last piece is a RESULT of the first three.

4th and 1: F= 60%, G= 30%, P= 10%, (F v P = 86% v 14%)
4th and 2: F= 60%, G= 25%, P= 15%, (F v P = 80% v 20%)
4th and 3: F= 60%, G= 20%, P= 20%, (F v P = 75% v 25%)
...
4th and 10: F= 60%, G= 5%, P= 35%, (F v P = 63% v 37%)

Even though the field goal rates have stayed exactly the same (at 60%), the increased number of punts has created the illusion that coaches are less likely to kick the field goal as the to-go distance increases when in fact this is not the case.

"Yes, FG values are subject to lots of things...

2009-10-12T12:03:29.990-04:00

"Yes, FG values are subject to lots of things. But those factors would need to be shown to be systematically connected to to-go distances enough to explain what we observe."

That's not quite right. The concern being raised is not that FGs are somehow more valuable in short go-to distances. The concern is that, in short go-to distances, the value of the FG has a larger effect on whether the team goes for it, and therefore whether the play makes it into your sample. This will be much less true on long go-to plays, simply because teams so rarely go for it. In other words, the relative value of the FGA has more impact on the go/no-go decision with short yardage than with long yardage.

I think you can largely settle this if you can report the expected WP for FGAs and punts in the 1- and 2-yard situations (when teams don't go for it), vs. the 7, 8, 9-yard situations (again, when teams don't go for it). My guess is you'll find that the WP ratio of FGA:punt is higher in the short yardage situations. There are still other possible explanations -- kicker's ability, different pools of coaches -- but the value of the FGA seems like the most plausible (other than your theory of irrationality).

I agree. Yes, FG values are subject to lots of thi...

2009-10-12T11:39:18.496-04:00

I agree. Yes, FG values are subject to lots of things. But those factors would need to be shown to be systematically connected to to-go distances enough to explain what we observe.

I am still not convinced that the behavior depicte...

2009-10-12T11:30:05.113-04:00

I am still not convinced that the behavior depicted here is irrational. Here is hopefully a more convincing explanation of what appears to be irrational behavior.

The expected value of a field goal is unseen here, and I believe it is highly unpredictable and prone to fluctuation. A number of factors- kicker, playing surface, and weather- all play a huge role in determining the expected value of a kick. From an arbitrary distance of 49 yards, there will be times when this field goal has a very high chance of success, and there will be times when it has a very low chance of success.

During optimal kicking conditions, the expected value of a field goal will be hands down the best option of the three, regardless of distance for a first down. During dreadful conditions, kicking a field goal will be hands down the worst option regardless of distance for a first down. I'm ignoring the gray areas here for simplicity but hopefully my point is clear.

When a field goal is the best option, obviously you kick it. When a field goal is more of a dicy proposition, you have a choice to make. Punt or go for it? Obviously this part of the equation is highly dependent on distance to go for a first down, which explains the nature of the graph.

I know this argument is a slight simplification. But if you agree that the value of a field goal is highly dependent on external factors more so than the other two options, which leads to the field goal *often* being either the best option or the worst option regardless of distance, then the behavior depicted by the graph is rational.

I think there are a lot of good possibilities here...

2009-10-12T08:43:16.297-04:00

I think there are a lot of good possibilities here, particularly the one Dave points out. However, that does not make it rational. Just because it can be explained does not mean it's logical.

There are 3 options. Rationally, you should pick the one that gives you the best opportunity to win, period.

The solution has been stated by a couple commenter...

2009-10-12T02:26:13.942-04:00

The solution has been stated by a couple commenters and I am hoping their wisdom does not go ignored.

I am quite sure that Vince and one of the anonymous commenters are absolutely right and the premise of the article is incorrect. The first decision the coach makes is whether or not to kick the field goal. Are we in field goal range, yes or no? After that, the decision of whether to punt or go for it is determined by the distance, which explains what appears to be odd behavior depicted by the graph.

What would really settle this debate is if the graph showed the third option- going for the first down. By eliminating this element you have created the illusion that coaches are less likely to kick field goals as the distance goes up, when in fact they are just more likely to punt (vs going for it). I believe that if you showed all three options, you would see the percentage of field goals would stay roughly steady, while punting increases with distance and going for it dwindles.

I'm not sure how difficult it would be to go back and incorporate all of the 4th down conversion attempts for the third element but I would be fascinated to see the results.

Here is a possible reason. In the case of a broke...

2009-10-10T14:40:55.436-04:00

Here is a possible reason.

In the case of a broken play, it may be that a punt formation has a better chance of picking up a first down, than the FG. I.e. a punt formation, with it's blocking, can allow the kicker to run (or pass) for a first down if the snap is bad and he can't kick it.

If a FG goes bad, it is almost impossible to run for a first down, cause it starts with your qb kneeling on the ground, and the blocking is different.

I notice that the function flattens out after five...

2009-10-10T13:06:24.004-04:00

I notice that the function flattens out after five yards. Beyond five yards the probability of a five yard penalty is irrelevant. If coaches had evidence that a field goal attempt had an increased probability of resulting in a five yard penalty, then we could expect to see a step change at five yards to go.

That notwithstanding, every NFL coach should have a statistician on their staff. Compared to what coaches get paid, we're not that expensive, and we could probably provide a lot of insight.

I'm curious about the statistics surrounding final outcome (including the next possession) of fourth and goal from inside the five yard line.

Often times even if the 4th down isn't converted the opposing team does nothing with their next position, so field position is retained. 3/7 is 43%, but with the retained field position an even lower expected conversion rate may justify going for it on fourth down.

I'm a fan of the go-for-it/quick kick option p...

2009-10-10T06:34:50.466-04:00

I'm a fan of the go-for-it/quick kick option plays, that always seem to work well.

So, a team gets it's QB practised on chipping a punt to inside the 10 yard line, that way when they get to 4th and short in no man's land they can keep the offense on the field. If the defense drops a play back to feld the punt, the offense goes for it (after all, it's 11 v 10 and the odds were in their favour when it was 11 v 11) - if the defense doesn't drop someone back the QB quick kicks - with the safety of being able to allow the ball to roll as noone is covering the kick.

Pittsburgh used to do this well with Big Ben and I don't see why other QBs can't be taught to make what is a very basic kick.

The premise of the article is that a coaches first...

2009-10-09T15:11:54.724-04:00

The premise of the article is that a coaches first instinct is whether or not to go for the first down. I believe this assumption is flawed and that the coaches first decision is whether or not to kick the FG. The other two options (Punt/Go) would be weighed 2nd and 3rd based on their probability of success. Basically in short yardage, FG = option 1, Go = option 2, and punt option 3. Since "Go" carries such a lower probability in long yardage, it falls behind "punt" into the option 3 spot.

Couldn't there be an expected points explanati...

2009-10-09T10:31:13.974-04:00

Couldn't there be an expected points explanation for this behavior? Referring to your past post on fourth downs, I saw that the expected points of a field goal from the 30 is around 1. The expected points for a punt from the 30 is around 0. The expected points for a conversion from the 30 depends, of course, on the distance needed. For a short conversion, the EP is between 1.5 and 2, while for a long conversion (7+ yards) it dips below 1. So might not a coach intuit that by punting on 4th and 1 at the 30, he is essentially trading 2 points for 0, but for 4th and 10 from the 30, he's trading 1 point for 0. In both cases, he's being irrational, but it could be that his appetite for risk (coaches being very conservative) makes a 1 point expected loss acceptable, but a 2 point expected loss just too great to stomach.

I agree mainly with Vince and Guy. I think its a ...

2009-10-09T02:29:29.944-04:00

I agree mainly with Vince and Guy. I think its a selection bias issue. Since some coaches go for it more often than others do, you are removing a definitively non-random subset by looking only at 4th and shorts where they didn't go. I also think the perceived strength of the kicker is a major issue.

With 4th and short, coaches go for it if they think their kicker can't make it. Kick if he can. So you have 50% goes and 50% kicks, but 100% of the non-goes are kicks. With 4th and long, you punt if you think your kicker can't make it. So you have 50% punts and 50% kicks. 50% of non-goes are kicks. This would be perfectly rational if the coach was accurate in his projection of the kicker's ability.

** Obviously, I know that its not 50/50 and the real numbers are more complicated. The above numbers are just to prove a point.