Who 'Dat Gonna Kick Onside to Start the Second Half?

Congratulations to the Saints and their fans for a great championship season. Head coach Sean Payton made three bold decisions on the way to New Orleans' first Super Bowl trophy. Toward the end of the 1st half, he called for run on 4th and goal from the 1. Payton decided to try on onside kick to open the second half, perhaps the game's most aggressive call. And after scoring the go-ahead touchdown in the 4th quarter, he called for the 2-point conversion. Let's take a look at all three of those decisions. Daring, yes; but did he have the numbers on his side?

The 4th and 1

According to the expected points model, it's clear that a team should go for it on 4th down and goal from anywhere inside the 6. But with time running out in the 2nd quarter, it might be better to turn to Win Probability (WP). With just under 2 minutes to go and a 4th and goal on the 1, the WP analysis agrees with the go-for-it call. FGs are successful 99% of the time, and overall, a decision to kick makes the score 10-6, giving the Saints a 0.32 WP. A successful TD ties the game, giving the Saints a 0.48 WP. A failed attempt gives the Colts a 1st down at their own 1 with 1:55 to play in the half, giving the Saints a 0.26 WP. It was certainly a high-stakes play.

On the goal line, 4th and 1s are converted 68% of the time. This makes the overall decision to go for it worth:

0.68 * 0.48 + (1-0.68) * 0.26 = 0.41 WP

Going for the TD was clearly the better call, 0.41 vs. 0.32 WP. If you don't buy the 68% success rate, the break-even rate, where going for it yields an equal WP as the FG, would be 42%. In other words, as long as Sean Payton believed his offense had a better than a 42% chance of getting the TD, he should go for it.

In fact, this is a great example of why going for it is often worth more than many people, including most NFL coaches, believe. Pinning the Colts at their own 1 is not such a bad thing. The Saints were able to get the stop, and get the FG before halftime anyway. There's no guarantee that would happen, but combined with the possibility of a TD, it tilts the scales in favor of being aggressive.

The Onside Kick

Had the Saints not recovered the onside kick to open the 2nd half, fans and 'analysts' would be second-guessing Payton for years. Onside kicks are surprisingly successful when they are not expected. Since 2000, slightly over 60% of unexpected onside kicks have been recovered by the kicking team. An analysis based on Expected Points suggests teams should occasionally attempt surprise onside kicks if they believe their chances of recovery exceed 42%. Let's also take a look at what WP would say.

In this case, the Saints were down by 4 points. A deep kick would typically give the Colts a 1st and 10 near their own 30, worth 0.32 WP to the Saints. A failed onside kick gives the Colts a 1st down at the Saints' 40 or so, worth 0.26 WP to the Saints. A successful recovery gives the Saints possession at their own 40, worth 0.39 WP. In total, the onside attempt is worth:

0.60 * 0.39 + (1-0.60) * 0.27 = 0.34 WP

The onside attempt was a good gamble according to the numbers, but not by much--0.34 vs 0.32 WP. it paid off, and the Saints capitalized with a TD drive to take the lead for the first time in the game.

The Two-Point Conversion

After scoring a go-ahead TD with 5:46 in the 4th quarter, Payton faced another decision. The score was 22-17, and an extra point would only make a 6-point lead, leaving the Saints vulnerable to losing with a Colts TD. A 2-point conversion would make it a 7-point lead, and a Colts' TD would likely only tie.

The general rule of thumb for 2-point conversions is to only attempt them when there is time for no more than 3 total possessions left in the game. In this case there was time for only 2 possessions at most, so it was probably an easy call for Payton. Let's check the numbers anyway.

Extra points are nearly automatic these days, succeeding 99% of the time. Kicking would be worth 0.78 WP for the Saints. A successful 2-point conversion would give the Saints a 0.88 WP, and a failed conversion attempt would give the Saints a 0.77 WP. League-wide, 2-point conversions are successful just under 50% of the time, but that number is polluted from several botched snaps that turned into 2-point attempts. For now, let's just say it's a 50% success rate. The 2-point attempt was worth:

0.50 * 0.88 + (1-0.50) * 0.77 = 0.83 WP

So the 2-point attempt was probably a good call, 0.83 vs 0.78 WP. But if the 50% rate doesn't seem realistic, the break-even rate would only need to be 9% for the attempt to make sense. There was almost nothing to lose.

The Colts successfully converted a 4th down of their own, which was also a good call, but the night belonged to the Saints. Another great NFL season, and another great Super Bowl.

  • Spread The Love
  • Digg This Post
  • Tweet This Post
  • Stumble This Post
  • Submit This Post To Delicious
  • Submit This Post To Reddit
  • Submit This Post To Mixx

30 Responses to “Who 'Dat Gonna Kick Onside to Start the Second Half?”

  1. Anonymous says:

    Not surprisingly, I was spouting out what I've read on all three of those calls at the part I was at.

  2. Anonymous says:

    party*

  3. James Holzhauer says:

    I'm a big fan of these analyses, but it's hard for me to take this seriously when you have a failed conversion attempt as a higher WP than kicking the extra point

  4. Brian Burke says:

    Thanks, James. I swapped them in the write-up. Lately, I've been trying to post these very quickly. It should be 0.77 for the failed attempt, and 0.78 for the XP.

  5. Anonymous says:

    2-point conversions had a 45% success rate this season, even after removing the botched snap attempts. See the following NY Times article.

    http://www.nytimes.com/2010/02/02/sports/football/02fast.html

  6. Anonymous says:

    When you consider the offensive nature of the two teams the value of the onside kick probably goes up. I assume these are league averages.

  7. Brian Burke says:

    Right. Like you said, this season. 53 attempts is tiny. We need a much bigger data set than that. The article even says the rate has been higher over a larger time-span.

  8. Fred says:

    I also quoted the surprise onside kick statistic while watching the game, everyone seemed pretty impressed!

  9. Fred says:

    Also, I know this is a bit off topic. But was I the only one hoping for OT? I think it is going to take an Super Bowl OT fiasco to change the rule. At the very least, the rule needs to be changed in the playoffs.

  10. Ian says:

    Brian - just on the technical side of things to add to your point about the small sample size (and also to make sure I'm remembering my statistics lessons from school), it's a bit of central limit theorem followed by a Z-test that one would use to show that a 45% conversion rate on 53 attempts is well within the acceptable limits of a true probability of 0.50, right?

  11. Anonymous says:

    "teams should occasionally attempt surprise onside kicks if they believe their chances of recovery exceed 42%"

    The great part about the Super Bowl is that we actually get to hear exactly what they thought: Payton said he believed it had 60-70 percent chance of working.

  12. Jeremy Johnson says:

    Using a 95% confidence interval (CI) and assuming 53 attempts the range of possible real mean values for success in the conversion is between 32% and 58%.

    If there were 250 attempts in the sample the 95% CI would be 39% to 51%.

  13. James says:

    Payton is quoted as saying he felt they had a 60 to 70% chance to recover the onside kick. That sounds familiar...

  14. Andy says:

    Brian, can you analyze the Colts FG attempt from 51 yards? It was a tough spot. Third and long from the area where a punt does no good. I bet generally a FG attempt would be the right call, but Stover is bad from long distances. I think the real mistake was on third down. If they had contemplated using all 4 downs they could have run a short play on third to set up a conversion or a shorter field goal.

  15. Andy says:

    Agree that the Colts erred by not running on 3rd and long.

  16. Brian Burke says:

    It's a little more complicated than a standard t-test measuring sample error. It's not like a weighted coin that's always 45% or 50% or whatever. Every 2-pt conversion is unique, so you have some situations that are 55%, some that are 40%, some that are 50%. Any individual season might feature more or fewer of each type of situation. So there are 2 random functions working on top of each other, and the variance is going to be wider than a standard statistical test would tell us.

  17. Jeremy Johnson says:

    I'm not a football expert, I know very very little actually. In which ways are each 2-point conversion unique?

  18. Kulko says:

    The OFF and DEF facing each other in each specific Attempt have varying level of skill. that could shift the true mean of any attempt like 20% in one direction or the other.

  19. Anonymous says:

    Shouldn't botched snaps be counted towards the 1-pt attempt conversion rate? Are they so rare that they do not significantly affect the 99% success rate?
    I could imagine that a small number of botched snaps could affect the 2-pt rate significantly without affecting the 1-pt rate because there are so many more 1-pt tries. What do the numbers say?

  20. Brian Burke says:

    I was thinking of exactly what Kulko said.

    In any case, I think the better direction to approach it is to start with the break-even success rate. In other words, what success rate would make the 2-point attempt (or onside kick, or 4th down attempt) worth it? Then the coach can estimate what he believes his own team's chances are, and whether it exceeds the break-even.

    The league-wide baseline success rate is still helpful in calibrating the coach's own estimate of his team's chances. He can take the baseline and bump it up or down depending on situational considerations, such as the relative strength of the lines, weather, or whatever else might come into play.

    Too often I hear coaches and analysts start with the mitigating considerations without knowing the baseline. Essentially they are just seeking a rationalization for an intuitive decision. We all do that everyday.

    I think real coaches are more likely to accept that 'break-even' analysis instead of just saying 'my computer says go for it.' They stay in the decision process and it basically just informs them without replacing their judgment.

  21. Jeremy Johnson says:

    It would be interesting to break down the 2-pt data. Take all the teams and put them into four buckets: 1) good offense, 2) bad offense, 3) good defense and 4) bad defense. Define good as the top 1/3rd in the league and bad as the bottom 1/3rd. Use hindsight to determine good and bad by using the entire season's rankings. This would give you four categories of matchups, for example good offense vs bad defense. You would be left with four percentages for 2-pt convertibility. This procedure can be done with the last 10 years worth of data. Hopefully you can get 100+ population in each sample. This analysis could be taken further to make the results more robust.

  22. Brian Burke says:

    I meant botched snaps on kick attempts. They often appear in the stats as 2-pt attempts because the holder may try to run or throw after an aborted kick. The Romo bobble vs. Seattle is a good example.

  23. Anonymous says:

    Brian, are you going to do an analysis on the 51 yard FG the Colts attempted? I don't think that is a high percentage play with Matt Stover.

  24. Dave says:

    What about the 51 yard field goal attempt? Any way to estimate Stover's chances of making it? There was a lot of second guessing on that one even though it was a 4th and 11...

  25. Dave says:

    Whoops looks like Anonymous stole my thunder...

  26. Brian Burke says:

    This may shed some light. It's a league-wide baseline, but it's pretty clear punting was a bad option.

    The argument is really between going for it and kicking the FG. In Baltimore, Stover would only be asked to kick from that distance in desperation. He must have believed he had a 50/50 shot or a veteran like him wouldn't have gone out there. The NFL average is 60% from 51 yards out, but that includes all types of conditions. Winds were very light last night in Miami--4 mph, and the footing was solid. So kicking the FG was a defensible decision.

    On the other hand, when there are two very strong offenses out there, POSSESSION IS EVERYTHING. This makes the go-for-it option much more lucrative. The league average on 4th and 11 is about 30%, but with Manning, Wayne, Clark, etc, it's certainly a little higher.

    In the final analysis, assuming Stover is 50/50 from 51 yards, the break-even success rate required to make the 4th down attempt worthwhile would need to be 75%.

    The WP model agrees . It shows that the conversion attempt would be a -0.02 WPA mistake.

    The Colts were literally in a no-win situation. The problem was that it was 4th and *11*. That screen to Collie on the previous play lost 3 yards. And the 1st down run by Addai only gained 2. Had it been a closer to 4th and 7, I'd say go for it.

  27. Jay Paradise says:

    Hi Brian,

    Quick question regarding EP values on 4th down. In your 12/16 post on expected values, with the values shown in a spreadsheet, it says expected points on 4th down on the 1 yard line is only 2.29. Yet in the image that you have attached in the above comment shows much higher values on 4th down.

    I'm just wondering if you have updated 4th down values based on yards to go or if there is an equation we can use to estimate 4th down values based on 3rd down values?

    Thanks Brian

  28. Brian Burke says:

    Jay-One shows the EP value of going for it from the 1 yl. The other shows the EP value of what coaches tend to do (kick the FG).

  29. Jay Paradise says:

    So if I wanted to calculate expected points added for a game or season, I should use the values contained in the spreadsheet, corret?

  30. Brian Burke says:

    Correct.

Leave a Reply