Belichick 4th Down Follow-Up

Thanks to the commenters on the original post, all 200+ of you, for all the great criticisms and suggestions. The Internet is a big place, and although not every comment was helpful, I was pleased to see such a thoughtful and constructive debate. In this follow-up, I'll try to address some of the most common criticisms and questions. I suspect some skeptics will never be fully satisfied, but I feel many of the comments deserve a response.

1. You used 38 yds for the expected punt distance, the average punt distance for that region of the field. But Patriots punter Chris Hansen had averaged 4 punts for 44 yds. Why not use that distance instead?

Four punts is a tiny, tiny sample of Hansen's and the Patriots' true expected net punt distance. Punters in NFL are generally all the same. Differences among punters are almost completely due to two factors: 1) where on the field they tend to kick from, and 2) luck. We're far better off using the league average, based on hundreds of punts, rather than a tiny sample of 4.

However, indoor punts may indeed tend to be longer. Even if we use the 44 yd punt distance, it doesn't change the final analysis.  The likelihood of an offense scoring from their own 28 instead of their own 34 is 0.27 instead of 0.30. This changes the ultimate comparison to: 0.79 for the conversion attempt vs. 0.73 for the punt. Those 6 yards would be gobbled up by Manning's offense in a heartbeat.

2. Your estimate of the Patriots' 60% chance of converting that 4th and 2 is too high, and it doesn't take into account the particular details of the situation.

It's true that this figure does not take into account the stress of the situation, the type of turf, the time of day, or the pull of the gravity of the moon. The 60% figure is a league-wide average for 4th and 2 situations outside the red zone, which is a reliably consistent stat across the recent decade.

Situational variables are usually grossly over-weighted. As far as I know, 2 yards are always 2 yards. The field is still level. The rules are the same. There are still 11 guys on offense and 11 on defense. Every single play in the NFL, including every 4th down in my data, is stressful and takes place in front of thousands, if not millions, of fans. The players are professionals accustomed to high-stakes risks, so let's not over-think this. The true answer might be 62% or 57%, but 60% is a reliable and fair baseline estimate.

Additionally, differences in strength and ability between any two NFL teams on any single play are usually very, very small. It's only when we tally the results of dozens upon dozens of plays in a game that the differences between teams accumulates to a point when we can tell the better teams from the mediocre ones. In any case, in terms of general efficiency, the Patriots offense is ranked 5th, and the Colts defense is ranked 4th.

3. Not punting was an insult to the Patriots' defense. Belichick offended his players by not showing confidence in them.

Agreed. If your goal is not not offend, then punting is the right call. But if your goal is to win the game, going for it is the thing to do.

4. The 30% chance of the Colts scoring a TD following a punt is too low. Same for the 53% chance for scoring from the Patriots' 28 following the failed conversion attempt. The Colts offense appeared unstoppable in the 4th quarter.

I agree. Their chances of scoring a TD from either field position was likely higher than the league average. Should we throw out the entire analysis? No, as I said in the original article, we should look at what happens to the numbers when we start increasing the Colts chances of scoring. And as we do, the go-for-it option becomes more and more preferable. Remember, as you increase the scoring chances from one field position, you have to increase the chances from the other. Manning isn't amazing from the 30 yds out, but average from 70 yds out, and he had plenty of time from either field position.

The better you assess the Colts' chances of scoring, the better the go-for-it option becomes relative to the punt option. Why? It's because more than half of the mathematical weight of the go-for-it option ignores how good the Colts offense is. If the conversion is successful, it doesn't matter how good the Colts are.

5. The 53% and 30% chances of the Colts getting a TD don't take into account the time and score. 

Yes, they do.

6. There are so many unquantifiable considerations that affected the unique situation at hand that you cannot possibly put them into a mathematical equation to know the precise probabilities.

True. There are any number of variables that might have an effect. So the conclusion is therefore...punt? That makes no sense. Why?

What I've tried to do is lay out the major, dominant considerations--down, distance, field position, time, and score. Other factors certainly come into play, but at some point we need to make our best estimate based on reliable facts. As Tony Dungy said after the game, "you have to play the percentages." Well, we all agree on that, and further, that statement assumes there are percentages to be known. The question is, what are the percentages? It turns out they're not what Dungy and most other critics think they are.

We can never know with perfect certainty that the Patriots' chance of winning was truly 79% if they went for it or 70% if they punted. The best we can do is estimate the center-points of two bands of uncertainty. And although that leaves a lot of room for error, it's far better than simply assuming the punt is better because that's what Rockne and Lombardi did.

7. Stats and numbers are just cold, stubborn, heartless things that don't consider how people feel or how foolish we often are.

Yes...yes, they are.

  • Spread The Love
  • Digg This Post
  • Tweet This Post
  • Stumble This Post
  • Submit This Post To Delicious
  • Submit This Post To Reddit
  • Submit This Post To Mixx

118 Responses to “Belichick 4th Down Follow-Up”

  1. Matt M says:

    Excellent responses, all. One question I had, but ran out of patience reading through all the comments on the other page: what is the WPA if the Pats make the 1st down there? You seem to assume that it's 100%, but since they can't run out the clock with kneeldowns it has to be a bit less than 100%. My best guess is that if they don't make the first, they're punting with 20-30 seconds left.

    I still agree with your overall conclusion, but it seems like it might be marginally closer based on the possibility of a turnover or punt after gaining the first down.

  2. Anonymous says:

    Excellent work as usual. I think there are just so many people, many of whom work for ESPN, who feel that admitting going for it was correct violates the sanctity of conventional NFL wisdom that they hold so dear. They are afraid to let go of their erroneous assumptions about the game.

  3. Harry says:

    Brian, do you think that after this season -- in the true 'long run' -- Belichick's highly publicized decision to go-for-it and the actual result of the play will end up being advantageous to him?

    Basically, if they had converted then the mainstream media would be calling him a genius and lots of other coaches would start going for it more often on 4th and short (even in their own territory) when it's a +WP decision. This would decrease the advantage that Belichick enjoys by being the only team to employ this strategy while others take the lower WP road.

    Since the conversion failed and the media used its results-based analysis to deem this decision "arrogant" (Wilbon) and a "brain fart" (CBS), coaches may not only think it's the wrong decision but may also simply avoid going-for-it in order to avoid the public fervor. Also, Belichick has the best job-security in the NFL (or maybe the only). So a coach with no job-security may not try something that the public/owner will go crazy over even if it gives his team the best chance to win.

    All in all, I think the RESULT of the play will deter other teams from employing highest WP strategies but Belichick will continue to use them.

  4. Matt M says:

    @Harry,

    Totally agree with your assessment of the media coverage. It's a good call if it works, but bad if it doesn't. I think it's generally assumed that only a coach with job security like Belichick could possibly go ahead and make a call like this.

    What really amazed me about the media coverage was that some of the people locally who trashed Belichick were singing the praise of Kevin Kelley (the HS coach who never punts) just a month or so ago.

    Personally, I hate the Patriots; I live in New England and one of their Super Bowls came at the expense of my favorite team. But man, was I hoping that they'd convert that play; now no one else will have the courage to try something like that for years.

  5. Anonymous says:

    Awesome. I think the problem a lot of people have with the decision (but are unable to verbalize) is this:

    Even though going for it was the correct choice from a probability standpoint, it placed enormous weight on one play. Prior to that play, win probability was 79%. After the play was run, it would be either ~100% or ~40%. By contrast, had the Patriots punted, each play by Colts after the punt would only slightly change the win probability for each team.

    An analogy would be investing your money in a stock with a high average return but high volatility, rather than diversifying into several stocks which would give you a lower average return.

  6. Tarr says:

    While I naturally agree with you on 99% of this, allow me to make two minor quibbles:

    - On the percentages of a 4th and 2: I think that looking at red zone numbers may be meaningful here. Due to the game situation, a 2 yard conversion was almost exactly the same as a 70 yard touchdown. In a normal 4th down attempt in the second quarter, this is not the case. This may have allowed the Colts defense to play a more high risk, high reward strategy. And indeed, the Colts came out with an uncharacteristic blitz on the play. So, this would suggest a slightly lower expected conversion rate.

    On the other hand, I think looking at Patriot short yardage conversion numbers over several years, and Colts defensive conversion numbers, is a meaningful way to increase the robustness of the estimate. I say "on the other hand" because these numbers strongly suggest that the Patriots were much MORE likely to convert than average. They've been very effective in short yardage over the years.

    - Purely from an emotional perspective, I think the "disprespect the defense" argument is BS. First, Belichick goes for it on 4th down all the time; his players have adjusted to this. Secondly, why isn't NOT going for it an insult to the offense? Doesn't that make at least as much sense? Finally, the defense still has a chance to win the game for you - they just have to successfully defend a short field. I remember Cowher said he was comfortable attempting a surprise onside kick in super bowl XXX precisely because he thought his defense could defend the short field.

  7. mcnallydp says:

    I think the analogy that compares the 4th and 2 decision to investing money agressively or conservatively in the stock market is way off. The fact of the matter is, at the end of the game one team was going to win and one team was going to lose. There's no such thing as having 70% of a win, the variance is inevitable. Heck, the punt could have even been blocked, which would give the Pats a lot less than a 70% chance of winning.

    ESPN analysts called Belichick's decision to go for it on 4th and 2 "gambling." By that definition, every play is gambling. There are no sure things in football, and you can't avoid taking a risk. The only thing you can do is give your team the best chance to win the game, and that is what Belichick did by going for it on 4th and 2.

  8. Anonymous says:

    People who believe statistics can't apply to real-flesh-and-blood-and-stress-and-a-million-little-unique-circumstance situations should request that their doctors not use any double-blind-tested medical techniques or drugs in their treatment, since, of course, all those tests are just statistics and don't apply to real clinical situations.

  9. Brian Burke says:

    Good question on the Colts' chances of a miracle if the Patriots were successful on 4th down. You're right that it's not absolute zero. There's always the Stanford Band.

    With 120 sec remaining and 1 timeout, and assuming 3 kneel-downs and then a punt, there would be 20-25 sec remaining.

    There were 38 examples of teams needing a TD that got the ball back from anywhere between the opponent's 35 and their own 30. Only 1 team out of those 38 successfully scored and won, and they started on the opponent's 47. (It was CHI vs CLE in 2001.)

    Once teams are within the opponent's 30 yd line, the odds suddenly improve to where about 1 in 3 teams are able to score the TD. It looks like you need at least 20 secs per 30 yds at a minimum.

    So while 1.00 WP given a successful conversion may be a tad high, it's probably no lower than 0.99 or 0.995.

    25 seconds simply isn't enough time, even for Manning.

  10. Alchemist says:

    Those who want to quibble with your analysis are missing the whole point. Even if you tweak the numbers a little bit here or there, even if you start making assumptions based upon Manning's/Brady's skills, even if you alter the analysis to include 4th-quarter/red-zone/last-minute stats, BELICHICK STILL MADE A GOOD CALL.

    When I listen to all those idiots on ESPN, NBC, and talk radio, they make it sound like punting would have given the Pats a 95% WP and going for it gave the Pats a 5% WP. Your analysis shows, based upon empirical evidence, that going for it gave the Pats a 79% WP and punting would have given them a 70% WP. Even if we assume that your numbers are a little off and that going for it wasn't quite the positive option that you paint it to be, it was still a good call!

    One of the most significant lessons from your analysis is that the go-for-it option is not the sure-lose proposition that all of the mediots claim it is.

  11. Eric says:

    Let's not forget that one of the NFL's most famous, long, 2 minute, TD to win drives happened with Bill watching, as his undefeated Pats lost to a Manning in the final minute.

  12. Ryan says:

    @Anonymous1 and w/ mcnallydp: I think the fact that you put all the weight on one play makes it even better. If the odds are in your favor going for it, which Brian has very well explained (and even if they aren't, they're damn close), then you WANT everything to come down to one play.

    Think of it this way, in terms of gambling: if you want to bet red or black on a roulette table, you bet all your money on ONE spin, because the more times you spin, the greater your odds of failure (essentially, how the house always wins: volume betting). Of course you'll feel like a schmuck if you lose, and it's "more fun" to let it play out and keep gambling, but it's not your best option.

    @Brian: Thanks for your analysis. As soon as that decision was made I was looking forward to seeing your numbers the next morning. Just got finished reading/listening to TMQ on espn.com, and they gave you a nice little shoutout (even if he did get the URL wrong). Congrats on the increased exposure and success, keep up the good work.

  13. Anonymous says:

    One thing that I think that you should emphasize is that there is a CORRECT answer. If we could perform this experiment an infinite number of times, then the results would give us exact percentages that you are estimating. To me, the most important point that absolutely no one can argue, is that this mathematical model perfectly describes the situation. Anyone who disagrees with the choice to go for it MUST be able to justify it using these equations. All that being said, I do think that some of your estimates aren't correct. I think that the Colts played that play more like a 2 point conversion, i.e. red zone 4th and 2. That lowers the percentage. I also think that the estimate of the Colts chances of scoring are higher than you estimated as well. But no one knows the exact parameters. But I think that most intelligent people would not argue that the decision was pretty close (within 10% either way). So all of the fighting can almost be boiled down to people arguing over a coin flip. "You idiot, you called tails! We have always called heads around here and you went against it!! You are a complete moron!!!!!!"

  14. Anonymous says:

    I am totally with you on this entire debate. Isn't it giving you headache trying to explain math to these morons? I know it sure makes my head hurt. And now, after arguing with them, I realize why the casinos make so much money! People trust their intuition, no matter what! Even when they can be shown to be completely wrong. I wish some of them would come on and try to explain why betting pass in Craps is profitable, but don't pass is a losing proposition.

  15. Anonymous says:

    Phenomenal analysis. But as thorough as the numbers are, I wouldn't call a .79 vs .70 winning probability a sure thing. I think it really boils down to the fact that it's not about whether or not Belichick made the "right" call. The only way to measure that is by results so it turned out in this particular case that it was the "wrong" call. That's not to say that punting would have won it.

    The point is that the call did make sense. EITHER call would have made sense given the situation. Belichick went with the unconventional call. Why is it unconventional? Because people fear making a decisive move until the last second. That's what makes Belichick different.

    I don't buy the no confidence in the defense bit. (1) It's HUGE confidence in your offense. (2) It's confidence that you defense will be able to hold them out of the end zone. That's why they tackled Addai at the 1 instead of letting him get in. That's why it took a perfect slant route and throw to complete that TD.

    I still think the Pats should've let Addai dive into the end zone earlier. Brady with the ball and about 30 seconds needing only a FG - that's a bet.

  16. John says:

    Analysis like this is very common, but also rather irksome. The result is absolutely irrelevant to the decision making process and has no bearing on whether or not it was the correct call. If for no other reason than the outcome wasn't available information when the decision was made so can't really be applied to whether or not the decision was a mistake.

    In other words, if I offered a bet where every time a coin came up heads I gave you $5 and every time it came up tails you gave me $1 then even the most risk adverse person would likely take this bet. However, if after the first toss it came up tails would the person who took this bet have made a mistake? By your logic it would appear they have as the result indicates they lost, but in reality they're huge winners if they continue with this bet.

    This is very similar to the 4th down call in question. While the decision may not be as easy as the coin flip example I gave, the decision still appears to be a winning one over the long run and the result of the play is completely irrelevant.

  17. Anonymous says:

    This is somewhat off topic, but do you have any comment on the dispute between Malcolm Gladwell (see the last two posts on the blog at gladwell.com), Steven Pinker (see his review of Gladwell's latest book in the NY Times at http://www.nytimes.com/2009/11/15/books/review/Pinker-t.html), and Steve Sailer (http://isteve.blogspot.com/2009/01/can-you-predict-who-will-be-good-nfl.html) over whether general managers can properly predict the success of quarterbacks? Dave Berri hasn't weighed in yet, but promises to do so soon. (See comments at http://dberri.wordpress.com/2009/11/17/yes-it-is-early-%e2%80%93-but-portland-is-the-best-in-the-west/#comments.)

  18. John says:

    While reading over my post I realized that it could appear that I disagree with Brian's analysis. This is certainly not the case as I am a big fan and always look forward to seeing it. The analysis I was referring to was in the anonymous post right above my original post.

    Just wanted to clarify. Thanks.

  19. Jeff Clarke says:

    I love all of the mainstream "analysis" that says Belichick was being arrogant by making this call. Since when is being mathematically correct arrogant.

    In first grade, all the other kids said that 2+2= 5 but one kid was arrogant enough to say it was 4.

  20. trustfundbaby says:

    What about looking at this from a different angle? ... I'm no statistician so please disabuse me of this if I'm wrong.

    Could it be that the difference in success ratios between the two options was small enough (9%) that you would want to take into consideration the risk of failure if the 4th down play DIDN'T work?

    Yes, I know its factored into the winning percentages, but realizing that if you fail on 4th and 2 ... your chance of losing (53%) is now almost double your chance of losing on the punt (30%) [with Brady I'd say those numbers were closer to 80% v 40%] gives you a difference perspective no?

    One could see why people think its so risky, even though they can't seem to articulate it.

    This (assessing the risk) only matters because the Pats were leading and the difference in the possibility of winning between the two plays was so small.

    I'd like to get your thoughts on that if you have the time.

  21. Kulko says:

    @ Harry,

    I am not so sure, that the mainstream media coverage was as one-sided as it seems.

    While I agree that the initial reaction was a negative one, this call has also spawned a very heated discussion which exposed a lot of people to the fact that the games has changed in the last two decades.

    While I dont expect any NFL team to play 4 Down football any time soon, I would guess the trys on 4th down will go up on the next season, when the coaches who care about winning catch onto what really happened out there in Indy.

  22. Anonymous says:

    I've been fully on board with this analysis, but another thing to consider might be to factor in the likelihood of succeeding on the punt attempt versus the likelihood of succeeding on the conversion attempt. While punting may only give you 70% chance of winning, you're likely to be successful (e.g., not have the punt blocked or returned to at least the Pats 28) probably 90% of the time. 70 x .9 = 63%. On the other hand, while converting the 4th down gives you a 98% chance of winning, it will only be successful 60% of the time. 98 x .6 = 58.8%.

    I guess the answer is that the Pats still had a chance of winning if they failed to convert. 47 x. .4 = 18.8%, which, when added to the 58.8%, raises that number to 77.6%. But if you do that you also have to add that number to the chances of winning following the unsuccessful punt, raising it to 81.8%.

    My numbers on the chances for a successful punt are only estimates, but it might be something that is worth looking into.

    BP Glass

  23. Anonymous says:

    On second thought, when you add the chance of winning following an "unsuccessful" punt, it isn't 18.8 percent but only 4.7% (47% x. .10), bringing that total only to 67.7%. I guess I'll have to think about this some more.

    BP

  24. Anonymous says:

    What I found fascinating was the vehemence of the initial negative opinions. Over the day that followed, after all of the stat geeks and unconventional coaches weighed in, it seemed that this raised awareness of how statistical analysis has questioned conventional thinking. Who knows, maybe next we'll see an onside kick attempt with two minutes left in a tie game.

  25. Anonymous says:

    There's another small kicker to the "go for it" argument. With 2 minutes on the clock, the likelihood that Indy scores from the Pats 28 with time on the clock is slightly higher than from their own 30. In the "go fot it an miss category", you need to factor in a small next step of the Pats getting the ball back after a Colts TD with some time and kicking a winning FG. I'm guessing that would add a hair to the favorable numbers for Belicheck's decision to go for it.

  26. Anonymous says:

    Also, it might be worth factoring in the chance of NE getting a field goal following a quick TD by the Colts (e.g., punt returned for TD). This might require a much more complicated analysis that involves a complicated tree of probabilities.

    I. Punt
    A. 90% chance of Successful punt to the Indy 34 (70%) = 63%
    B. 2% chance of Returned for a TD (50%) 1%
    C. 8% chance Blocked or returned to the 28 (47%) = 3.76%
    II. go for it
    A. 60% chance of 1st & 10 (98%) = 58.8%
    B. 40% Turn over on downs (47%) = 18.8

    And so on, and so on. Query whether considering enough possibilities might change things.

    Note that this assumes that NE would have a 50% chance of getting a field goal following a punt returned for a touchdown by the Colts. I have no idea what the actual chance would be.

    BP

  27. Anonymous says:

    I would like to see someone find a correlation between support for Belichick's decision and level of education. It won't convince anyone of anything, but it would be satisfying to see.

  28. Tarr says:

    I just remembered how a similar smart decision to go for it on 4th down came up in a Georgia Tech game a couple weeks ago:

    http://sports.espn.go.com/ncf/playbyplay?gameId=293110059&period=5

    This was OT in college, and the other team had already kicked a field goal in their chance to score.

    GT faced a 4th and 1 (really less than 1) 5 yards out. If they kick the field goal, they go to another round of OT. In stead, they trusted their (strong) running game, got the first down, and won the game on the next play.

    To me, this is a pretty clear decision. If you go to 2OT, that's more or less a coin flip situation. Getting about half a yard, on the other hand, is much better than an even proposition, especially with Tech's physical, running QB. So you go for it.

    When it became clear that GT was going to go for it, the announcers were VERY critical of this decision. If they had been stuffed, I have no doubt that the announcers, as well as the talking heads in the studio, would have laid into the coach for his "bonehead decision". In stead, it was called gutsy, and everyone quickly moved on and got ready to criticize the next coach who made such a call.

    This is a big part of why coaches are so timid about these decisions. There's a massive asymmetry between the praise/credit they get when these decisions work out, and the criticism they get when these decisions don't work out. None of the announcers were admitting they were wrong and apologizing for their bonehead advice.

  29. Anonymous says:

    I was wondering about the education angle as well.

    I was especially wondering about it with all of the ex-players and coaches pontificating on what a bad call it was.

    I know I might be feeding a stereotype here and there are some Bill Bradleys out there that are both athletes and Rhodes scholars.

    But I think the majority are getting somebody from the band to do their homework. If you've never seen an expected value analysis or a decision tree before, you are likely to distrust it.

  30. Jeff Clarke says:

    Tarr,

    Good point.

    Another factor in the Georgia Tech game was that Tech would have gone first in the second overtime.

    The team that goes first is at a large disadvantage because the second team will never kick a field goal if they know they need 7. Historically, they've only won 45% of the games.

    Tech's coach might have used 45 and not 50 as the hurdle he needed to jump in order to make going the right call.

  31. James says:

    Tarr,

    I've spent all week disagreeing with the people who've been saying the mainstream media would be calling Belichick a genius if they'd gotten the first down, and that Georgia Tech game supports my point very nicely.

    Nowhere near as high-profile, obviously, but I also remember how quickly the announcers lost interest after the conversion. Looking around online, the coverage is mostly positive, but from the standpoint that he was lucky and/or gutsy. For what it's worth, here's what I got from Google News searches limited to articles between Nov. 7 and Nov. 9:

    "Paul Johnson" + "Georgia Tech" + genius = 0 results
    "Paul Johnson" + "Georgia Tech" + gutsy = 313 results

    Sure, Belichick's reputation is better than Johnson's (which is still pretty good), but I don't see how the coverage of a successful Belichick "gamble" would be any different. It'd just be on a larger scale.

    (by the way, I'm a fan of your work on RSD)

  32. Griffin says:

    Here's what I don't get with regards to 'offending' the defense: Why don't those same people think punting offends the offense?

  33. James says:

    I completely agree that the "punting shows confidence in the defense" stuff is absurd, but I can think of one point that sort of supports that line of thought. Whether you punt or fail on a conversion attempt, the defense will be taking the field, and the offense will be going to the sideline.

    So if we're going on the theory that NFL players have the emotional stability of toddlers, maybe a few minutes on the bench helps them heal their psyches. If that's the case, we only need to worry about the defense.

  34. Anonymous says:

    Another thing to consider: after the turnover on downs, it took the Indy offense 2 plays in 45 seconds to move the ball 28 yards to the Patriots 1 (and that's on a compressed field where the defense supposedly has the advantage). That's only 10 yards short of what a typical punt would net. Assuming things would play out similarly following a punt, then a punt would just be the equivalent of burning a minute of clock and handing the ball to the Colts at the Pats 29.

    I think that further supports that Belichick made the right decision.

  35. DM says:

    Keep in mind that in the Georgia Tech game, they had the ACC Championship on the line. A loss would have all but guaranteed that they wouldn't have made it to the title game, a win and all they had to do was beat Duke.

    After the game Paul Johnson said two things that I'm sure Bill Belichick was thinking:

    "If we can't get one yard, we don't deserve to win" and "if you tell the kids play to win, then you have to play to win."

  36. Edward Lee says:

    @Tarr: I have to think that throwing to an away route is an option against a blitz, instead of having 5 receivers posting up at the sticks. A throw over the top will be slightly easier to complete without the back line of the end zone. (Welker did, in fact, get away from his defender, but only about 5-7 yards downfield).

    Re: punting indoors: The indoor surface surely helps both offenses as well, which partially negates the improved punting. In addition, Hanson is one of the weakest punters in the league -- he does have a big leg but shanks way more than his fair share of punts.

  37. Anonymous says:

    I still don't buy the 60% 4th and 2 conversion rate. This was essentially a 2 pt conversion since the Colts could sellout on defense and didn't care if the Pats got 2 yards or 72 yards on the play. 2 pt conversions have historically been converted about 45% of the time.

    So using the given 53% and 27% chances of scoring from Pats 28 or Colts 28 (I'm using a 44 net since that's what Hanson had done that day and it was an ideal punting situation):

    .45* + .55*(1-.53) = 71% chance.

    Punting =73% chance.

    Punting is the smarter decision if you use 2 pt conversion as your percentage.

    50% conversion would be roughly the break even point.

    I think the variances and uncertainties of each of the different factors (conversion percentage, net punt expectation, probability of scoring from Pats 28, probability of scoring from Colts 28, etc) is such that you can't state definitively that the Pats should have gone for it. In fact, I'd say that the net probabilities are close enough that he should have kicked and not cause a locker room distraction or potential morale problems with his team.

  38. James says:

    It makes absolutely no difference that a spot in the ACC Championship was on the line. As far as Paul Johnson is concerned, the only thing that's on the line is a game against Wake Forest.

    Are you suggesting that ACC title game scenarios have a bearing on decisions regarding how best to beat Wake Forest in overtime? Unless there's some bizarre set of circumstances in which Tech would improve their position in the conference by losing (and I'm not going to commit any mental energy to trying to figure out if that's even possible), their goal is to beat Wake Forest. Even in that fantastical scenario where they're trying to lose, the strategic considerations remain the same--they'd just have to do the opposite.

  39. Tarr says:

    @Anonymous:

    1) It's slightly easier than a 2-point coversion, because a 15 yard completion wins the game, in stead of being out of bounds.

    2) The historical 45% number for 2-point conversions counts every botched snap where the holder desperately tries a bootleg or heaves the ball into the endzone as a "failed 2-point try". If you take out those plays, the percentage is higher.

    3) If we're going to try to tweak the numbers, we really ought to consider the Patriot's historical success in short yardage over the last few years, as well as how much more dangerous the Colts offense is (especially at that stage of the game against the tired Pats D) versus a league average offense. All enhancements to the offense make going for it look better.

    @Red Line:

    True. I didn't mean to imply that the blitz negates the possibility of a longer gain. What I meant was that the combination of the blitz and very tight press coverage of the receivers increases the average length of a successful play, while reducing the expected probability of a successful play. This tradeoff is probably bad in general, but it's good in this particular case when the Colts only really care about whether the Pats gain 2 yards or not.

  40. Jonathan says:

    "I still don't buy the 60% 4th and 2 conversion rate. Tis was essentially a 2 pt conversion since the Colts could sellout on defense and didn't care if the Pats got 2 yards or 72 yards on the play."

    I touched on this in the previous thread, and I completely disagree with this line of thinking. All 4th and 2 attempts encourage the defense to sell out, since possession of the ball is at stake. It's also very infrequent for an offense to pass downfield on a 4th and 2 attempt. The only exception I can think of is if a team trailing at the end of the game, and doesn't have time to run it up the middle for a 2 yard gain. But if a team is forced to pass in that situation, that would decrease the probability of converting the fourth down anyway.

    Bottom line, nobody is going to play the prevent defense when it's fourth and two. The fact that the Colts were unconcerned with downfield threats is not a departure from any other fourth and two situation.

    Comparing this fourth and 2 to a two points conversion should be less accurate, because NE did not have to deal with a compressed field. This makes a big difference when your QB is Tom Brady and he has Randy Moss to throw to. If anything, the Pats' offense is so reliant on the pass that the Colts would have to respect the pass more than the average 4th-and-2 defense.

  41. James says:

    (alright, one more, and then I'm taking a break)

    This is an interesting point:

    "...the net probabilities are close enough that he should have kicked and not cause a locker room distraction or potential morale problems with his team."

    By going for it (and failing), he directed all the scrutiny and controversy at himself. If they had punted and lost, all the attention would be on the fact that they somehow blew a 17-point lead in the 4th quarter, which actually is a legitimate concern. And nobody's talking about how Peyton Manning is apparently now officially unstoppable (he doesn't even get full credit for winning that game, since most pundits think the Patriots "gambled" it away) either. Seems to me that constant talk about either of those stories would be much worse for morale.

  42. Anonymous says:

    Tarr,

    Even if we increase the 2 pt conversion % to 50% to account for these factors, that's merely break even with punting. The trouble is that each variable has a certain +/- margin of error.

    1) 4th and 2 conversion 50% +/- a%
    2) Colts chance of scoring from 28 53% +/- b%
    3) Net punting expectation 44 +/- c yards
    4) Colts chance of scoring from their own 28 27% +/- d%

    Remember, all the numbers being thrown around are merely estimates, not exact values. Given any set of a, b, c, and d, it's pretty much impossible to state definitely that going for it is the right decision or punting is the right decision. It only takes very small values of a, b, c, and d to affect significant changes in the results.

  43. Matt says:

    Anon's most recent post is a key point. We can't say for certain whether punting was 70% or 72% or 68%. There's some margin of error, or confidence interval, in play here. And that's true of both numbers. With some level of confidence, we can say punting gave his team a 71% chance to win. Similary, we can say with some level of confidence that going for it gave them a 75% chance to win (I'm more apt to take 50% for the conv of the 4th down). The two confidence intervals can certainly overlap, so it's certainly possible that BB decreased his team's true chance of winning by 1% or 2%. It's not likely, but possible.

    But coaches make mistakes like that all the time. Heck, Managini punted with 6 minutes left down 16 on Monday night. I'm certain he cost his team more than 1% off their win percentage with that decision.

    The announcers on the broadcast, and commentators since, have acted as though he took his team's win percentage from 70% to 50%, or worse. That's just not possible based on any reasonable set of assumptions for the applicable percentages.

  44. Matt M says:

    Brian,

    Thanks for the response on the WPA if the Patriots get the first down.

    Great comment on the Celtics broadcast just now. Mike Gorman, apparently bored with yet another double digit lead, asked Tommy Heinsohn what he thought of the 4th and 2 call. Heinsohn's response: "Coaches make moves; some of em work and some of em don't."

  45. Pat Laffaye says:

    Take a look at this key NE series and no question there were a few blunders...

    P.McAfee kicks 70 yards from IND 30 to end zone, Touchback.
    Timeout #2 by NE at 02:23.
    New England Patriots at 2:23
    1-10-NE 20 (2:23) (Shotgun) K.Faulk up the middle to NE 20 for no gain (R.Brock).
    Timeout #1 by IND at 02:18.
    2-10-NE 20 (2:18) (Shotgun) T.Brady pass short right to W.Welker to NE 28 for 8 yards (J.Lacey, C.Session).
    Timeout #2 by IND at 02:11.
    3-2-NE 28 (2:11) (Shotgun) T.Brady pass incomplete short right to W.Welker (J.Powers).
    Timeout #3 by NE at 02:08.
    4-2-NE 28 (2:08) (Shotgun) T.Brady pass short right to K.Faulk to NE 29 for 1 yard (M.Bullitt).

    1) TIMEOUTS: Clearly NE wasted at least one, and worse have none left to challenge the spot on the 4th down play.
    2) PLAY CALLS: 3 out of 4 passing plays is not going help NE run time off the clock.
    3) CLOCK: Time mismanagement is evident by NE inability to reach the 2:00 warning in a mere 23 seconds.

    Now for a little constructive criticism... (I don't mean to pick on you Brian -- this site is a definite fave of mine!)

    Unfortunately, I think both the analysis and the BB model fall a little short in this particular situation. Empirically, it IGNORES TIMEOUTS and 4 of them were called in 4 plays!! Think about this... that's 2/3 of the FULL HALF allocation FOR BOTH TEAMS, in again only 23 seconds!! Huge increase in leverage index and probability swings. Without a doubt, that 4th down play had the highest leverage index of the entire game -- and that is not considered either, is it?

    If this were a chess match, we are in an end-game situation, not the opening sequences, not the mid-game. The game is on the line, the division race is on the line. The problem was NE coaching needed to be looking 2-3 moves ahead. Too cocky perhaps, they completely ignored failure, and neither has most of the analysis I've seen.

    Making the choice solely based on PUNT vs. GO FOR IT, I would agree the latter is the correct choice. But at the same time, I would argue that in this case, analysis of just one play is NOT SUFFICIENT. Perhaps NBC media may not have said it outright, but playing the percentages should indeed include "what if's" for the 2nd event.

    So if we consider each possession as a separate event, we then need to evaluate the possible results of both team's possessions. Two consecutive successes result in a WIN. Both events would be deemed a success for NE only if they 1) can hold on to ball until the end of the game OR 2) score (unlikely) OR 3) give up the ball AND prevent IND from scoring a TD. Both events would be a success for IND only if they can 1) get the ball back AND 2) score a Touchdown, nothing less.

    Since it is an end-game situation, there remain few possibilities, and I know for sure possession results, hence probabilities of both a win AND A LOSS can be computed as described.

  46. Anonymous says:

    I still don't see how you can generalize a "4th and 2 outside the red zone" to "4th and 2, leading, inside your 30" situation. They're different. No, a defense facing 4th and 2 with a 10 point lead and 2:04 on the clock and 70 yards at their back will not "sell out" as is argued above. They want to deny the TD, not the first down. They may not play a prevent D, but they're also going to try to keep everything in front of them. Facing 1st and 10 at the 2 minute mark with the opponent still needing 65 yards is a positive outcome for the defense, if not the ideal one. Just how many examples of a coach going for a 4th and 2 (or more) conversion to seal a win in the last two minutes are there in the data? And of those, how many are inside their 30 (or even 40) against a strong offensive team? (I doubt anyone questions the decision if it was the Lions and the 48, for instance.)

    Again, I just don't see how that generalization works. As league-wide, long-term numbers, they may be accurate, but all "4th and 2 situations outside the red zone" are not the same.

    Situational statistics do matter; it's why the casinos love a "Dummy's Guide to Blackjack" player and hate the card counter...

  47. Becephalus says:

    I think you might be giving IND a little too much credit as far as not being knowing what the optimal counter move there is. they certainly played it well, but to just assume they will adopt an optimum counter strategy isn't exactly waranted. On top of that I am not sure I wouldn't have expected a 7yd pass as IND there.

    It is easy to say treat it like a 2pt conversion, but it wasn't a 2pt conversion.

    And of course finally even if you do treat it that way it is not really that interesting a decision, and decisions that should be 10 times more contrvertial are made each game.

  48. Anonymous says:

    I am curious as to what Carl Von Clausewitz would say about this entire situation.

  49. Jonathan says:

    "a defense facing 4th and 2 with a 10 point lead and 2:04 on the clock and 70 yards at their back"

    Granted. That specific situation will naturally cause a higher conversion rate for the offense. The defense has a huge lead, the offense has 70 yards to drive. MOST IMPORTANTLY the offense has the two minute warning to fall back upon, so they have the option to run.

    If it was the same situation with over 2:20 on the clock, or under two minutes to go, then the offense isn't just thinking about the conversion. Like the defense, they are taking the entire situation into context. Therefore, they will pass most of the time, even if and when the defense plays a defense to stop the pass. This should reduce the offense's chances of converting on fourth down.

    I would imagine that most fourth-and-2 plays happen in "no man's land." Coaches historically punt from their own territory, and kick a field goal from inside the opponents' thirty. So most 4-and-2 plays happen in, say, the middle of the third quarter, on the opponent 40 yard line. Or something like that.

    I don't have any statistical data to back this up, so I may very well be wrong. But from what I have observed, teams in fourth down situations tend to focus all of their resources on either getting or preventing a first down. It seems quite unusual for the offense to go for the home run ball in a fourth and short situation, because a long pass should have less than a 50% completion rate, even if the WR is only single covered. Once in a while they will try a play like that to catch the defense off guard. But such plays are the exception, so the defense shouldn't commit anybody to covering deep threats, beyond what is necessary to prevent a first down.

    Intuitively, I would think that selling out is the right call for the defense. If I'm on defense, I'm thinking the ball is already in our territory. Who cares about the off-chance that they get a big gainer? If they get a first down, they're probably going to drive right into the red zone anyways. But if we stop them, we get great field position.

    All of this points towards the Indy game being more of an average 4th-and-2, and less of a 2-point conversion. The back of the end zone matters more than the ability to sorta ignore the deep threat. Maybe Indy could ignore the deep threat a little more than the average defense, but not enough to bump the % any lower than 57%.

    The historical averages factor in fourth down situations from desperate, two-minute situations. Factor those out, and we're back around 60% again.

    Ultimately, it looks to me like the situational factors roughly cancel each other out.

  50. Anonymous says:

    Dear "BP"

    Please get a clue before posting anymore.. your fake math lowers the credibilty of us all. thanks.

  51. free ads says:

    I am curious as to what Carl Von Clausewitz would say about this entire situation.

  52. Anonymous says:

    "Intuitively, I would think that selling out is the right call for the defense. If I'm on defense, I'm thinking the ball is already in our territory. Who cares about the off-chance that they get a big gainer? If they get a first down, they're probably going to drive right into the red zone anyways. But if we stop them, we get great field position."

    You seriously don't see the difference between giving up a TD or a 30 yd gain and letting the other team have a 1st down on the 40 yard line after a 2 yard completion?

    It matters a GREAT deal when the other team has a 4th and 2 on the 40-45 yard line how many yards they are held to even if they convert the 1st down.

    Hence this was not a "normal" 4th and 2 and more like a 2 pt conversion.

  53. Edward Lee says:

    In 1996, Texas went for it on 4th-and-1 from their own 28 while up 3 at the end of a game:

    http://www.youtube.com/watch?v=igWLuCfPnc4

    The stats are obviously different for these teams, but it does demonstrate that the defense does have to worry about the throw over the top at least a little bit.

  54. Jonathan says:

    "You seriously don't see the difference between giving up a TD or letting the other team have a first down on the 40 yard line?"

    Please quote where I said that there was no difference, because I didn't. I said the defense would not be particularly concerned with the big gainer, because:

    a) Possession of the ball is at stake, so it is very unlikely that the offense is going to go downfield since that is a low percentage play.
    b) The difference between first down at the 38 yard line and first down at the 8 yard line, while significant, is relatively small. Therefore, the risk/reward for a downfield play is not in favor of going for it, so it is unusual for it to happen.

    Even if they do send receivers downfield, the defense would still single-cover them. That alone should discourage downfield passes. The thing is, Indy would also have to send cornerbacks downfield to cover Randy Moss. If he's unaccounted for, he will easily convert the first down (similar to the youtube clip above).

    Indy may be encouraged to sellout more than the average defense, but only by a small amount. And that effect is counterblanced by the fact that the offense is also pouring all of its energy into converting the first down, unlike a few offenses who will desperately throw 20 yards downfield and towards the sideline.

  55. Justin says:

    The Onion's got a good tweak on this game if you need a laugh:

    http://www.theonion.com/content/news/patriots_lead_colts_at_halftime?utm_source=onion_rss_daily

    But I don't agree that Belichick made a blunder here.

    One criticism Burke should've addressed is the suggestion of using team-specific data from years past, as it's appeared in this thread. Someone suggested using Patriots 4th down data from years past...

    What about using Vikings, Bengals, or Saints data from 2 years ago? How relevant would that be seeing how improved all these teams are this year (for that matter, how relevant is it to compare the Titans and Panthers this year compared to last)? The Patriots certainly aren't the same team they've been for 2 years, though they've enjoyed decent records from year-to-year matched by very few other NFL teams?

    We're best off using the 10 year NFL baseline, which is mix of good an bad teams. Once you have that, then you can estimate how good your team is in this situation compared, but as Burke pointed out if you believe something like "Indy has a better than average offense" then that helps Indy in both situations, whether they get the ball on their own 30 or on NE's thirty. Or if you believe "New England has a better than average defense" then that helps the Pats in both situations on their own 30 or if they punt it to Indy's 30.

    But as someone pointed out in the other thread, if Belichick was going to go for it on 4th down, he should've made that decision before 3rd down and tried to run on 3rd and 2 in hopes of making it at least 4th and 1 (if not outright converting), that would've forced the Colts to defend both a run and a pass, whereas 4th and 2 is an obvious passing down in current NFL thinking.

  56. nottom says:

    I have a problem with the argument that not punting is a slap in the face of the defense. While I think this argument can be made against the "they should have let the Colts score" camp, I think you could certainly make an argument that going for it actually shows CONFIDENCE in your defense, because if you don't make it you feel they will keep the Colts out of the endzone.

  57. Jeff Clarke says:

    There is another factor that has only been barely touched upon here. I don't think the Colts believed there would ever be a play. I sure didn't.

    In that situation, the offense often times tries a hard count. Fuck with their timing and get them to jump offsides. Why does this work? Defenses routinely try to estimate the offenses timing and get a head start. They do this because just waiting until the play has already begun screws with the center of gravity. When the off linemen and def linemen collide, the off linemen have an advantage if they were moving first. If a defense never tried to estimate the snap count, they would have no offsides penalties but they would also probably have the worst defense in the league.

    I'm pretty sure that in this instance, the def coordinator played it safe. He told his players that it was highly unlikely there would be a play. "Whatever you do, don't jump offsides" was probably the defensive plan for the play. The surprise of the snap had to work in New England's favor.

    I'm not going to try to quantify this because I have no idea where to start, but I think it probably means more than some of the other fairly minor factors that have been thrown around.

  58. Justin says:

    Another thought I'd like to introduce in there...

    I think that even Indy sensed that Belichick was doing the right thing here. Without looking up specific quotes, remembering some Colts reactions on the highlight shows Monday they were well aware they were about to lose the game without getting their offense on if the Patriots converted.

    Almost as if they "sensed" it would be to their advantage to be assured the ball in a punt, rather than having to risk stopping a 2 yard conversion.

    Perhaps Manning is extra appreciative about the value of the ball after not getting on the field against San Diego in last years playoff overtime.

    Just think it's interesting that many people think New England made a mistake from their perspective, but no one is looking at this from the Colts perspective. I think they would rather be assured the ball, even if it means 40 yards back, than risk losing the game for failing to stop a 4th and 2. Belichick's decision force the Colts into the latter situation.

  59. Anonymous says:

    my guess is those studies are not very representative of the situation that we had in the NE/Indy game. Most fourth downs attempted at that place on the field (28 yard line) are not done under the pressure and known circumstances that were in place.

    1) Most fourth down situations in one's own territory involve trickery,fake punts or plays, or rushed attempts designed to have the defense unorganized. NE lined up at scrimmage with the known intention of going for it; therefore, trickery was not at hand.

    2) Most fourth down situations are not make or break situations for the defense. Here, Indy had to have a stop in order to win the game. Therefore, they had every incentive to shallow up coverage and protect against a sub-5 yard game because such a play would almost certainly be the type of play that teams run trying to gain 2 yards. It's why teams can move down the field so easily then get clogged up in the red zone. The defense in a red zone can shallow up coverage because there is no down field threat bc there is no down field. This situation is exactly analogous.

    3) Most fourth down attempts are usually less than 2 yards. Here, NE had to go 2 full yards. 2 yards on a running play with a shallow defense would have been extremely difficult. Therefore, NE had to pass the ball and Indy would know this. Indy knew a sub-5 yard pass play was most likely coming, so they would throw the house at preventing a short yardage pass. They did, and stopped the conversion and won the game.

    So, I am extremely skeptical of any study done that shows 60% because I doubt the study is representative of the exact situation that we had. My guess is that this type of situation would be in the other 40% almost all the time.

  60. Anonymous says:

    "....in the other 40% almost all the time"


    Ummmm no....

    As we've gone over in this thread a zillion times a considerable number of the factors that make this case different (i.e. NE's strong offense, Ind's weak secondary, playing indoors, the fact that Ind was playing extremely conservative against the expected hard count, etc) actually increased New England's odds from the 60% listed. There were a couple of factors (no defense at all against a 40+ yard pass, etc) that decrease the odds.

    The 60% number is just an estimate. An argument could be made to lower it to the low 50s. But if you seriously believe that it is much lower than that, give me your number and we need to set up a series of wagers on the next football game.

  61. Happy says:

    Matt M brings up a good point that reduces the WP of going for it, but I still think a major argument in favor of going for it has been overlooked.

    In the event that the conversion failed, there is a greater likelihood of Indy scoring with too much time left on the clock if they only had to travel 28 yards vs. 70. They couldn't pull an MJD since they needed a touchdown. That meant that Belichick had three ways to win (convert, keep Indy out of the end-zone, score a field goal after Indy scored a touchdown) verses one way to win if he punted and Indy drove down the field scoring a TD with only a few seconds left. (Actually, 4 if you consider drawing IND off-sides as a separate proposition).

    As it turned out, Indy left only 13 seconds. But the probability of Indy scoring and NE eventually winning was higher in the go-for it scenario.

    As Brian alluded to, it's not possible to consider every variable but that shouldn't keep us from trying. What he's done in this analysis is consider the dominant factors and made an assumption that the neglected variables and variability are roughly evenly distributed and therefore don't change the final analysis. I think he's succeeded in this effort. Nice work!

  62. Happy says:

    Regarding the emotional argument and the defense:

    * Belichick may have a unique relationship with his defense and may be able to explain why he did what he did in ways we cannot imagine.

    * Belichick is the coach, and his 1st task is to win. Coddling his players is not his reponsbility.

    * The players are supposed to be professionals. They ultimately have to deal with it.

  63. Anonymous says:

    "The 60% number is just an estimate. An argument could be made to lower it to the low 50s. But if you seriously believe that it is much lower than that, give me your number and we need to set up a series of wagers on the next football game."

    A wager on the next football game would not work because it would be extraordinarily unlikely that this situation would present itself again. The 60% is based on all 4th down and 2 game situations outside of the red zone (at least from my reading). I am simply arguing that this is not representative. The 40% that are not successful are likely unsuccessful for two reasons: 1) luck (i.e. something we cannot statistically quantify) and 2) fourth down events that fail with regularity and can be quantified. This study was not done in a procedure aimed at determining why 40% of attempts fail. I am hypothesizing that a good portion of these failed 4th and 2 attempts are due to game situations like this one that will fail at a high rate. 4th and 2 at the opponents 40 yard line in the middle of the game is a completely different situation that 4th and 2 at one's own 28 yard line with the entire game on the line.

  64. Anonymous says:

    the percentage for goforit in the boston globe is utterly incorrect. using the input data, it should be 67.6. mr burke, learn probability theory!!

  65. Anonymous says:

    "Please quote where I said that there was no difference, because I didn't. "

    "Who cares about the off-chance that they get a big gainer?"

    That was your EXACT words. Who cares if they get a big gainer is essentially saying that there is no difference.


    "b) The difference between first down at the 38 yard line and first down at the 8 yard line, while significant, is relatively small. Therefore, the risk/reward for a downfield play is not in favor of going for it, so it is unusual for it to happen."

    Relatively small??? A TD has a point expectation of 7 pts obviously. A 1st and goal from the 8 probably has about 6 pts or maybe a little higher point expectation. A 1st and 10 from the 38 has maybe a 4 pt expectation. A defense against a 4th and 2 from the 40 in the 3rd quarter has to respect the threat of a deep pass because an extra 30-40 yards is HUGE in that situation, unlike the Indy one.

    I'm pretty sure most coaches think the difference between 1st and goal at the 8 and 1st and 10 at the 38 is more than "relatively small".

  66. Jonathan says:

    Sorry, I didn't mean there was no difference, but I'd rather not argue semantics. I'll simply concede that there is a difference...cause clearly there is.

    Anyways...

    A touchdown is worth 6.3 expected points, since the scoring team becomes the kicking team. This leads to an additional EP of -0.7 for the kicking team.

    So 6.3 EP scoring for a touchdown.
    2.5 EP for having the ball at the 40 yard line.
    And -1.3 EP for giving up the ball at the same spot.

    www.advancednflstats.com/2009/09/fourth-down-study-part-1.html

    The difference between a stop and a 2 yard first down is 3.8 EP. The difference between a touchdown and a 2 yard gain is the same, 3.8 EP.

    A long play may happen 5 or 10% of the time. A 2-yard conversion happens 60% of the time. But the first two yards hurt just as much as the next 38 yards. The defense should focus on the first two yards.

    My conclusion is that the Indy defense may sell out a tad more given their in-game circumstances. But not enough to bump the % down past 57%, and then there are situational factors that I mentioned previously which would bump it back up to 60%.

  67. Anonymous says:

    Even assuming those EP values are correct and even assuming coaches know these values, why the heck should the defense focus on those 2 yards in normal circumstances when by your own admission, the TD is TWICE as valuable as the 2 yards? And it's a heck of a lot higher than 5-10% completion percentage if all the other team's receivers are single covered man to man if there was a sellout defense.

    It still appears that the 2 pt conversion percentage is the more accurate one under this specific circumstance.

  68. Jonathan says:

    Okay, now yor arguments are sounding intellectually dishonest. I clearly did not say there was a 5-10% completion % if all of their receivers are single covered. I said that a big play may happen 5-10% of the time. That's a massive semantic difference.

    The rest of your post is asking me a question that I literally answered in the previous post. If 40 yards is twice as valuable as 2 yards, you focus an inordinate level of attention on the 2 yards, becaus the first two is worth as much as the next 38 yards. It's way more efficienty to focus your energies on the 2 yards.

    But we don't even need those stats to realize that the defense can afford to sellout. When is the last time you ever saw a deep ball on a fourth and one? Usually when the offense does a play like that, the announcers remark that the defense got caught off guard. Or that they did a great job of not getting caught out by an unexpected play. It's a surprising move. Nobody expects a home run shot on fourth and short.

    Here's another point: if the defense really is inclined to think about the deep ball, isn't the offense just as inclined to think about it? And if they are, then aren't they inclined to go for the end zone, rather than settling for just two yards? Wouldn't that hurt the "average" fourth down conversion rate instead of skewing it upwards?

  69. Anonymous says:

    All 4th and 2 attempts encourage teh defense to sell out, since possession of the ball is at stake. It's also very infrequent for an offenes to pass downfield on a 4th and 2 attempt.


    Are you on drugs? Why the HELL dont you start turning up the actual statistical number instead of making stuff up.

    I saw this happen (long pass on short yardage) at least twice this weekend alone. And I only watched 4 or 5 games. I think it happened Pitt/Notre Dame and Washington/Denver. Not matter what teams, I am quite sure I saw it happen twice this weekend and only watched several games.

    This is so typical of all statisticians, they quote whatever statistic that they happen to have (league wide 4th and short, irrespective of time remaining) and then they stubbornly refuse to consider a reasonable counter argument because they dont have that stat (4th and short, game about to be over) conveniently on hand.

    So instead of doing what would be intellectually honest and forthright, which would be to actually figure out what that number is....

    You revert to just repeating yourself with no logic to back this up. Same with this QUOTE:

    "The 60% figure is a league wide average for 4th and 2 situations outsidethe red zone, which is a reliably consistent stat across the recent decade."

    What the hell does that mean?

    I'll tell you what it means: It means you dont have the energy to actualy look up what the real statistic for that situation is.

    I guarantee you that if the end of hte game is on the line the odds of making 4th and 2 are not 60% in the NLF.

    ANd here's what I'll do. You conjure up the energy to get your lazy ass off the computer and actually run the numbers and I will be you either:

    $500; I say that odds are less than 55% of making 4th/2 with the end of game on the line, OR

    $100, I say hte odds of making it are 52% or less of making 4th/2 with the end of game on the line. (run numbers for both last 5 years and 10 years in the NFL. The bet is a push a.k.a. tie if the result is different for 5 yr/10 years.)

    Here is my email: gatodelsole@hotmail.com

    My guess is your too lazy to actually do some work and figure out the actual stat. But maybe you do...?

    Go ahead mine through your data base

  70. Anonymous says:

    "Okay, now yor arguments are sounding intellectually dishonest. I clearly did not say there was a 5-10% completion % if all of their receivers are single covered. I said that a big play may happen 5-10% of the time. That's a massive semantic difference."

    I think you're the one being intellectually dishonest. Your whole implication when mentioning a 5-10% number was in reference to a 4th and 2. If it's not in reference to the 4th and 2, then it's intellectually dishonest and disengenuous to mention that figure in this context. You seem to react very badly to being called out on your questionable statements.

    "The rest of your post is asking me a question that I literally answered in the previous post. If 40 yards is twice as valuable as 2 yards, you focus an inordinate level of attention on the 2 yards, becaus the first two is worth as much as the next 38 yards. It's way more efficienty to focus your energies on the 2 yards."

    Why? The TD is twice as valuable. So why would you sellout under normal circumstances and single all the opponent's receivers man to man? (Which is what a sellout defense does).

    "But we don't even need those stats to realize that the defense can afford to sellout. When is the last time you ever saw a deep ball on a fourth and one? Usually when the offense does a play like that, the announcers remark that the defense got caught off guard. Or that they did a great job of not getting caught out by an unexpected play. It's a surprising move. Nobody expects a home run shot on fourth and short."

    The offense only would do so if the defense truly did play single man to man on everyone, especially if the offense has an excellent deep threat or the defense has poor corners. How often do you see that defense on 4th and 2s?

    BTW, more intellectual dishonesty. Referencing 4th and 1 when clearly that is completely unrelated to the Colts game. 4th and 1 and 4th and 2 have very different strategies and conversion percentages.

    "Here's another point: if the defense really is inclined to think about the deep ball, isn't the offense just as inclined to think about it? And if they are, then aren't they inclined to go for the end zone, rather than settling for just two yards? Wouldn't that hurt the "average" fourth down conversion rate instead of skewing it upwards?"

    This is known as circular logic. The offense only would do so if they perceive that it favors them based on the defense. And that would only be the case if the defense did indeed completely sold out.

  71. Anonymous says:

    Question about the two-point conversion rate: is it representative of the entire NFL, or is it overrepresented by teams who are behind at the time (and this likely weaker)?

  72. Anonymous says:

    QUOTE: " When is the last time you saw a deep ball on fourth and one..?"

    I dont know what is wrong with you, but I saw it twice this weekend (on fourth downs, not necessarily 4th/1 which doesnt change the argument) and saw maybe 5 games....

  73. Anonymous says:

    HEre is one example of a medium throw on 4th/1 from the KC/Oak game w/ 3.22 left:


    http://www.nfl.com/gamecenter/2009111508/2009/REG10/chiefs@raiders#tab:watch

    See also MN/Det game w/ 2.29 left. The pass w/ in the short flat but the formation was a spread w/ receivers going down field. It's not like they dont have to cover those threats..

  74. Anonymous says:

    We should be grateful that they went for it. It has stimulated conversation and debate - one of the joys of following any major sport.

    Also, it has raised the profile of a more reasoned approach to coaching a game.

    EdBed

  75. Anonymous says:

    I wish somebody could explain what it is I am not seeing in this argument about how going for it on 4th down is akin to the advent of the forward pass. Forget the NE game for a second and consider all games. Brian's analysis is great, but it does not address what the score is, he says ...

    "5. The 53% and 30% chances of the Colts getting a TD don't take into account the time and score.

    Yes, they do."

    I followed the link expecting to see something about the score and nothing ... just "... how likely an offense that needs a touchdown to tie or win is to score one with about 2 minutes remaining in the game." To me that describes a margin, lead, or deficit, not a score. The first consideration by any good coach before making this crucial decision on 4th down whether to punt or go for it is the score, or more exactly the total score. If the game is a shootout then Brian’s analysis is correct, if not it’s horribly wrong.

    I personally don’t think the NFL will turn into a shootout style league. The rules will change and defense will get better. Football is not like the NBA where a “whoever has the ball last wins” game is good. Football should be 50% offense and 50% defense and keep punting as a key element. I think some of the best games in history are the low scoring defensive battles. And no, I don’t have a son who is an aspiring NFL punter.

    KenYonLV

  76. Anonymous says:

    @KenYonLV: You're most likely going to end up with sample size issues if you try to restrict the 2-minute TD stats to games that ended up with scores in the 30s. It's better to start with the NFL-wide stats, then adjust from there.

  77. Jonathan says:

    "I think you're the one being intellectually dishonest. Your whole implication when mentioning a 5-10% number was in reference to a 4th and 2...etc"

    I'll just agree to disagree then. I stated my case and you stated yours. Fair enough. :)

  78. Gordon says:

    Brian,

    You use 0.6 for 4th and 2, apparently based on stats for the past decade. I'm told that average for 2009 for converting on 4th and 3 OR LESS is 0.45 (136 for 303). Why not use more recent data? (303 seems, intuitively, like a decent sample size, or 2008 data could be added)

  79. Brian Burke says:

    Good question. The recency of the data is far less important than the size of the data set. If we constrain ourselves to looking at just 1 (or less) season of data, we're going to have a very large sample error. There just aren't that many 4th and 2s where teams go for it.

    To have a reliable estimate, we need to look back further and gather more instances of that situation. Plus, there has been no rule change or development in the NFL that would make 2009 any different than other seasons in recent history.

  80. Anonymous says:

    By your own numbers the 60% success on 4th and 2 is by run or pass. Your own numbers drop that to 56% for pass, 63% for run. The call was not a run by Belichick, it was a pass. Wouldn't the pass % be more exact? Why give him the benefit of the doubt for a run when he didn't call a run???

    Also, this is assuming all things are equal. Peyton will drive the colts to a win no more often than Jamarcus Russell, the Colts D is no better than the Lions D.

  81. Anonymous says:

    As for the no rule changes to make this season different from the past entire decade...

    2004 pass interference re-emphasis

    QB protection that has influenced sack totals.

    defensive radio communication

    no longer allowing a team to reset the game clock for crowd noise (in a noisy dome for example...)

    Down by contact rule changes

    Horsecollar rule changes

    peel back blocks illegal

    blindside blocking rule changes.

  82. Anonymous says:

    Just curious, what would the confidence interval be for just sampling this year thus far instead of the past 5?

  83. Brian Burke says:

    Hmmm. Those changes I guess could have an effect. Point taken. I'd be surprised though. They haven't changed the average gain for either runs or passes (still 4.2 and 6.1). Turnover rates are still the same compared to past years too. So I'd really doubt there is any notable difference, but you're right. We can't rule it out yet.

  84. Gordon says:

    Brian,

    Just as a minor note, I think the equation
    (0.60*1) + (0.40*(1-0.53)) = WP

    Could be more simply and intuitively written instead as probability of Pats LOSING. Two things and only two things have to happen: (1) fail to convert and (2) Colts scoring TD from Pats 28. So joint probability of Pats LOSING is simply (1-0.60)*0.53.

  85. Jeff Clarke says:

    Let me just say whoever said a TD was worth twice as much as a first down in that situation was just blatantly and obviously wrong.

    NE's WP if they failed to get the first: 47%

    If they got only 2 yards: 99%

    If they got a TD: 99.99%

    The two yards were worth 52%

    The extra 60 yards would be worth 0.99%.

    It really was all about the two yards.

    The point has been made repeatedly that this lowered NE's conversion percentage because the defense had no reason to protect at all against the big play.

    This is obviously true but the corollary is also true. NE had no reason to try for a big play. If under a normal 4-2 a team had to choose between play A (a 65% chance at 2 yards and only 2 yards) and play B (a 35% chance at a game changing TD), they would pick B a healthy percentage of the time.

    I sort of feel like the fact that neither the offense or the defense considered a 40 yard play cancels each other out. I think the 2 point conversion numbers might be a good place to start. You still need to make a couple adjustments. #1, you need to strip out all the failed 2 point conversions that were really desperation attempts after an extra point bobbled snap. #2 you need to adjust for the fact that this offense vs. this defense had a higher prob of succ than league average 2 pt conversions (which are attempted the majority of the time by the losing team).

    I still feel like 60% was the correct prob of success for the 4th down play. I could be probably be convinced it was low to mid 50s. Anybody that thinks it was under 40 is clearly insane.

  86. Gordon says:

    Jeff,

    I don't think they cancel each other out. I assume it's easier to defend only against short yardage. Some empirical support for that intuitive assumption can be seen in the lower conversion rate inside the 10 (blue line) vs. outside the 20 (green line) at http://www.advancednflstats.com/2009/09/4th-down-study-part-3.html

  87. Sam says:

    By the way, an interesting scenario played out in the Baltimore - New England game just a few weeks earlier. With a little over 5 minutes left and trailing by 6, Baltimore went for it on their own 45 on a fourth and short one. They failed to convert. New England ended up with a fourth and four at the Baltimore 39 with 3:32 left and opted for a punt, which resulted in a touchback. Baltimore then drove all the way to the New England 14 before the drive stalled. Just some food for thought.

  88. Anonymous says:

    After looking at the replay, it looks like Welker (left slot) runs a crossing route left-to-right, Faulk (right slot) runs an out route, and Moss (right sideline) gets jammed by his defender, stands around and clogs Faulk's route.

    Moss probably should have just run straight upfield, giving Brady either the option to go over the top or throw to Faulk, who may have been able to escape his defender.

    The Patriots didn't seriously consider going farther than a few yards downfield, but they probably should have.

  89. Jeff Clarke says:

    You're right. They don't completely cancel each other out. I just don't think it makes the enormous difference that some people seem to think it does. The advantage that the defense gets is at least partially mitigated by the higher probability the offense gets...


    That Baltimore game story is interesting. I wonder if Belichick second guessed himself on that one. He very well might have said something along the lines of "Next time I'm in that situation...."

  90. Gordon says:

    Brian,

    In my comment above I state that my friend's data for 2009, with a sample size of 303, is for 4th and 3 OR LESS. So my questions are (1) Is 303 a sufficient sample size for our purposes, and (2) Is 4th and 3 OR LESS a good enough surrogate for 4th and 2 (either sufficiently close or likely a lower percentage than that for 4th and 2)? If the answer to both is "yes", aren't we better off using that 2009 figure vs. using data for the past several years or decade?

  91. Anonymous says:

    Gordon,

    The standard deviation for a sample size of 303 would be sqrt(p*(1-p)*303) / 303, where p = the approximate probability. Plugging in p = 0.5 will be fine; it actually doesn't change very much as you vary p. We get that the error in that stat for a sample of 303 would be 0.0287, or 2.9%. Two standard deviations is what most people use for a confidence range, so the figure for 4th-and-3 over 303 samples has a confidence range of +/- 5.8%. So your friend's stat would be 45% +/- 5.8%.

    This is actually very surprising to me. I would have expected the 4th-and-3-or-less stat for 2009 to be higher than the decade-long 4th-and-2 stat because (1) it would be overpopulated by 4th-and-1s and (2) almost all the rule changes over the decade have favored the offense.

  92. Gordon says:

    Thanks "Anonymous". So it seems that using the 60% figure that Brian uses for Pats conversion is too high, and we should at least bring it down to 51%, and adjust from there based on other factors as one sees fit. If average is between 0.39 and 0.51, and if we assume Pats have, say, a 20% better than average chance, then we'd get range with low end of 0.39 X 1.2 = 0.47 and a high end of 0.51 X 1.2 = 0.6 , correct?

  93. Anonymous says:

    mr. burke I apologize for impugning your probabilities calculation chops. i was misled by that cockamamie goforit pie chart in the 11/17 boston globe story, which seemed to say that 0.81 of the drives from 30 yds led to a td, statistically speaking. Your calculation based on 0.53 success after getting the ball there is ok. sorry, mea maxima culpa.

  94. Anonymous says:

    test

  95. Anonymous says:

    Good question. The recency of the data is far less important than the size of the data set. QUOTE: "If we constrain ourselves to looking at just 1 (or less) season of data, we're going to have a very large sample error. There just aren't that many 4th and 2s where teams go for it...."

    I love it how you purport to be all logical with your analysis of this and then someone challenges you with 300 data pts and you claim it's a large sample error.

    Have you heard of standard deviation? This is probability class 101. 300 data pts is certainly plenty of data pts to form the type of statistical estimates that you based your entire analysis on..

    QUOTE: "To have a reliable estimate, we need to look back further and gather more instances of that situation...."

    Sez who? You? Why because the recent data doesnt support your conclusion you need to go back further.

    That's real logical.

  96. Anonymous says:

    Gordon:

    Your estimates would be reasonable.

    I'm still puzzled that the 4th-and-3 or less rate from the first half of 2009 would be so low, though. Brian has 9 seasons of data vs. 1/2 a season (18 times as many games), so one figures that he'd have around 18/3 = 6 times as many plays in his 4th-and-2 figure. Maybe only 4x as many if we think that 4th-and-2s make up less than 1/3 of the 4th-and-3-or-less set.

    It could be just a statistical anomaly, it could be that one or the other data set is biased, or it could be that the defensive rule changes really make a difference.

  97. Edward Lee says:

    Gordon:

    I just checked NFL.com's stats for each team, and assuming I'm not an idiot, NFL teams are 146-for-304 (48%) on *all* fourth downs in 2009 (up to and including Thursday's MIA-CAR game).

  98. Brian Burke says:

    Yeah, that 2009 4th down data someone posted is way off. If we just take 2009 (which I still think is a bad idea) for all 4th and 3 or less, the conversion rates are 107 for 183 (58.5%).

    That includes attempts inside the red zone and on the goal line, which would be more difficult than the 4th down we're discussing. There have only been 27 4th and 2s outside the red zone this year.

    Sorry to sound all 'logical.'

  99. Brian Burke says:

    One other thing. The data most people see, like on nfl.com or espn.com, may not include 4th downs converted via penalty.

  100. Dr. Duru says:

    Your two posts did not address whether the nine percentage points is significant enough to warrant going for it. Also, what about factoring in cost of losing vs the benefit of winning? Given the playoff implications, I think you assign a very high cost to losing here. In other words, a win gives the Patriots marginally improved playoff prospects from an already good position. A loss causes a significant hit to their playoff prospects. Of course, this is subject to debate (more statisitcal analysis).

  101. Dr. Duru says:

    Sorry! I just realized that the cost of losing vs winning is probably immaterial in your analysis! You just want to increase the chance of winning, period.

  102. Anonymous says:

    Dr. Duru,

    First, everyone knows that the statistics discussed here are estimates. We'll never know the "true" probability of winning via going for it on fourth down, but neither will we ever know the "true" probability of winning via punt. There is no reason that punting should be considered the "safe" option. That being said, the 9% edge that going for it is pretty big, as you can check if you try to manually adjust the probabilities that go into the computation.

    Second, the effect of a win or loss doesn't change whether Belichick goes for it or punts (unless you put a lot of stock in all the pop-psychological arguments about undermining his defense, etc).

  103. Gordon says:

    Edward and Brian,

    Thanks for the correction re: 2009 4th and 3 or less (assuming you're correct). As I said, the figure I presented was from a friend. I didn't verify it, so if it's incorrect, my apologies.

  104. Gordon says:

    Brian -- Can you please provide a link to your source for that 4th and 3 or less data for the NFL in 2009?

    Edward -- Can you do the same for your source for ALL 4th attempts in NFL in 2009?

    thanks

  105. Edward Lee says:

    Gordon:

    http://www.nfl.com/teams/statistics?team=ARI

    Repeat for all 32 teams and add up the results (which by now have been updated to include today's games).

  106. James says:

    "Also, what about factoring in cost of losing vs the benefit of winning? Given the playoff implications, I think you assign a very high cost to losing here. In other words, a win gives the Patriots marginally improved playoff prospects from an already good position. A loss causes a significant hit to their playoff prospects."

    If you're concerned that the cost of losing is greater than the benefit of winning, the appropriate decision is to not play the game in the first place. Once you show up and the game starts, you're either going to win or lose. Every decision that increases your chance of winning by 1% also decreases your chance of losing by 1%, and vice versa.

    Obviously, there's plenty of room to argue about how to give yourself the best chance of winning, but the importance of the game is entirely irrelevant.

    (The importance of the game does have a bearing on what the media backlash will be like after a loss, so an argument can be made that the "conventional wisdom" is the better choice in terms of protecting the coach's reputation. That's what's so infuriating about all this.)

  107. Gordon says:

    Edward -- thanks.

  108. Anonymous says:

    Brian - You mention stats that most people see on nfl.com and espn.com may not include 4th downs converted via penalty. On the other hand, do the stats in which you rely on, include offensive penalties on 4th down conversion attempts as failed attempts? Should they be included and if so, is it possible to get an adjusted sample?

  109. Brian Burke says:

    The data I have comes with a code for any play that makes a first down. It says P if it was a passing play, R if it was a run, and X if was a penalty. So yes, 1st downs by penalty are considered in the data, but only if the penalty happened on the play. Penalties between plays do not factor in.

  110. Gordon says:

    Brian,
    Re: penalties, we have 4 categories:
    1. Convert without pentalty on Defense.
    2. Convert with penalty on Defense.
    3. Fail without penalty on Offense.
    4. "Fail" with penalty on Offense.

    Including penalties on Offense in the final figure, thus including it as a factor in the weighted average of conversion percentage (probability) artificially decreases the attractiveness of going for it (not necessarily lower than punting; I just mean directionally). "Failing" due to Offensive penalty means (at worst) punting from 5 or 10 yards back, meaning Colts ball on 32 or 27, which would mean much lower probability of Colts TD than Colts starting on Pats 28 after an actual failed attempt. In other words, each incident (data point) of "failing" due to offensive penalty shouldn't be factored in with nearly as much weight as each incident of actually failing (without penalty) and turning ball over on the spot. In effect, counting the "failing" due to Offensive penalty equally erroneously implies that there is no difference in winning probability between Colts starting on Pats 28 and Colts starting on either their own 32 or 37.

  111. Gordon says:

    Typo: I meant to say punting after Offensive penalty would mean Colts ball on own 32 or 37.

  112. Gordon says:

    If I set up this Google doc correctly and pasted the link correctly, at this link http://spreadsheets.google.com/ccc?key=0Am3TwX_gP-RCdHk1dDhFNmlNQ3FjQ1EyaWxxY29jMHc&hl=en is a spreadsheet tool I've created that anyone can use to plug in your own assumptions (and tinker with them) to assess Punt vs. Go For It.

  113. Gordon says:

    Additional notes re: penalties:

    - Converting with Defensive penalty should be considered equal to converting without Defensive penalty, because result is the same.

    - Optimal equation for WP in Go For It scenario would consider probability of converting, probability of failing without penalty applied to probability of Colts scoring from Pats 28, and probability of "failing" with Offensive penalty applied to probability of Colts scoring from own 32 or 37 (corresponding to penalty of 5 or 10 yards). And yes, after a "fail" with Offensive penalty the Pats could go for it again from 5 or 10 yards back instead of punting, but assuming rationality they would only do so if doing so provided a higher probability of winning vs. punting, which is why I said in previous comment that "at worst" (from Pats perspective) a "fail" with Offensive penalty means Colts ball on own 32 or 37.

  114. Gordon says:

    To elaborate on my comment above and lay out the algebra and an illustration:

    If "failing" with (i.e., due to) offensive penalty were not a possibility, for Pats to LOSE after going for it, two things would have to happen, (1) Pats fail to convert, and (2) Colts TD starting from Pats 28. So, if P (fail) = 0.42 and P(Colts TD from Pats 28) = 0.53, than P(Pats losing if go for it) = (0.42)(0.53) = 0.223.

    But since "failing" with offensive penalty IS a possibility, and would result in (at worst for Pats), a punt from back 5 or 10 yards (let's just say from back 7.5 yards assuming even number of 5 yard and 10 yard offensive penalties), we have to separate out that possibility.

    Suppose 1 in 7 failures to convert are with offensive penalty. Then P(failing to convert without offensive penalty) = 0.36 and P("failing" to convert with penalty) is 0.06.

    And let's suppose that P(Colts scoring from own 34.5) = 0.32

    Then P(Pats losing if go for it) = (0.36)(0.53)+(0.06)(0.32) = 0.21.

    So properly separating out actual failure to convert from "failure" due to offensive penalty gives us probability of Pats losing if go for it of only 21%, rather than the erroneously higher 22.3% we get if we inappropriately treat "failure" due to offensive penalty as if it has the same consequences as actual failure (consequences would be quite different: Colts ball on own 34.5 vs. Colts ball on Pats 28).

    So a proper treatment of actual fail vs. "fail" due to penalty results in going for it being more attractive for Pats (lower chance of losing) than if we inappropriately lump the two scenarios together.

  115. Gordon says:

    Brian,
    It sounds like you are saying that conversion via Defensive penalties are included in your data. Are Offensive penalties included, too (i.e., offensive penalty during the play such as holding)?

  116. Brian Burke says:

    Yes, offensive penalties are included too as long as they occur during the play. Pre-snap penalties are not included, and any other penalty between plays, because there is still the option to punt following the penalty.

  117. Gordon says:

    Brian,

    thanks. Per my explanation above, ideally we should strip out "failed" attempts due to offensive penalty and apply that percent of total attempts to the probability of Colts scoring after Pats punt from 5 or 10 yards back (weighted average or just 7.5 yards back if probably close enough), rather than lumping in that category of results with actual failed attempts and applying it all to probability of Colts TD from Pats 28.

    Depending on the data and on one's assumptions, this more appropriate approach could either make a significant difference or not.

  118. Anonymous says:

    Hey another long pass on short yardage during mon nite football.

    I think it was 1.09 left in 3rd quarter and they threw a legitimate long pass on 3rd/one.

    Dont tell me that no one covers long passes on short yardage plays. It happens every week in the NFL.

Leave a Reply