Thomas Bayes Would Approve of Seattle's Defensive Tactics

The following is a guest article by Gary Montry, a professional applied mathematician. Editor's note: Gary uses net yardage as the measure of utility, and we might prefer something like EP or WP, I think the general point of the article stands, and its strength is in the construction and solution to the problem. It's also a great refresher on conditional probabilities and Bayes' theorem.   

Last week a WSJ article about the Seahawks' defensive backs claimed that they "obstruct and foul opposing receivers on practically every play."  I took a deeper look in to the numbers and found that as long as referees are reluctant to throw flags on the defense in pass coverage (as claimed in the article), holding the receiver is a very efficient defensive strategy despite the risk of being penalized.

The following is an analysis using the concepts of expected utility, expected cost, and bayesian statistics.

The reason defensive holding is an optimal strategy comes down to one word. Economics. The referee's reluctance to call penalties on the defensive secondary is analogous to a market inefficiency. The variance in talent on NFL rosters, coaching staffs, and front offices between the best and worst teams in the league is probably very small. Successful teams win within a small margin. Seattle has found a way to exploit a relaxation in marginal constraints within the way the game is called that their competitors have not, and turned it into a competitive advantage.

If you think about committing a penalty in the same way as committing a crime, the expected utility is essentially the same. The expected utility (EU) for defensive holding is (opponent loss of down due to incomplete pass - probability of being penalized x cost of penalty). In other words, EU is the benefit of an incomplete pass minus the cost of the penalty times the probability of getting caught.

Denver and Seattle are facing off in the super bowl, Imagine that the game is happening right now, it's in the third quarter, the game is close and Denver's offense has the ball on third and 7 and are not in field goal range. Manning averages about 12 yards per completion. Denver's utility from a completed pass is 12 yards and a first down, an incomplete pass results in a punt. The difference between success and failure on this play for Denver is huge.

Manning completes 68% of his passes. Denver's EU for this play is 0.68 x 12 or 8.3, (this is the same thing as yards/attempt). On average a pass play in this situation will move the chains for Denver.

*For now we will make an assumption that if the Seattle DB covering the receiver that Manning is throwing to holds, the result will be an incomplete pass.

If Seattle is flagged for defensive holding, Denver gains 5 yards as a result of the infraction and a fresh set of downs. However, that's 3.3 yards less than the expected outcome of a Manning pass. Even when Seattle is flagged for holding they still come out ahead by committing the penalty. Denver get's the first down but only 5 yards rather than the expected 8.3.

But, as Mike Pereira points out in the article, Seattle's DBs doesn't always get called for holding. We have no way of knowing exactly how many times Seattle got away with holding but we do know that the secondary was flagged 10 times for defensive holding during the regular season. 541 passes were thrown against Seattle during the regular season which gives us a rough estimate of the probability of Seattle being flagged p(F) for holding on an any individual pass play, 10 flags / 541 attempts is a little under 2%. (It is important to note that this is without regard to whether holding actually occurred or not, it's just the overall rate of Seattle being penalized for defensive holding)

What that means is that 2% is not the conditional probability of being flagged given that a penalty was actually committed.. It's the unconditional probability of a flag being thrown for defensive holding on any play. The conditional probability (the probability of being flagged GIVEN a penalty was actually committed), p(F|P) , could be derived from Bayes theorem, however we would need to know the actual defensive holding penalty rate including holding that isn't flagged which we don't know. (we'll get around that)

Seattle's expected cost (EC) for committing a defensive holding penalty can be approximated as (cost of penalty x probability of being flagged), or in the case of defensive holding (-5 yds x p(F|P)). Here p(F|P) is the conditional probability of a flag being thrown GIVEN that the player committed a foul (in a perfect world this would be 100%. A penalty would be accessed every time a player commits a foul).

Now we don't know exactly how often players get away with defensive holding, (in the article Kevin Gilbride estimates Seattle's DBs do it on nearly every play) but let's be a bit more conservative and say Seahawk DBs hold 60% of the time (p(P) = 0.6), we then only need to know the conditional probability p(P|F) of a penalty being committed GIVEN that a ref threw a flag (this is the true positive rate), It's important to note that p(P|F) and p(F|P) are NOT the same. As a shot in the dark guess I'll call p(F|P) 90% (that means that 10% of flags were thrown on plays where there was no foul, in other words the false positive rate). Now we can compute the probability of a DB being assessed a penalty GIVEN that he did in fact commit that penalty, p(F|P). This is done by applying Bayes theorem.

Bayes theorem:
p(F|P) = p(P|F) x p(F) / p(P) = 0.9 x 0.02 / 0.6 = 0.03

Somewhere in the neighborhood of 3%. That's the probability that a defensive back will actually get flagged for holding the receiver when they foul the receiver 60% of the time.

To estimate the Seahawks expected utility of holding, we subtract the EC of the penalty from Denver's EU of a pass play.

Seattle's EC = 0.03 x -5 yds for an average loss of 0.15 yards.if DBs hold
Denver's EU = 8.3 (Manning yards/attempt) if the DB's don't hold

Seattle's EU = -(8.3 - 0.15) = -8.15 yards (negative value to distinguish offensive yards gained from defensive yards prevented)
Denver's EC = 0 yards (incomplete pass) + 0.15 (Seattle's EC) = +0.15 yards

In summary:
Denver Passes, DB's don't hold: Denver gains 8.3 expected yards
Denver Passes, DB's do hold: Denver gain 0.15 expected yards

Seattle may get a boost in the EU of defensive Holding during the super bowl. The article also states that there was a 40% drop in penalties during the wildcard playoff games with respect to the regular season games, which if true would lower Seattle 's expected costs for defensive holding even further. But because that sample size is so small, I would elect not to adjust for any post-season penalty calling trends and keep the numbers from the regular season.

What if Kevin Gilbride's estimate that Seattle's DBs do it on nearly every play?

If Seattle's holding rate were 95%:
p(F|P) = 0.9 x 0.2 / 0.95 = 0.2

Seattle's EC = (-5 yard) x 0.2 = - 1 yard
Seattle's EU = -(8.3 -1) = -7.3 yards

Denver's EC = 0 yards + 1 yard = + 1 yards

It's still a pretty sweet deal for Seattle.

There is a caveat. Defensive holding also gives the opponent a automatic first down. So the utility (and cost) of holding on first and ten is a different than holding on third and two.

An optimal pass protection strategy would have the defensive backs commit holding penalties on every third down and on plays when Denver needs less than 8.3 yards for a first down. Or as Kevin Gilbride states it, on nearly every play.

It may be at best morally ambiguous, but from a tactical standpoint, it's an extremely effective pass defense. As long as ref's are reluctant to call penalties on every play, the gains far outweigh the penalty. That probably wont't happen though, because the constant interruption in play would outrage most fans. How long would it take to play the Super Bowl if the game had to be delayed 95% of the time Manning threw the football?

Forty-Niners' (and Broncos') fans would call this cheating, and they certainly could make a strong case as technically it is, but through the filter of statistics and economics, it's extremely efficient and actually quite brilliant.

  • Spread The Love
  • Digg This Post
  • Tweet This Post
  • Stumble This Post
  • Submit This Post To Delicious
  • Submit This Post To Reddit
  • Submit This Post To Mixx

22 Responses to “Thomas Bayes Would Approve of Seattle's Defensive Tactics”

  1. Unknown says:

    If Seattle's holding rate were 95%:
    p(F|P) = 0.9 x 0.2 / 0.95 = 0.2

    You're missing an extra 0 in the above equation, it should be:
    If Seattle's holding rate were 95%:
    p(F|P) = 0.9 x 0.02 / 0.95 = 0.02

    but of course if the Seahawks hold on 95% of the plays instead of 60% of the plays, their p(F) is probably higher than 2%...

  2. John Hirschauer says:

    very similar to catchers garnering immense value through framing pitches

  3. Brian Burke says:

    Regrettably, I think how the refs call this game will have a much bigger impact on the outcome than it usually does, and will probably determine the outcome.

    Standby for a 'WPA for each penalty' post.

  4. Anonymous says:

    This line

    p(F|P) = p(P|F) x p(F) / p(P) = 0.9 x 0.02 / 0.6 = 0.03

    well, i don't know how to say it politely, is ridiculous. Does anyone who has watched an NFL football game think that a holding play actually gets called only 3% of the time?

    Other than that, great article. Put that value up to about 90% of the time, because while sometimes a player gets away with a hold, the refs are much better than to miss it all the time.

    The original assumptions are flawed, but there are also other assumptions made. The broncos have more than one receiver.
    1) Is the defense holding the 2 wrs, the TE, and the RB on each play? (inconceivable that this situation would not always draw a flag)
    2) if only one receiver is held per play, manning has several more options to complete a pass. Assuming a certain incompletion on a hold is invalid.

  5. Anonymous says:

    Also, the use of Bayes Theorem seems a bit of a name drop. All the author did is assume seattle held on 324 plays (60% times 541 passes) and got flagged on 10 of them. 10/324= 0.03. Not really necessary to invoke Bayes at all.

  6. Unknown says:

    @Brian - Been working on WPA penalties post. In last year's Super Bowl, Baltimore gained 0.18 WPA in penalties (0.14 came in the 4th quarter); in the first NE/NYG Super Bowl, the Patriots gained 0.20 WPA (and somehow still lost). I agree that penalties will play a huge role in this SB as well, especially considering Montry's article.

  7. Anonymous says:

    Besides that the article has major flaws (not every pass play in which holding occurs falls automatically incomplete; there are pass plays when no defender ever touches the receiver like on screens), the basic statement seems much likely true (big payoffs for cheating instead of defending by the rules). We can only hope that SEA doesn´t come away with all the cheating they will do. The best possible outcome would be that the most flagrant holdings and interferences will be called, thus leaving DEN in good field positions.
    If SEA will prevail it would be another low point in the most recent history of SBs decided by bad officiating (like last year, PIT-SEA, SL-NE, PIT-AZ).
    Not talent decides the outcomes, but bad officiating and cheating in all kind of ways.

    Karl, Germany

  8. Unknown says:

    You can't use the average yards per pass when making the statement "Committing defensive holding on plays when Denver needs less than 8.3 yards" is optimal. you need to use the medium yards per pass. Taken to the extreme if that 8.3 average is arrived by throwing an 83 yard pass 10% of the time, and 0 yard pass 90% of the time you optimally would never commit holding.

  9. Anonymous says:

    indeed Karl.

    the author assumes 324 holding plays by seattle, but they only had 215 incomplete passes all season.

    not to mention, this does not even mention pass interference calls.

  10. Anonymous says:

    I would only interject one thing: IMHO there are 2 QBs ( Manning & Brady) in the league where giving them a fresh set of downs or a replay of a down is actually strategically a worse option...they are that good...why would you ever want to erase a play from Manning or Brady that didn't work or "do the job"? Honestly, I only see a few exceptions to this i.e. penalty yardage resulting in being out of field goal range or if they are in "4-down territory"...

  11. Dragon Pie says:

    Are you going to do an analysis on the similar claim that Denver commits offensive pass interference on nearly every play?

  12. Anonymous says:

    Dragon Pie,
    No, And the reason is that offensive holding is flagged much more consistently than defensive holding, so the probability of being flagged for offensive holding is substantially higher. Combine that with the penalty for offensive holding (-10 yards) which is a devastating penalty. I would think that the expected cost probably outweighs the expected utility.

  13. Wizard says:

    interesting article and some interesting comments. first, as somewhat of an aside, I am surprised to hear the word 'cheating' a few times. perhaps it was said to make a point, but I would never think of that as cheating. you do what you can get away with. let me clarify. no different that getting the call on the outside corner (what is a pitcher supposed to do, stay away from that part of the plate lest he feel guilty for cheating?)

    also, I agree that some of the assumptions in the original article are absurd. the referees do not miss THAT many calls. but further, there was some namedropping. Bayes theorem was not needed to make the point here. finally, I disagree with the last statement that offensive holding is flagged much more consistently than defensive holding. did you ever notice how often an offensive player will go for a ball, and interfere with the defender and there is a non call, but on that same play, ask yourself 'what if the players were reversed', (the receiver was on defense and defender on offense), you can answer your own question. 'yep, they would have called it'. I do agree that players get away with defensive holding more often in the playoffs, and in that sense it is less consistent. . but when comparing when it should be called as opposed to when it is is called, I think in general refs are more reluctant to call offensive pass interference, super bowl xl notwithstanding.

  14. Anonymous says:

    Really THE TWO glaring issues, already identified, but worth amplification are 1) Holding results in an incomplete pass, a) That's just not the case observationally, often a holding call is made and the passer will make a completion to another receiver on the route, b) You cannot claim 60% holding rate unless the completion percentage of opponents is <=40% and suggest that holding results in incompletions 2) The use of average per attempt versus median is actually a big issue, its probably easier to appreciate In The context of a running play. Suppose encroachment causes running plays to get blocked up and not gain yardage, then you should've always done it against Adrian Peterson last year, right? But no, his median yards per carry was well below his average and you would risk trading 5.6 for 5.0, its 5.0 versus his lower median.

  15. Anonymous says:

    Without reading all these posts, I have a feeling (just a feeling, just an opinion) the refs will call this game closer than most others, increasing the probability of defensive holding calls - particularly since this has been pointed out publicly here.

    If we assume there are several early holding calls, the Seahawks would be wise to alter their strategy, which could favor Denver significantly. Wondering if that analysis has been done?

  16. Reilly Pinkelnwert says:

    I normally wouldn't want to nit-pick against the nit pickers, but I find this subject interesting. First of all, people are complaining about valid but somewhat pointless statistical issues in this article, for which there is no alternative but to speculate based on the data we actually have at hand. Secondly, people seem to be overlooking the real gist of this article, which is "Would a referee being willing to throw a flag every time he sees a penalty?" The idea that you could overwhelm a referee with constant penalties, and cause them to just let the majority of a team's penalties slide, is probably true, and an interesting approach.

    On a related subject, let's consider the cornerbacks that we are discussing here. You have Richard Sherman, Brandon Browner, and to a lesser degree Byron Maxwell. So, that is a 5th round pick, an undrafted player, and a 6th round pick. What are the probabilities of one team being this much better at finding superior cornerback talent so late in the draft? That their respective 40 times are 4.54, 4.63, and 4.43, which are rather below average (except for Maxwell's 4.43), is one red flag. That their times in the short shuttle and 3-Cone drill were also markedly below average, would certainly make me wonder how they are proving to be so successful at covering people. Then you consider their size, where the smallest one is 6'1" and 207 pounds which is still significantly above average (but nothing compared to Browner's 6'4" and 221#), and along with these accusations of holding, it starts to seem a bit suspicious, doesn't it? It would make perfect sense for slower, less agile, but extremely large corners, to just pummel their opponent, rather than attempt to cover them. Either that, or people somehow think that Pete Carroll is a cornerback whisperer, who somehow is able to identify cornerback talent better than anyone else in the history of the league. Hell, they also have the 6'1", 220#, DeShawn Shead waiting in the wings, to be their next monster sized corner. Coincidence?

    None of this is meant to be a criticism of the Seahawks, because I find their approach to be very interesting. Still, if people don't think there is something fishy about it, then I don't know what to say. Either way, I enjoyed this article.

  17. Anonymous says:

    Reilly: in my opinion, the answer is yes. If players keep making obvious penalties, the refs will keep calling it. It seems ridiculous to think a ref would start to let it go just because it happened a lot.

    Not to mention, even a handful of penalites like this would be extremely damaging to the penalized team. If you give the broncos 5 free first downs, you will not win the game. It would be a disastrous to attempt an 'intentional penalty' strategy.

    I agree, it's an interesting point. too bad the article is poorly written.

  18. Anonymous says:

    I often read this site for objective football analysis, and of course stumble across this horribly biased view of the Seahawks. The assumption that the Seahawks hold on 60% of their defensive plays is FAR OUT OF LINE with what the WSJ reports, and Kevin Gilbride's article is not based on an unbiased film review (remember he is trying to say it wasn't his fault that his team got shut out to position himself for his next job, and preserving the pride of his INT prone QB and soft WRs who don't fight for the ball).

    I'd refer you to two articles, the first is a film review of bot postseason games for the Hawks and at most there are 17 total penalties in 2 games, but many of those are borderline calls, were inconsequential to the play, or helped the offense: http://thebiglead.com/2014/01/30/seattle-seahawks-defenders-how-often-do-they-really-commit-fouls/

    The WSJ reports two separate film reviews that state Denver committed illegal picks that were not flagged on 20 plays during the regular season (and didn't assess for any other form of missed penalties on offense of defense), and references another film review that estimated (projected to 16 games from a 5 game review) the Hawks should've been called for up to 26 more total penalties on the season (not limited to OPI/holding/illegal contact), 16 of which would;ve been on the secondary for illegal contact (holding or OPI). So the Seahawks got away with 1 defensive illegal contact per game... not exactly 60% of their plays.

    So, when it comes to "missed penalties", it's likely that there were equal numbers for each team over the course of the season:
    http://online.wsj.com/news/articles/SB10001424052702303743604579350864192256256?mg=reno64-wsj&url=http%3A%2F%2Fonline.wsj.com%2Farticle%2FSB10001424052702303743604579350864192256256.html

    Starting with a base assumption that Seahawks commit penalties on 60% of their plays... this work is a joke of a joke and shows where statistical analysis can go so horribly wrong that it skews the conversation far from the truth.

  19. Anonymous says:

    Is there even a positive correlation between defensive stats and DPI penalties called?

  20. Anonymous says:

    This is terrible. You start with an assertion with no basis in reality and then add statistics to it. You know what really happens when opponents complain about Seattle's defensive tactics? Officials throw some flags for defensive holding and an equal number for offensive pass interference. Receivers push off on nearly every play, so holding is a counter tactic.

  21. Anonymous says:

    This on par with the eugenics research of the 1920's.

  22. Jason Drake says:

    Reilly Pinkelnwert -- Yes, there is a better explanation.

    The 2013 Seahawks applied plenty of pressure up front, as they were 7th in the league in sack percentage (7.7%).

    Free safety Earl Thomas is first-team All-Pro, and strong safety Kam Chancellor second-team All-Pro. Neither is in position to hold receivers.

    Yes, the Seahawks are better at finding talent. Doug Baldwin, the team's #2 receiver, was undrafted. Super Bowl MVP LB Malcom Smith was taken in the 7th round. Two-time Pro-Bowler Russell Wilson was the sixth quarterback drafted in 2012. An even better illustration of talent evaluation is Chris Clemons, considered a fringe-of-the-roster player, woefully undersized as a DE, and acquired from the Vikings as a throw-in on a trade in 2010. Just as with the oversized corners, Carroll had a plan for exploiting the talents of an undersized defensive lineman, and Clemons was the team's sack leader for three straight seasons.

    And, yes, Pete Carroll IS a secondary guru. He's a former defensive back himself, and coaching the secondary has long been his specialty. When he was DB coach of the Vikings from 1985-89, they were 15th, 7th, 8th, 1st, and 7th in opposing QB rating. (In 1988, those Vikings allowed a historically ridiculously good 2.1 adjusted net yards per attempt).

    When he was defensive coordinator for the 49ers from 1995-96, they were 1st and 6th in opposing QB rating.

    In his last 5 seasons at USC, the Trojans were 19th, 1st, 5th, 14th, and 10th in opponent's passing yards per attempt (per http://www.sports-reference.com/cfb/years/2005-ratings.html). That's counting from 120 schools.

    The Seahawks also lead the league in interceptions and were credited with PLENTY of "PD"s (passes defensed or passes deflected, not sure which it specifically stands for). According to Pro-Football-Reference.com, the league's top 10 defenses (in terms of points allowed) accumulated PD's as follows (pass attempts against in parthesees, percentage of PDs following):

    Seahawks 111 (524) 21.0%
    Panthers 71 (563) 12.6%
    49ers 96 (585) 16.4%
    Saints 74 (507) 14.6%
    Chiefs 112 (592) 18.9%
    Bengals 109 (614) 17.8%
    Cardinals 103 (625) 16.5%
    Dolphins 78 (580) 13.4%
    Colts 88 (536) 16.4%
    Patriots 80 (589) 13.6%

Leave a Reply

Note: Only a member of this blog may post a comment.