Dec 29, 2009

Run-Pass Imbalance on 2nd and 3rd Downs

I've recently been looking at the imbalance in the payoffs for running and passing on first downs. The results suggested that most teams should generally pass more often outside the red zone and run more often inside the 10-yard line. What about 2nd and 3rd downs?

Game theory tells us that when the payoffs for two strategy options are unequal, the strategy option with the higher payoff should be selected more often. As the opponent adjusts to counter the new mix of strategies, the payoff of the favored option will decline while the unfavored option becomes more lucrative. Eventually, the payoffs for both options equalize, and at this point the overall payoffs are optimum. In two-player zero-sum games this is known as the minimax, or more generally as the Nash Equilibrium.

I used Expected Points (EP) to value the payoff of each play. Expected Points measures the net point advantage that the play result gives to an offense. It captures the value of yardage gained and lost, first downs, sacks, penalties, turnovers, and everything else in terms of equivalent point value. The change in EP resulting from a play is called Expected Points Added (EPA).

One of the things EP does not measure is the time value of a play. In situations when a team has a significant lead, the true value of a run includes the time burned off the clock. To a team behind late in a game, pass attempts have more value because they are more likely to stop the clock. For this reason I only include plays in the first and third quarters and when the score is within 10 points. This excludes trash-time plays and plays affected by the clock.

Unfortunately, there aren't nearly as many examples of 2nd and 3rd downs plays for each to-go distance as there are 1st down and 10s. For example, since 2000, between an offense's own 30 and 40 yard lines, there were 4,459 runs on 1st and 10 but only 166 runs on 2nd and 9. For this reason, to get  a statistically reliable estimate of the values of 2nd and 3rd down plays, I had to make a couple compromises. I did not break out plays in terms of field position like I did for first downs. Instead, I grouped plays as either inside the red zone, between the 20s, or inside a team's own 20. In this post, I'll focus on plays between the 20s. I also grouped to-go distances as very long, 10 yards to go, long, mid, short, and 1 yard to go.

Second Down

The tables below list each distance to go for 2nd and 3rd downs. For each combination of down and distance the table lists the proportion of pass plays, the EPA for runs, the EPA for passes, and the difference between the EPA for passing and running. Positive numbers for the EPA difference  indicate an advantage for passing, and negative numbers indicate an advantage for running.

2nd down &...
Pass %
Pass EPA
Run EPA
EPA Diff
13, 12, 11
72%
0.06
0.00
0.06
10
48%
0.15
-0.01
0.16
9, 8, 7
63%
0.11
0.01
0.10
6, 5, 4
45%
0.00
-0.10
0.11
3, 2
29%
0.00
-0.09
0.09
1
21%
-0.07
-0.07
0.00

For second downs between the 20s, passes have been more lucrative. Except for 2nd and 1 plays, which indicate an advantage for neither play type, an offense can currently expect about 0.1 more points for a pass than for a run.

One interesting thing to note from the 2nd down table is the relatively low proportion of passes on 2nd and 10. Notice that for 2nd and 'very long' teams pass 72% of the time, and for 2nd and 'long' teams pass 63% of the time. But for 2nd and 10, teams pass only 48% of the time. Note also that 2nd and 10 indicates the biggest advantage for passing at 0.16 EPA.

This is a curious result, and it's is almost certainly due to the inability of some coaches to randomize. When attempting to be unpredictable , most people (including football coaches apparently) tend to alternate rather than be truly random. Random sequences are far less alternating than most people think. Because 2nd and 10s usually result from incomplete pass attempts on 1st and 10, offenses tend to run more often than the situation calls for. It appears defenses may be be aware of this and are scheming against the run on 2nd and 10.

Third Down

The next table outlines the differences in payoffs for passing and running on 3rd downs.

3rd down &...
Pass %
Pass EPA
Run EPA
EPA Diff
13, 12, 11
92%
0.01
0.32
-0.31
10
93%
0.11
0.08
0.03
9, 8, 7
92%
-0.02
0.19
-0.22
6, 5, 4
88%
-0.02
0.19
-0.22
3, 2
72%
-0.09
0.12
-0.20
1
21%
0.04
0.11
-0.07

For 3rd downs we see the opposite result. Runs are more lucrative in nearly all cases by about 0.2 points per play. This is big, really big. Think of it this way: An offense can expect to improve its net point advantage over an opponent by 0.2 points simply by choosing a certain type of play...a single play. This essentially measures the value of a play before we learn the actual result, and only measures the play choice itself. According to these numbers offenses should run more often on 3rd down. This is roughly consistent with my initial look at 3rd down playcalling over a year ago.

Note again the results for 10 yards to go. Although there is no discrepancy in the proportion of passes vs. runs, the advantage for running disappears, and there is a slight advantage for passing. This could be due to the 'alternation' effect that I mentioned for 2nd and 10, or it could be due to some other cause.

The other thing that stands out to me is the very large EPA for runs on 3rd and very long. The bulk of the advantage from running in this situation comes in the 20 to 40-yard line region of opponent territory. I think this result suggests most teams are better off playing it safe, running for better field goal range rather than passing and risking a sack, which might knock an offense out of FG range, or an interception, which would be even more costly.

Even better, coaches should almost always think of this region of the field as 4-down territory. On 3rd and very long an offense could run for an easy 6 or more yards, then set up for a manageable 4th down conversion attempt. Once defenses respond by playing tougher against the run on 3rd and very long, passes become less predictable making sacks and turnovers less likely.

Just like on first down, runs are more lucrative inside the red zone for 2nd and 3rd downs for nearly every to-go distance. I intend to post full results for the red zone in a future article, but for today I'll just say that the numbers suggest offenses should run on 2nd and 3rd down more often inside the 20-yard line, and inside the 10-yard line in particular.
By Brian Burke

40 comments:

  1. I've noticed the good teams tend to follow your recommended strategies more than the bad teams. For example, the Colts are primarily a passing team between the 20s, but they call a lot more run plays inside the red zone. This seems true with the Saints as well, although I don't have the numbers and haven't seen all their games this year.
    ReplyDelete
  2. "When attempting to be unpredictable , most people (including football coaches apparently) tend to alternate rather than be truly random."

    I've always had a theory for how plays should be called. Teams should decide before a game, what ratio of run/pass plays they want in any given situation. For example, maybe they've decided that they're going to pass 66% of the time on 1st-and-10 or 50% of the time on 2nd-and-5. Then, as each situation arises during the game, the play-caller (usually, sitting upstairs in the booth) actually uses a random number generator (you could literally run this extremely simple app on a cell phone) to determine whether a run or a pass is called.

    This is not the same as randomly picking plays. Once the random generator picks a run or a pass, the offensive coordinator would still be responsible for picking the particular play he wants. But this system would keep coaches from falling into the pseudo-randomizing/alternating trap that you describe.
    ReplyDelete
  3. Alchemist - I believe there are rules in the NFL prohibiting the use of a computer in calling plays. I'm not sure if that rule would be applicable here or not (you could just as easily flip a coin) I just wanted to point out that it was there.
    ReplyDelete
  4. Brian,

    There's a test - which I can't remember the name too right now - that can show you how to group stuff based on how similar they are. I think this test would be good to do on the grouping of plays on 2nd and 3rd down. If I can find it, or remember what it is, I'll let you know. Pretty much it would allow you to group the distance-to-go's in a more statistically sound way, although I'm sure your grouping isn't too far off.
    ReplyDelete
  5. That would be interesting. Thanks.
    ReplyDelete
  6. "Game theory tells us that when the payoffs for two strategy options are unequal, the strategy option with the higher payoff should be selected more often."

    I've seen you post this theory several times, and I just don't believe it's actually applicable to football. A Nash equilibrium is not when the payoffs of the two choices are equal - it's when there's *no gain* to changing strategy (changing the frequency of choosing one option over the other). If the two choices change *differently* depending on how often they're used, then you could easily end up with an ideal situation where the two payoffs are unequal.

    Stated more clearly, the only way you can claim that the current situation isn't a Nash equilibrium is by showing that the outcome improves by changing strategy (running less/passing more) - and I don't think you can even suggest that that's true unless you can show that passing efficiency doesn't decline as it's chosen more (And even then, I don't know how you justify it entirely because choosing passing more is usually done by teams who need to do it).

    I think it's really, really hard to claim that coaches aren't making optimal choices considering they have orders of magnitude more information available as to why their choices were made.
    ReplyDelete
  7. If, all things considered, one type of play has a higher payoff then why not do it more?

    Coaches do not have orders of magnitude more information. They do have a very rich, but very small and very biased data set--their own observations filtered through preconceived notions.
    ReplyDelete
  8. "If, all things considered, one type of play has a higher payoff then why not do it more?"

    Because we don't know the point in which our opponent will wise up. Perhaps throwing just a few % points more often will cause our opponent to play exclusively pass defense, causing our average payoff to decrease. You're assuming that a decrease in pass payoff will result in a corresponding increase in run payoff. What if this isn't the case, ie. our ability to run is only minutely effected by the defense's choice of defensive play?

    And we aren't even sure that our model is correct for defining a play anyway. A lot of information can be displayed by the defense -- packacking, alignment, etc. -- before a play than can change the offense's strategy before a play. On the whole it may appear to be non-optimum, but at the time may have actually been the best decision to make. I remember another poster explaining this fairly eloquently in the past.

    And anonymous is right on his definition of Nash Equilibrium; it's often common for a game/situation to have multiple Nash Equilibrium points with neither being the absolute best situation for a given side.

    Furthermore, randomly choosing plays, without adjusting for opponent tendencies, isn't optimization either.

    Note:(Average payoff isn't the most ideal way to treat football, but you get the point).
    ReplyDelete
  9. Then why ever mix? What's suggested above is that there should be a dominant strategy.

    If you doubt the general zero-sum game theory aspect of run pass, please read my 3rd down article from over a year ago.

    I'm well aware that defenses display information, and there are tendencies to exploit. But given that information, and given those tendencies, we see a pretty clear imbalance of offensive play calling types.

    There is no mathematical way out of equal payoffs at the minimax. For 2-player zero sum games with a linear utility function, the equilibrium will equalize average utility. It's been settled math for half a century. Don't confuse minimax with NE. A NE is a generalization of the minimax. The two commenters above are mixing up their zero and non-zero sum theory.

    If want to debate the appropriate definition of utility, that's fair. But I'm happy with linear net point maximization. I want to have as big a lead as I can before the clock becomes a factor.

    You're right about exploiting opponent tendencies, but that's only because they are failing to properly randomize within the appropriate strategy mix proportions.
    ReplyDelete
  10. Here is a nice summary from Steve Levitt's recent game theory paper:

    "Von Neumann’s Minimax theory makes three basic predictions about behavior in such [two-player zero sum games] games. First, since the player must be indifferent between actions in order to mix, the expected payoffs across all actions that are part of the mixing equilibrium must be equalized. Second, the expected payoff for all actions that are played with positive probability must be greater than the expected payoff for all actions that are not played with positive probability. Third, the choice of actions is predicted to be serially independent, since if the pattern of play is predictable, it can be exploited by an optimizing opponent."
    ReplyDelete
  11. Found the model the other guy had theorized here, and haven't seen it picked at yet (but would like it to be):

    "It isn't an argument with von Neumann, it's a suggestion that the model for football might include information about what the other team is doing (personnel choices). Sorry about not being clear that I was picking at the the model, rather than the result.

    Specific example of how marginal play EPA could matters
    --------------------
    The game has an extensive form where the defense lines up, the offense makes a read as to the type of defense and then makes a play call. The offense three different reads on the defense, run stack, unknown, and pass stack. In that order, the payoffs for passing are 0.36, -0.3, -1, and the payoffs for running are -1, -0.18, 0.09.

    When the defense can be decoded the payoffs indicate calling the other type of play. When it's unknown, it's worse to try passing than to try running.

    Assuming the defense plays 50% run stack, 25% unknown, 25% pass stack, this payoff structure is consistent with the observed datapoints for the Colts. The Colts pass 55% of the time, all the run stack defenses and a bit of the unknown defense, average EPA is 0.30. The 45% runs are called on the pass stacks and most of the unknown defenses, average EPA is -0.03.

    However the marginal play is against the unknown defense, where the payoffs suggest that the Colts should run more, not less, despite the observed average passing EPA being higher than the observed average running EPA.
    ------------

    Just because the value for one strategy is higher than for another doesn't mean it should be called more often, if the choices for the participants are interrelated."
    ReplyDelete
  12. I think the best way to look at this is simply to maximize your yards/play and like Anon said, this isn't necessarily where Pass and Run produce equal results.

    If I run/pass 50%/50% for 3ypp/6ypp then my average return is 4.5ypp

    Suppose I increase my passing so my run/pass is 40/60 and I average 3.25/5.75 for a 4.75ypp which is better

    But if I pass 80% then perhaps the defense adjusts enough so that now reach "equilibrium" of 4.6ypp for either a run or pass. In this case I have achieved an equilibrium, but this is not the best possible result since a 40/60 run/pass split gains an extra .15ypp.
    ReplyDelete
  13. I'd also point out that this should be apparent to anyone who has played poker. Game theory tells us that we should mix up our play with any given hand in order to maximize our overall expectation. Some times this means playing hands in ways that are, in a vacuum, unprofitable, but if raising with 75o causes our opponent to make mistakes against our better hands then it can become a profitable play despite the fact that we may expect to lose money on that particular "play".
    ReplyDelete
  14. That's mistaken on a lot of levels. Poker is usually a multi-player game, and there is variable betting. In football the 'cost' or bet on each play is fixed--1 down.

    If you did have a 2-player poker game with fixed bets on each hand, and it had 2 general strategy types, then having a long-term imbalance between 'bluff' and 'play straight' would indicate you are bluffing too often.
    ReplyDelete
  15. "Coaches do not have orders of magnitude more information"

    Yes, they do. They have the statistics on the results of all of the plays that they ran in practice. We do not.

    The best example of this is in field-goal kicking - if the coach is in any way sane, he probably had some of the stat guys on his staff give him success percentages based on temperature, wind, distance based on the hundreds of field goals the kicker attempted in practice.

    But the same idea applies to plays, too. They certainly know exactly how many yards a run gained on average during practice, and it was run many, many more times than in a game.

    "If, all things considered, one type of play has a higher payoff then why not do it more?"

    Braess's paradox in traffic management, or the Ewing Theory in basketball. Choosing a *worse* option can improve your *better* option such that even though the worse option is, well, worse, its effect on the better option means you should run it more.

    See http://gravityandlevity.wordpress.com/2009/05/28/braesss-paradox-and-the-ewing-theory/

    As a silly model: assume yards/carry are 5 - 2*(runs/total plays).

    Assume yards/pass are 10-(2*(passes/total passes))^2.

    Yards/play are maximized at ~72% passes, at 6.95 yards/play, where the ypc is 4.4, and the yards/pass is 7.92. In fact, here, passes always gain more than runs at all usages.

    I'm assuming you already know this, but are discounting it because you're assuming that play choice is optimal using a minimax strategy. This kind of a dependency is impossible in a finite zero-sum game, but a single play isn't a finite zero-sum game.

    Namely, the von Neumann assumption of "the choice of actions is predicted to be serially independent" is strictly false because the choice on 2nd down leads to a change in the payoff matrix on 3rd down.

    Therefore, there's no reason for the defense to follow a minimax strategy on 2nd down because it worsens their payoff matrix on 3rd down.

    The goal is not to minimize the expected points allowed on a single *play* - it's to minimize the expected points allowed on a single *drive*. And that entirely changes the strategy.
    ReplyDelete
  16. Patrick-Some good points, but I think I can defend each one.

    Regarding magnitudes of information, if I published results based on practice outcomes I bet you'd be among the first people critical, saying practice and real games are not the same thing. I'm analyzing half a million real-game plays since 2000, and still need large groups to get reliable estimates. Coaches are making decisions based on probably the last 10 or 20 plays they remember or according to a preset gameplan.

    I'd be very surprised if many coaches even attempt to do the basic kind of analysis you suggest. Is there anything at all you've seen in the NFL, or heard from former coaches on TV that gives that indication? (Outside of NE?) You actually hear the opposite quite often. Just this past weekend I heard 2 coaches on NFL network stressing how important it is to run a lot because teams that do, tend to win (the old running causation fallacy). These guys are mostly former jocks with a talent for leadership and organization.

    Contrary to your kicking example, Stover used to do his pre-game warm up kicks, then tell Billick his "max range" for the day based on the conditions. Great, so what does max range mean? His 50% range, his 60%, his 1%? They don't know--it was just an intuitive number. There's no statistical analysis involved.

    I think the von Neumann assumption holds because expected points (EP) accounts for the future utility for subsequent down states. True, one of the unstated assumptions here is that we're treating these plays as infinitely iterated games. Almost all applied game theory does this, and there is nothing to suggest that this invalidates the method. All you need to assume is that the "game" will be repeated at least once in the future (the n+1 method of proof), and it becomes valid for infinite iterations.

    There is nothing that says that playing the minimax doesn't optimize net points on a drive. On the contrary, EP as a utility tells us the potential prospective net points not only for the current drive, but subsequent drives.

    If you can show that the EP values for certain situations are different based on the preceding play type (or multiple preceding plays), you might have a point. You'd have to think that defensive coordinators are real big suckers, though.

    The Braess's paradox is not applicable here. That's an n-player, non-zero sum coordination problem. And the Ewing problem is a non-zero sum game too. In that case, the "game" is not the game. The "game" is an n-player question of how to distribute shots to score overall team points. Each player is, in effect, his own agent conspiring to maximize the total utility of his real team. That's completely different from the run-pass question.

    If there is one thing I'd stress to you guys, it's to stop confusing yourself with n-player and non-zero-sum theory.
    ReplyDelete
  17. "Is there anything at all you've seen in the NFL, or heard from former coaches on TV that gives that indication? (Outside of NE?)"

    Yeah, there was an interview with Akers a while ago that gave a lot of information about the stuff they keep track of during the season in terms of kick distance, accuracy, etc. It's fairly detailed. The coach doesn't know most of it, of course, but he doesn't need to. I can try to find it.

    "They don't know--it was just an intuitive number."

    Yeah, intuition based on a lot more information than we have from the games themselves. Which means it has a lot better chance of being right than statistics with limited numbers.

    "If you can show that the EP values for certain situations are different based on the preceding play type (or multiple preceding plays), you might have a point."

    I don't need that. All I need is a skew between the average EP and the most likely down/distance outcome.

    Just look at the down/distance distribution from 2nd and 10 for run vs. pass, and compare that to the most likely down/distance distribution for 2nd and 10 where the play resulted in 0.15 EP gained.

    "I think the von Neumann assumption holds because expected points (EP) accounts for the future utility for subsequent down states."

    If the means/modes of the distributions were equal, I think you'd be right. But because they're not, and, in fact, the EP distributions get really, really widely varying near the goal line, I think there's no way that minimax on expected points on an individual play is the right way to go.

    "And the Ewing problem is a non-zero sum game too."

    I think you're missing the point there - the entire reason that field goal percentage drops with usage is because of the defense's behavior, which means it is a zero-sum game (if you think of it in terms of point differential).

    Although I guess you could also make the argument that that when you alter the shot distribution, defenses would alter, as well, and you'd tend towards a minimax solution. Which I guess is possible.

    That does ignore the fact that players do get tired, however, and it's certainly true that the run/pass choice also could easily have a tiring component.

    "You'd have to think that defensive coordinators are real big suckers, though."

    Watch any player jog off to the sidelines after catching a 30-yard pass to catch his breath. Doesn't require that defensive coordinators be suckers at all. High-payoff plays can eliminate a player that you can use, which means your next play will have a lower average payoff.

    Poof, there's your serial dependence. And that's probably the simplest example - others include the entire existence of play-action.
    ReplyDelete
  18. Finally! A valid criticism of the minimax theory in practice. There could be serial dependence due to player fatigue. Defensive coordinators can see a winded RB and expect a pass.

    However...offensive coordinators see this too, and know what defensive coordinators are seeing. So around again we go.

    You bring up play action, which has already been discussed on previous posts. What this result suggests is that teams are running more often than necessary on 2nd down for play action to be effective.

    On the other points, I'll stand pat.
    ReplyDelete
  19. "Defensive coordinators can see a winded RB and expect a pass."

    I see winded WRs much more often than RBs. A WR can get winded just by running a pass pattern. A RB actually has to have a long run to get winded.

    The offensive coordinators can't do anything about it. Their team's strength is simply reduced. A defensive coordinator would be less likely to blitz, for instance, knowing that the WRs wouldn't be able to get behind his CBs no matter what.

    I should also note that this could explain your observation that you interpreted as "inability to randomize." If most offenses are simply incapable - or less capable - of running a pass, then they would naturally pass less frequently following a pass, leading to alternation.

    The observed high success of passing on 2nd down could easily be a bias - the teams that are best *capable* of passing twice in a row are going to be quite good at it.

    "What this result suggests is that teams are running more often than necessary on 2nd down for play action to be effective."

    I don't see how play action is any different, whatsoever, than players being tired. Fundamentally, it's the same thing - a serial dependence of a *future* play's payoff matrix to a previous play; and to make things more complicated, it could easily be non-linear with frequency.

    A simple model would be a game where the pass payoff to the offense for each previous pass decreases by 10%, the pass payoff for each previous run beyond the first increases by 10%, the run payoff increases for each previous pass defense by 10%, etc. That'd easily push the Nash equilibrium off of a minimax point.
    ReplyDelete
  20. And I'm pretty sure that it is true that serial correlation alone can result in the NE not being located at a minimax point, and non-equal payoffs.
    ReplyDelete
  21. Pat-I don't think there is a problem with serial correlation due to the way the analysis was constructed.

    In Brian's methodology, each down-distance combination is treated as its own independent game. 1st and 10's are treated independently with other 1st and 10's. Second and longs are compared to other 2nd and longs. Third and shorts are compared to only other 3rd and shorts, etc

    Unless you believe that there is a serial correlation between a team's first 3rd and short and then other 3rd and short plays later in the game, then there is no correlation problem. So for example your "winded player" problem isn't a concern.

    Identical down/to go situations are very rarely consecutively repeated. 2nd and 3rd downs, which are analyzed here are never consecutive. They are always separated by several other plays and often by changes of possession too.

    I don't see your logic with winded WRs either. If DCs are less likely to blitz after a long pass thinking it will be a run, then OCs can expect this too, become more likely to throw, and we're back at the general equilibrium.
    ReplyDelete
  22. "Unless you believe that there is a serial correlation between a team's first 3rd and short and then other 3rd and short plays later in the game"

    Of course there is. It's called "memory" - the defense saw what happened on a previous 3rd down, and will correct it. Whether or not it's *possible* for the OC to counter that correction is not guaranteed.

    But that doesn't even matter. The problem is that you're looking just at 2nd down plays, and saying "why don't OCs use the more valuable plays more often!" If play sequencing matters - that is, if "R,P" produces a better payoff matrix on the second play ("P") than "P,P", on second down you may see "R" more often than you'd expect because the OC is 'setting up' the defense. In essence, you "think" the choice is between R and P, but it's really between "RR, RP, PR, and PP" and if "RP" has a high enough benefit in the second play, the "PR/PP" choices can be devalued.

    (Likewise for 3rd down - in that case you might ask "why don't OCs use these super-effective plays more often!" and the answer is "because they need to be set up by less-effective plays previously, and running those too much puts you in a bad position for the combination of the two").

    "If DCs are less likely to blitz after a long pass thinking it will be a run, then OCs can expect this too, become more likely to throw, and we're back at the general equilibrium."

    Why would an OC pass if the payoff matrix is run-dominant? That is, why would an OC pass if passes from that situation are much, much worse than a run?
    ReplyDelete
  23. More details regarding play sequencing (including examples) here.
    ReplyDelete
  24. Pat, your examples make no sense. You've locked the defense into calling the same type of defense for two consecutive plays.

    Also, the fact that Levitt and Kovash found a small amount of serial correlation in play types doesn't support your contention that the sequence of play types is important. It only reinforces what we already know, that coaches are bad at randomizing. Both findings support the same conclusion: Coaches don't play minimax, but probably should.

    The importance of sequence is a flawed argument. Running first is not required for a subsequent pass to be effective. What's required for the effectiveness of a pass is the *threat* of a potential run on the current play. By intentionally sequencing, a coach is allowing the defense to acticipate a pattern, reducing the overall effectiveness of his offense.

    Certainly, running a certain proportion of the time is important to establish that threat, just as passing a certain proportion of the time is important to establish its threat. The required proportions are determined at the minimax, where expected payoffs are equal.
    ReplyDelete
  25. "Pat, your examples make no sense. You've locked the defense into calling the same type of defense for two consecutive plays."

    No, I didn't. You might want to read it again. The math works out such that you can treat it as a zero-sum game with a different payoff matrix, but the defense is completely free to choose run D or pass D on either play.

    Since the defense's choice is independent on both, when you work out the math, the defense's "pass D percentage" variable just merges in the math.

    It's also not necessarily consecutive plays: the example was just "a two-play sequence." Those plays may be consecutive, but they don't need to be. It makes more sense if they're not consecutive (because otherwise the defense would obviously play pass D after every run given the choice).

    If it would help, I can show the results of a simple simulation where the defense's choices are independent. The results are the same, though.

    "The importance of sequence is a flawed argument. Running first is not required for a subsequent pass to be effective."

    Why? If the pass is a play-action pass version of the first run, the play won't be effective unless the defense reacts the same as the first play because they *expect* the play to continue the same way as the first. If the play is a complete disaster if the defense plays straight, then it won't be run until the run is played.

    In addition, the offense might need the "run" version of the play to be run in order to see how the defense reacts such that they can modify it afterwards. The defense has no real way to 'not' react in that same way later because they don't know when the 'new' version of the play will show up.

    "By intentionally sequencing, a coach is allowing the defense to acticipate a pattern"

    There's no pattern. The defense has no idea when the second half of the sequence will be called. The pattern is only known from the offense's point of view.

    "The required proportions are determined at the minimax, where expected payoffs are equal."

    I'm sorry, that's simply not true. The offense is not trying to maximize their output per play. They're trying to maximize their output per drive - and in fact, output *per game*. If there is *any* way to improve the odds of a later play by running a poor play early, they'll take it.

    Assuming that the best option is to play minimax on the output of each play *completely* assumes that there is *no benefit to a play other than its output* - i.e., that there is no serial correlation between plays. The paper *specifically finds* this is not true.

    So either you have to assume that coaches are stupid in *two* ways - they don't play minimax, and they randomize poorly - *or* you can assume that *your one assumption* - that the goal is to maximize output per-play - is wrong.

    The fact that people quite often don't randomize and don't play minimax is completely unimportant. People also screw up assumptions when modeling real-life events quite often as well.

    "It only reinforces what we already know, that coaches are bad at randomizing."

    You're assuming your conclusion! If serial correlation is *a feature of the game* rather than an *effect of the coaches*, then minimax *on an individual play* is not the ideal solution at all.
    ReplyDelete
  26. Also worth pointing out that the assumption of a linear utility function is a bit questionable.

    1) Why would it be linear? It's only linear if the output doesn't change regardless of how often it's used - why wouldn't you expect it to be quadratic or higher, doing better and better as it sees more of the same?

    2) It's semi-testable: if you take teams with equal average run offense, run defense, and plot, say, run EPA for a given play as a function of run percentage, what do you get?

    A crazy model with a quadratic utility function (teams getting progressively better on defense as they see the same thing) is here. The local optimum mentioned there is especially reminiscent of the current situation.
    ReplyDelete
  27. Pat-The measure of utility in this analysis is the net point advantage gained that a team can expect based on resulting down, distance, and field position. As mentioned in the original article, the data was limited to game conditions when the score is close and time is not yet a factor. In these situations, every point is equally as valuable as the next. That is what makes it linear.

    In other words, as long as you consider that being 6 points ahead is twice as good as being 3 points ahead, and so on, that's all you need to satisfy linearity.

    Also, as long as teams aren't running the same exact run and pass plays over and over, then the 'learning' model wouldn't apply.
    ReplyDelete
  28. "In other words, as long as you consider that being 6 points ahead is twice as good as being 3 points ahead, and so on, that's all you need to satisfy linearity."

    I think we're talking about two different things - yes, treating EP as linear in value is fine.

    What I was saying is that you're assuming that the payoff for a run is linear with defensive run D choice; that is, p(Run) = a*(run D fraction) + b.

    That is, the payoff on any given play depends only on the play choice (run/pass) or defense choice (run/pass), and not on anything else - specifically, not on the history of the game (or season!), or the previous play.

    That's an *extremely* unlikely assumption, as so much of the game is pattern recognition and misdirection. And once non-linear situations come up, you can get all sorts of bizarre things like limit cycles, multiple equilibria, etc.

    "Also, as long as teams aren't running the same exact run and pass plays over and over, then the 'learning' model wouldn't apply."

    The defense isn't learning the plays. They're learning what fraction of time to expect a run or expect a pass. The simple idea of that model was that if a pass defense sees 99% runs, they're going to do better against the run than if they only see 1% runs.

    I can't imagine that's a tremendously controversial idea - corners and safeties are going to jump a running play faster if it's all they see. They'll be more vulnerable to a pass at that point, but that's part of the model as well.
    ReplyDelete
  29. I think there's a fairly simple explanation for why the pass % on 2nd and 10 is lower than expected and jumps up again on 2nd and 7-9. And I don't think it's a refusal to randomize, but an adherence to traditional though (though it could be pointed it this is almost the same as lacking the ability to randomize - both reasons get you to the same spot, but the path is slightly different).

    If it's 2nd and 10, in all likelihood the first play was an incomplete pass - traditional theory is then you run the ball to set up a manageable 3rd down. So instead of facing 3rd and 10 after another incomplete in 2nd and 10, you face third and 6 or so, allowing for a more realistic run or pass scenario on third down. So it's probably the result of traditional NFL play-calling thought.

    Then the number jumps up on 2nd and 7-9 because the first play was likely a minimal run that puts the team 'behind' the marker (less than 3.33 yards per down, so to be ahead of the marker on 2nd down, a team would need to be at 2nd and 6 or less). Thus being behind the marker would, under traditional NFL play-calling, tend toward more passing to try to get beyond the marker again.
    ReplyDelete
  30. To clarify, it might be two ways of looking at the same thing but using the term "ability to randomize" would indicate to me the play caller is making the attempt to mix up play calling. That result though is the play caller lacks the ability to mix it up.

    Instead, I think it's a purposeful decision biased toward running on 2 and 10 because that is what the 'book' says. So it isn't a lack of the ability to randomize playcalling but instead a specific bias toward certain play calls in certain scenarios for a specific purpose.
    ReplyDelete
  31. 1.) Do i understand it right? On 3rd and 2-9 Yards, the run have more chances to end the drive with scoring points then with passing??

    2.) Or is it that only the "gap" between pass and run closes (example: on 1 & Ten its 4,0 Y/R while 6,0 Y/PP = 2,0 Yard-"Gap" favouring the pass; on "predictable" 3rd & Longs its 5,5 Y/R, while 6,0 Y/PP = 0,5 Yard-"Gap" favouring the pass)?

    I think 2.) is correct. And if 2.) is correct i would go with an extreme Martz-Strategy: Pass on every down in every situation until the Pass gives you less Y/Play then a Run (Turnovers should be included in this calculation of course, but i just want to make my point simple)

    As far as my Stats go, ONLY on 1st and Goal situations 3 consecutive runs (without offensiv penalties) have the advantage over the pass, and 4th and 1 - Plays.

    Even if your offense is predictable, the pass (almost) ALWAYS has the advantage, but only the "gap" closes.

    Conclusion: Teams should not run more on 3rd and Longs, but PASS MORE (until defensive co-ordinators find a way to close the "gap" of the more efficient pass completly).

    Karl from Germany
    ReplyDelete
  32. Hi, Karl. #1 is correct. It's surprising, but there is a good reason. Runs on 3rd down have a higher payoff because defenses are likely too biased to defend against a pass. Offenses should probably run more often to take advantage.
    ReplyDelete
  33. With 3rd and Long i mean 2-9 Yards to go. Just to avoid misinterpretings.

    Karl from Germany
    ReplyDelete
  34. When i look at 3rd down stats i see no drop-off in pass-efficency (Rate and Y/PP), but a huge jump in run-efficency. But still the "predictable" pass gains more yards per play then the run.

    Didn´t your EPA mean expected points added. That doesn´t mean the added points close the gap. I´d really like the totals of your Stats.

    Otherwise i would not understand why Martz, Coryell, Marino etc. were so efficient with their all out passing strategies.

    Karl from Germany
    ReplyDelete
  35. Since Brian Burke could not bring up the exact EP for each situation, i have to assume the following:

    For example a team has a 1st and 10 at the opp. 35yL, the chart says the EP for a play there is aprox. 3,5 Points. Since a run gains about 4 yards and a pass 6 (sacks included), i have to guess (again: i don´t have Brians total stats) that a run has a 2.8 EP and a pass 4.2 EP.

    So if we go down to, lets say 3rd and 9, Brian says the run has 0.19 points added = EPA = total EPA of 2.99.
    The pass looses (call it minus-EPA) 0.02 points = total EPA of 3.98.

    So everybody should see that its missleading to look at the charts in the article above. Even tough a run gains 0.21/0.22 points vs. the pass EPA, the total Run-EPA is still way behind the total Pass-EPA. Only the "gap" closes.

    Conclusion: A pass is still better then a run on 3rd and 9 (and other situations, of course). So a clever coach would pass 99% of the time and only run on the rare occasion when he is 100% SURE he gains a 1st Down by running, or getting into BETTER Field-Goal-Range then by passing.

    People should trust Martz, Coryell, Beli-Cheat and others. "500-Points-Offenses" came by efficient passing all the time, not by running.

    Only teams with a bad OL and/or QB should depend on the run and pray their defense helps out...

    Karl from Germany
    ReplyDelete
  36. Edit

    "The pass looses (call it minus-EPA) 0.02 points = total EPA of 3.98."

    should be, of course:

    "The pass looses (call it minus-EPA) 0.02 points = total EPA of 4.18."

    Karl from Germany
    ReplyDelete
  37. Karl-I'm sorry. I must not have explained this very well. Your interpretation above is mistaken.

    Anytime you see the EPA values for passing higher than that for running, it means that a pass play in the given situation will tend to lead to a higher net point advantage than a run. And when the EPA values for runs is higher, it means a run in that situation tends to lead to a higher net point advantage. It's not that the "gap" closes by the amount. The EPA is the amount.

    EPA works like this: a 1st and 10 at the 50 is worth about 2.0 Expected Points (EPA). Suppose you gain 5 yds on that 1st down. Now you're at 2nd and 5 at the 45, which is worth 2.2 EP. That play, whether it was a run or pass was worth +0.2 EPA. Passes tend to gain more EPA on 1st and 2nd downs, but runs can often gain more EPA on 3rd down, principally approaching field goal range.

    We always need to value the outcome of a play by its relative improvement from its starting point. A 1-yd TD run is not worth 7 points because the EP of a 1st and goal from the 1 is 5.9 EP or so. It's only worth +1.1 EPA.

    If you're interested in the baseline EP values for all situations, I've published them for public use in this post:
    http://www.advancednflstats.com/2009/12/expected-point-values.html
    ReplyDelete
  38. OK, i think i got it (almost) now.

    It means a 2nd Down and 5 on the opp. 45yL is 2.2 EP. If a run is called on this 2nd & 5 the EPA is minus 0.10 = Run-EP is 2.1. If a pass is called on 2nd & 5 at this yL, the EPA is 0.00 = 2.2 EP.

    Correct?

    It just makes me wonder why high scoring teams are always good "Passing-Teams", while Running-Teams tend to score less points.

    A perfect example are the 85-Bears: Even tough they had the same (or better) Defense in 84 and 86, they won (and were 2nd in points scored) when they had a efficient passing game (speak BigMac was relativly healthy). In 85 they had the best Pass-Offense in the NFC (6,5 Y/PP). They year before and after they failed, because they had to rely on their running game only...

    Since i dont have computer program, i´ll do some stats in the off season "per hand" comparing running and passing on 3rd down, because i see toooo much successful passes on "predictable" 3rd and Longs.

    Omg, that will be tough and long work, but i´ll let you know :-)

    Karl from Germany
    ReplyDelete
  39. I think if I understand correctly, you are right. Those are averages of course. (Not every run on 2nd and 5 at the 45 will result in -0.10 EPA.)

    You are correct that the high scoring teams are passing offenses. Note that the higher EPA values for runs are on 3rd down situations only. And this is only because of the current strategy mix. Runs are infrequent on 3rd down, so they often take defenses by surprise.On 1st and 2nd down, passing is more lucrative.

    I'm not advocating always running or always passing in any situation. The imbalance just suggests, in general, teams should inch toward selecting the play type with the higher payoff more often.
    ReplyDelete
  40. Brian,

    you made me interested :-)

    Because up to yesterday i was 100% SURE passing (almost) always is the better option then running, no matter how predictable you become (hail Martz, even tough i might be the only one doing so :-).

    Since i am not good in computers (actually i am VERY bad) i have to make stats by hand. I´ll do "only" 3rd downs (which will still lead up to aprox. 3.500 plays, omg), since thats the down when passing "fails".

    Up to now i only have stats which tell me on 3rd downs (no matter if short, long or very long) there is no drop off in passing efficiency, even tough running efficiency is way up.

    To leave out tendencies i´ll make only 1st and 3rd Qtrs with max. point difference of 10. Is that a good starting point?

    Karl from Germany
    ReplyDelete

Subscribe to: Post Comments (Atom)