Comments on Advanced NFL Stats: Offenses Run Too Often On 1st Down

Hi, Sorry to post a comment years later, but I wa...

2011-11-23T13:22:08.644-05:00

Hi,

Sorry to post a comment years later, but I was directed here via deadspin. After reading some of the comments, it seems that some of the people arguing over average v. marginal utility are using odd models of game theory (or just don't really get how one accomplishes figuring out maximizing overall utility). I think all of the arguments stem from some version of a model where a team that runs 100% of the time and then suddenly passes would benefit greatly from that pass. From an individual play standpoint, sure, doing the thing that maximizes utility would be the proper move if the game were only played on one iteration. But for a multi-iteration game, they're essentially implying that the reward values in the matrix will not be constant amounts but will be frequency dependent. And that's fine; the calculus to find the Nash equilibrium becomes a little more complicated with multiple references to the frequencies in the equations, but there is still a Nash equilibrium. And as you correctly point out, the equilibrium stabilizes the per play value (and should therefore render the average pass and run plays equivalent), though it may be outside of the proper frequency ranges, of course.

Regardless, I think what you're really identifying in this article is that they are not at the Nash equilibrium, and the way to drive toward it would be both for the offense to pass more often and the defense to defend the pass more often as a response. But that is something different than saying that the offense "should pass more." To argue that, you'd have to demonstrate that the Nash equilibrium returns a higher total value than the current state. If the defense did not respond at all to the offensive change in strategy, then obviously the ideal strategy would be to pass 100% of the time. But guessing that the defense would change - thereby altering the frequency-dependent matrix values and driving closer to the Nash eq. - you really need to know whether the current state is better or worse than the Nash eq. *for the offense* before recommending that they change strategies. This violates some "rational acting" tenets of game theory, but if you know that your opponent is behaving inefficiently and is unlikely to change unless you draw attention to that fact, then staying pat may be the better utility move.

Again, without actual data for frequency dependent utility returns - which would necessitate WAY more data points - it's hard to say exactly what the proper course is. But the fact that the offense is not maximally exploiting the current defensive strategy does not entail that one should definitely maximally exploit it if you know that would cause the opponent to behave more rationally.

Do you factor in whether the other team's offe...

2011-01-18T16:20:40.458-05:00

Do you factor in whether the other team's offense is better than yours. Much like teams that play against Peyton Manning or Tom Brady, sometimes the best way to defend those guys is to take time off the clock and shorten the game. A 3-and-out running the ball takes at least 2 more minutes off the clock then 3 incomplete passes.

Also, how do you account using the run to set-up the run in the 4th quarter? A lot of coaches say the only way to insure you run the ball when you need to in the 4th is by banging away and weakening the defense in the first half. By limiting the analysis to just those plays when "neither score nor time is important," have you factored that out?

Question: When a QB fumbles, is that considered a...

2010-12-15T21:17:19.202-05:00

Question: When a QB fumbles, is that considered a running play? If so, this would throw off the stats big time.

Running on 1st is an investment for future gains i...

2010-12-15T21:04:39.648-05:00

Running on 1st is an investment for future gains in the 4th quarter. Everyone knows that running is most effective in the 4th quarter.

Moreover, the risks of passing go up with more passes, and the single most important factor in the NFL is turnovers.

Brian, I am new to the site, fantastic stuff. Ano...

2010-11-23T01:50:04.881-05:00

Brian, I am new to the site, fantastic stuff.

Another "set-up" thought: Running takes a toll on the d-line. Passing takes a toll on the o-line.

Can you see this in the data? For example, do teams who have executed more running plays (not a higher % of running plays, but I higher number of running plays) in Q1 and Q3 have more offensive success in the 2nd and 4th quarters?

Didn't read the whole thing because I have a h...

2010-05-14T18:23:35.021-04:00

Didn't read the whole thing because I have a headache, so if my critique was addressed, then please just say so.

Anyways, suppose you are right that NFL offenses run too much on first down. Does that take into account that perhaps some teams are just better at running than passing and are playing to their strengths? Take for example, a team with a rookie QB. That team will likely run a lot more than pass.

The defense may game plan for the run, which under normal circumstances, make it smart for the offense to pass. But given that the offense has a rookie QB, and the expected outcome from passing, even given a run-oriented defense, may be only slightly higher than running the ball, then perhaps a team may run more than pass.

Basically, you may be right the NFL as a whole runs too much, but on a team vs. team basis, matchups actually dictate what is happening.

"Maybe coaches shouldn't be so cavalier a...

2010-01-21T19:27:00.425-05:00

"Maybe coaches shouldn't be so cavalier about 'test plays.'"

I don't understand this comment. You have no idea how much value they result in, since you don't know what plays were "test" and what plays were changed due to that "test."

It's relatively easy to test your assertion that teams should run less: group teams by run/pass percentage on first down in 1st/3rd quarters, and plot average total EP/drive for all drives in those quarters.

If it's true that teams should pass more on first down, the teams that *do* pass more on first down should be doing better than those that *don't*.

"Option plays, of any kind, will nicely collapse into a sub-game at the parent node of the general play type (run or pass)."

Play 1: Offense notes that defender crashes down on a certain type of run - OC adds a run option, which allows a pitch-out if that defender does, otherwise a keep.
Play 2 (not necessarily next): Offense runs previous play - defender repeats previous action, and offense gets a big gain.
Play 3 (not necessarily next): Offense repeats previous play, defender adapts (playing better gap discipline, etc) and play results in a minimal gain.
Play 4 (not necessarily next): Offense repeats first play, defender plays as before, play results in a decent gain.

Those plays depend on the previous history in the game. If play 1/4 average (together) less than play 2/3 average (together), you might ask "why don't you run play 2/3 more often?" and the reason is because play 2 (the big gain) only occurred because of what was noticed on play 1, and by play 4, the defense had adapted to play 2/3.

You might think that this would trend towards an equilibrium, and it could - but that equilibrium may not be at the minimax point, and that equilibrium probably wouldn't be reached in a 60-play game anyway.

Maybe coaches shouldn't be so cavalier about &...

2010-01-20T17:51:31.981-05:00

Maybe coaches shouldn't be so cavalier about 'test plays.'

Option plays, of any kind, will nicely collapse into a sub-game at the parent node of the general play type (run or pass).

"I don't need to call a run first to get ...

2010-01-19T18:05:58.195-05:00

"I don't need to call a run first to get you "set up" for a pass on the next play."

I don't see how you can claim this with absolute certainty when coaches have *flat out stated* they call plays in order to see how a defense reacts. Calling a 'test play' to see how a defense reacts is the same thing as 'setting up' a defense, just using different terms - in that case, the offense's "follow-up play" is determined by the defense's reaction to the first play, rather than the defense's reaction to the follow-up play being determined by the first play.

You're also only thinking about it from the defensive playcaller's point of view, and *not* the defensive *player's* point of view. A defender has to react to what he sees, not what *could* happen. His reaction time is going to be modified by what he knows about the team, and what he's seen that game.

"Chris Brown (of smartfootball.com) once wrote that play calling in football is not much different than rock-paper-scissors."

In terms of the offensive coordinator/defensive playcalling matchup, yes. But that's not all that's going on. Defenders reactions can be manipulated, personnel groupings on the field can be manipulated, etc.

And you can't follow a mixed strategy with either play reactions (since it's 11 different decisions - 11 random actions wouldn't work well) or personnel groupings (since the defense doesn't have completely free substitution and doesn't know what the offensive formation is).

Chris is also a (self-stated) option purist - and the entire idea of an option is the fact that you can react to what a defense does. Which means that the *next* time, the same play's outcome depends on the previous play's outcome as well.

A good example is back when Chris described the inverted veer option from TCU: here. When a formation shifted, the defense reacted based on a previous formation they'd seen on film, and it completely crossed them up.

"I'm guessing coaches think like you do, believing they need to run first, and that might help explain why they run too often."

What I'm confused at is to how you can state with a certainty that they *don't*. There's no theoretical reason why the success of pass plays couldn't be entirely set up by leading the reaction of the defense.

We *know* it's possible to manipulate a defense's reaction based on things that occur during the play. That's what QBs do when they move DBs with pump fakes and look-offs. All this is saying is that early-play action can also manipulate a defense's reaction.

"Deliberate sequencing is a very bad idea for an offense. It creates a pattern, which however slight makes an offense predictable to some degree."

Of course the defensive players could try to predict it. But the problem is that they have a grand total of a half of a second to predict it. The defensive playcaller can say "on this play, look out for C, because they ran A earlier" but a defender still has to recognize the difference between A and C, and whether or not he is *capable* of doing that is a matter of skill, not game theory.

Pat-You still seem hung up on sequence. "Foll...

2010-01-16T11:10:30.966-05:00

Pat-You still seem hung up on sequence. "Follow-up" plays and preceeding passes are irrelevant. I don't need to call a run first to get you "set up" for a pass on the next play. I just need to have you aware that I'll call a run with some probability on the current play.

Chris Brown (of smartfootball.com) once wrote that play calling in football is not much different than rock-paper-scissors. Say I plan to play rock, hoping you'll play scissors. I don't need to play paper on a preceeding instance to set you up for scissors. You just need to be on guard against the possibility I might play scissors now.

I'm guessing coaches think like you do, believing they need to run first, and that might help explain why they run too often.

Deliberate sequencing is a very bad idea for an offense. It creates a pattern, which however slight makes an offense predictable to some degree.

"Teams are running more often than necessary ...

2010-01-12T17:41:59.414-05:00

"Teams are running more often than necessary to set up those longer passes."

Without knowing
1) which specific runs are intended to be set-up plays
2) which specific passes are follow-up plays
3) how the results of the follow-up plays would change based on the number of set-up plays

I don't see how you can possibly justify that statement.

"If what matters is overall, long-run net point maximization"

... *per play*. Per-play net point maximization may not be the same as per-game net point maximization if the outcome of a play depends on prior plays after accounting for field position, down, distance, etc.

If we agree that in 'normal' situations th...

2009-12-05T06:17:34.362-05:00

If we agree that in 'normal' situations that net point advantage is all that matters, and that every point is equally valuable, then that's exactly what I'm saying. Teams are running more often than necessary to set up those longer passes. If what matters is overall, long-run net point maximization, then teams should generally run less often.

There is nothing controversial or debatable about that.

Brian- I agree that each time a team runs on 1st d...

2009-12-05T05:49:48.438-05:00

Brian- I agree that each time a team runs on 1st down, they are sacrificing optimum performance...on that play. But they are setting up those long TD passes to a degree we cannot measure. Can two 1-yard runs cause a 90-yard TD pass? Would that 90-yard pass have happened anyways without the run plays? I don't think these questions can be answered by the game theory model. This is why I believe measuring the success of a single play is not an appropriate method of judging the full-game chess match that is play-calling.

Brett- Long TD passes count, don't they? Other...

2009-12-03T04:36:44.252-05:00

Brett- Long TD passes count, don't they? Otherwise, we completely agree. My point is that teams are running too often to constrain defenses to the run. And each time they do, they're actually sacrificing optimum performance. The interdependency of playcalling is exactly what the game theory model is for.

We sure can put those subtleties into numbers. It's just a matter of how detailed our data is.

Brian-- This is another good article and I usuall...

2009-12-03T00:50:27.909-05:00

Brian--

This is another good article and I usually agree with you, but I think you are greatly underestimating the importance of having a balanced offense. By isolating individual plays to compare their EP values, you are overlooking the interdependency of all plays, which is pretty much the essence of play-calling. I'm guessing the average EP of passing plays is heavily weighted by long TD passes that were made possible by the fact that the defense was expecting a run. You might argue the converse is also true, but most people will agree that run/pass interdependency is not entirely mutual. For example, the more a team stacks the box to defend the run, the more susceptible it is to long TD passes. But the converse is not true; when a team is playing prevent pass defense, it is not more susceptible to long TD runs--just a bunch of 8-12 yard runs. This is a subtlety of the game you simply can't put into numbers, as much as i'd like you to. Keep up the good work.

May I make a suggestion? Before I begin, let me s...

2009-11-30T21:38:23.012-05:00

May I make a suggestion? Before I begin, let me say that your conclusion is in line with my feeling on the subject, but I think that analyzing the value of certainty would be helpful. If your data set allows it, I believe that I can suggest a potentially interesting avenue of investigation.

For each 'bucket' of field position, for both runs and passes, evaluate the following:
* Run a distribution of results (i.e., X1% of Y1 yards gained, X2% of Y2 yards gained, etc.)
* Evaluate the +/- for each result and then evaluate the full distribution - because the value of +/- 1 yards is almost certainly not linear with respect to yards gained/lost.
* Take the weighted average of these distributions.

I believe that these results will yield a similar conclusion in a much more persuasive manner with respect to the value of uncertainty.

-------

Secondly, while this analysis is meaningful for the value of a current drive and the next drive, it does not evaluate something that the coach likely holds dear - long, clock-grinding 'body-blow' drives meant to tire the opposition. If the entire analysis (with the above change in methods!) were repeated with respect to likelihood of getting an additional first down (or touchdown if inside the 10 yard line), that mightly more closely model the actual goals of 1st and 3rd quarter coaching goals.

-------

Finally - is your data set available in excel? I'd love to play with this myself if possible.

Brain: As you may no I came to your cuntry throug...

2009-11-30T11:14:30.259-05:00

Brain:

As you may no I came to your cuntry through cadana. Does your 1st down theereys work fo the Ligue of the Cadana Football teems? - yes.

If so do you like the Rough Riders or the Roughriders.

PS I support your war of terror!

So a low variation decision that leads to the same...

2009-11-28T17:20:39.504-05:00

So a low variation decision that leads to the same expected win percent is equal to the high variation decision in a small sample?

Brian is completely right about the absurdity of a...

2009-11-27T16:01:54.419-05:00

Brian is completely right about the absurdity of applying portfolio analysis to football WP decisions. The bend in the utility curve is the only reason why you need to bother with things like "risk management", etc. This is a really important topic, because it appears to come up a lot. It seems like people are saying that a stable 75% WP is worth more than a 50% shot at 50% WP, 50% shot at 100% WP. Thats just wrong, even though it is conventional wisdom.

This isn't to say that all football decisions are perfectly linear. On a run, the first yard is worth far more than the 50th. In some cases, the 6 inches immediately in front of the first down marker can be worth >30 yards. But all of this is accounted for in WP already.

I'm amazed at how many conventional football people are willing to spout cliches like "winning is the only thing" but not actually use that principle in their decision making.

Yeah, I never saw the value of the portfolio analy...

2009-11-27T12:57:19.030-05:00

Yeah, I never saw the value of the portfolio analysis. I value low variance in financial returns because I may really need the money at a particular time. In football, everyone always needs their points at exactly the same time -- at the end of 60 minutes. And the EP and WP metrics incorporate that reality.

The 69% to 65% conversion spread is VERY strong evidence. If passing leads to more conversions and -- I assume -- higher yardage gain, that's obviously a higher EP. My resistance is breaking down! However, a couple of factors remain that might mean the actual inefficiency is smaller than the league numbers would indicate:

1) if better offensive teams pass more often on 1st down (this doesn't appear to be true overall, but may be true in "normal football";

2) if the missing plays -- QB fumbles, penalties before the snap, others? -- are disproportionately coming on planned passes.

But a question for Brian on the larger game theory model: your team run and pass metrics in the team efficiency model have zero correlation this season. If game theory is right, the correlation should be 1 -- a team's run and pass rankings should always be identical (I think). Any thoughts on how that could be?

One other note about the run-and-shoot offense. As...

2009-11-27T11:19:53.054-05:00

One other note about the run-and-shoot offense. As I understand it, it wasn't about passing a lot on 1st down. It was a specific kind of read-react doctrine for QBs and receivers. It failed because pro defenses quickly learned how to counter it effectively, not to mention that the teams that employed it weren't all that good.

I'll post a bunch of graphs on this in a few days, but here are some initial results. Outside the red zone, passing on 1st down subsequently leads to a conversion 69% of the time. Running on 1st down leads to a conversion 65% of the time. Conversions are what keeps a defense off the field and rested, whether they come from passing or running. Buddy Ryan would be happy.

I'm no longer a big believer in the portfolio ...

2009-11-27T10:55:33.671-05:00

I'm no longer a big believer in the portfolio metrics for football. I think they can help measure which teams are aggressive or conservative, but they cannot be prescriptive.

Financial portfolios are very different from a football game. You don't get to carry forward gains from one game to the next. In other words, you can't take a 20 point lead with you into the next game, like you do with portfolio gains.

The utility curve of monetary gains bends. In other words, the first $100k in savings is more important to me than my next $100k, and far, far more important than my 20th $100k. This gives value to low-variance, low-risk decisions.

Football isn't like that, my fist 10% chance of winning is just as important as my next 10% chance. Win Probability is a perfectly linear utility. And in 'normal' football (as I try to define it-close score, time not a factor), every net point advantage over an opponent is equally valuable.

Also, variance is not necessarily a big concern. In a strict 2-player zero-sum world, I don't care whether I win with highly erratic large swings in gains and losses, or with tiny incremental gains. I only care if I win.

Lastly, the consistency or low-variance value of the run vs the pass is already factored into the EP model. The EP value of any play takes into account the prospective likelihood of converting first downs.

Concern about 3-and-out only makes sense, of cours...

2009-11-27T10:20:30.125-05:00

Concern about 3-and-out only makes sense, of course, if the 1st down conversion% is lower when a team runs on 1st down. Perhaps Brian can tell us if that's true.

I think Jim presents a strong case for skepticism about a large inefficiency, despite what the EPA data seems to show. Another angle on this is to see how much variance exists in terms of teams' 1st down pass%. I would guess it's pretty high, and that the teams which pass the most tend to be good passing teams. If so, that makes the inefficiency unlikely, because you will have some teams clearly succeeding more often by passing. Even if coaches/teams don't understand the probabilities, immitation will tend to wipe out the inefficiency over time.

For a large inefficiency to persist, we should observe very low variance, and thus no teams effectively exploiting the inefficiency. That's what we see with 4th down. If everyone follows the CW, then the CW can remain wrong. But that's not the case with passing.

That's why I think it's worth trying to validate this with more traditional metrics: do high 1st down pass% teams convert more often than they should? Score more points? Win more games?

Guy said... Could coaches have a rational re...

2009-11-27T01:28:17.900-05:00
Guy said...

Could coaches have a rational reason to want to minimize the risk of running 3 plays and punting, especially early in a possession, because of the impact on the defensive squad ?... Just thinking aloud here....

Being that Buddy Ryan punched out Kevin Glibride in mid game while the latter was calling the run-and-shoot for just that reason -- Buddy's players were collapsing from exhausion -- we can safely say that some coaches think along those lines.

Of course to win the game the team wants to maximize total efficiency, not just offensive efficiency. If improving offensive efficiency in a given way reduces defensive efficiency, then...

It's common for coaches to manage the O to protect the D. Sometimes it works -- when Parcells was in NY he always ran the play clock down to near zero on O-plays to give the D rest. Sometimes it doesn't -- when Herm was coaching here too ... well, best not to think of that. But if a D is the team's comparative strength, it can make sense to play to it smartly.

In the spirit of "thinking out loud", some thoughts relating back to the past discussions here of "yards per play" and the "passing premium".

Passing yields more yards per play on average than rushing over the long run because of the higher risk involved -- picks, sacks, 0-gain incompletions, etc.

Still, one might say that since passing does produce more over the long run, I'll just play for the long run, pass, pass, pass, eat the bad things as they occur, and come out ahead in the end as my reward for being smartly brave and aggressive.

The problem is that in a football game the long run isn't so long and gets shorter every play, and bad things can occur by random chance not spread out over time but together in a big clump -- with your remaining run not long enough to recover from them.

E.g.: Peyton Manning throws six picks against SD. Houston running a run-and-shoot blows a 32-point playoff game lead to Buffalo. Look at the play-by-play for Hou at the end and it reads like: inc, inc, inc, sack, inc, pick, sac, inc, pick, inc, sack ... and that was really a tremendous passing offense in the aggregate all year long. When you incur risk very bad things can happen by dumb luck.

As a result, as Brian explored in detail at the time, just as in financial practice, it can make sense to construct a "portfolio" of plays that reduces passing and the higher average yards per play expected from it to a certain extent to reduce the downside risk involved, with the risk savings more than offsetting the loss of average yards, to an optimal point -- the risk a team can afford to incur varying by the time left in the game, the score, etc.

I've been wondering about how corresponding risk is handled in this EP analysis. I'm sure its considered, as it is hardly an unfamiliar idea on this site. But, e.g, as to:

"For example, a 1st and 10 at midfield is typically worth 2.0 EP. A 2nd and 5 at an opponent's 45 is worth 2.2 EP. So a 5-yard gain on 1st down at midfield would produce +0.2 EP. This method factors in risks such as sacks, fumbles, interceptions, incompletions, penalties, safeties and everything else."

I wonder how the risk is factored in. Are all the "bad things" considered to just produce an average EP value for each play, in which case the situation is analogous to the higher average yards per play we get from passing, and we'd expect a significant passing premium?

Or is the risk considered as in developing portfolio analyis, considering the greater compound risk of a bunch of bad things happening all at once, which methodology we'd expect to largely eliminate the passing premium, but with the big passing premium still mysteriously remaining?

If this has been covered earlier, please forgive, I think I've missed parts of the explanation of EP.

Jim - There are a number of possible explanati...

2009-11-27T00:02:33.299-05:00
Jim - There are a number of possible explanations off the top of my head.

Coaching is too slow to adapt to the new reality of passing supremacy ... The Ellsberg Paradox ...Coaches fall prey to the run-win causation fallacy

Well, the coaches and owners who brought the run-and-shoot to the league, and proselytized it (as they did -- they certainly weren't adverse to the idea of passing supremecy), wouldn't seem to have been held back by such things. They're still around. Why'd they give it up?

My point here is that there isn't the natural bias against "more passing" in the NFL that there is against "going for it on 4th", or that there is in "Moneyball"-type situations against underarm throwers and fat, slow first basemen -- which require a James to analytically prove, and a Beane to then show that underarm throwers can pitch and fat, slow first basemen with high OBP can have more value than lean, mean athletes, contrary to all "common sense".

There's always been bias in favor of passing in pro football. The owners have repeatedly changed the rules to promote more passing. The AFL was created as a passing league, and those owners and coaches came into the league. Owners and coaches have brought in and marketed the run-and-shoot, Air Coryell, the Greatest Show on Turf (updated Air Coryell), and the fans have loved it. Since the days of Don Hutson, every generation of pro football has had its "record setting passing heros" ... which is pretty inconsistent with a universal bias against passing, either for the purpose of winning games or selling tickets, which is the sort of universal bias-based inefficiency we see in the other cases.

So it's hard for me to see institutionalized or psychological resistance causing passing to be greatly underutilized -- not comparably to the other cases.

Another oddity is that when inefficiencies are exposed, the new successful practice tends to spread quickly and the old prejudices evaporate. Once the Dodgers signed Jackie Robinson, the rest of baseball followed. Moneyball analysis really has changed how baseball values players. For all the Belichick brouhaha, teams are going for it on 4th down more than ever before (as he did).

Pro sports are extremely competitive, and the NFL in particular is highly imitative. Whatever succeeded recently gets copied by half the league in short order. Look at the Wildcat. Three years ago, who would've thought the option would be spreading through the NFL like the flu?

But for all the high-power passing attacks and passing advocates there have been in the NFL, and for how the owners favor passing and fans love it, this major under-use of passing persists. Leaving both wins and fan $$$$ on the table. It's strange.

Note -- I'm not arguing, just discussing, thinking out loud. I believe your data.

But I think all this points to interesting issues.

The team-level analysis you mentioned could be really informative here, ISTM. I could suggest a lot of ways to slice that data to illuminate what's going on here -- but it's your data, and you'd do the work, and you're so good at it, I'll leave all those ideas to you. I just hope you see the point I'm trying to make.