Fighter Pilots and Firing Coaches

At its heart, football strategy is game theory. But at the heart of game theory is utility theory--the science of defining and measuring what is useful. Utility in football is tricky. A 4 yard gain on 3rd and 5 isn't very helpful, but the same gain on 2nd and 4 is.

Utility theory is also at the heart of economics, which may be why we see some coaches with economics backgrounds do well. One of the cornerstones of utility is Prospect theory, a concept developed by researchers Daniel Kahneman and Amos Tversky. Simply stated, Prospect theory says that people are averse to losses more than they value equivalent gains. It was a major breakthrough, and it won the 2002 Nobel prize for economics.

But this article isn't about Prospect theory as much as it is about some of Kahneman's earliest work. While Tversky was always an economist, Kahneman's training was as a psychologist.

In the late 60s, Kahneman was a consultant for the Israeli Air Force. He lectured instructor pilots on the latest research that showed that reward was far more effective than punishment at improving performance. Instructor pilots were not buying any of it.

They told Kahneman, when student pilots have a bad flight we yell and scream at them, and the next day they tend to do better. But when they have a good flight we'll praise them like you suggest, and they tend to do worse.

It was then that Kahneman realized how natural variation in performance, and its natural regression to the mean, were fooling the flight instructors into believing it was their yelling and screaming that improved the student pilots' performance. Flight instructors saw this:

student has poor flight --> yelling and screaming --> improvement on subsequent flight

When in reality, they were just as likely to see this:

student has poor flight --> improvement on subsequent flight

What does this have to do with firing coaches? Owners rarely fire coaches who just finished a successful season. A new coach comes in and there is usually an improvement, but that improvement would tend to happen anyway, with or without a new coach, simply due to regression to the mean.

For those who aren't familiar, regression to the mean is not metaphysical mumbo-jumbo. It just means that the change in performance from one day to the next is most often in the direction of the overall average performance (and not directly to the average itself, or to the average and no further). It occurs because when things go really bad it's usually because several bad things all happened at once to spoil the outcome. And when things go really well it's usually because several good factors conspired to win the day. It's not likely that all those factors will repeat in the same way.

Regression to the mean is not a rule, but it is a law. In other words, in some cases good outcomes get even better the next time, or sometimes bad outcomes get even worse. But the fact that most of the time good outcomes won't be as good and that bad outcomes won't stay as bad is an unalterable fact of life.

For example, when the 2008 Detroit Lions went 0-16, it was because they had injuries at critical positions, little talent on defense, a tough schedule, plus the ball never really bounced their way. Next year, it's unlikely that all those things will occur again, and they'll likely win a handful of games. And when the 2007 Patriots went 16-0, several factors all went their way. They had acquired lots of talent on both sides of the ball, had a relatively easy schedule, and stayed healthy all year. In 2008, they didn't stay healthy and mustered 5 fewer wins.

NFL owners and team executives, repeatedly witness this cycle:

team does poorly --> fires coach --> team does better

And they are tricked into attributing the team's improvement to the firing of the coach. The result is a belief that replacing the coach is more helpful than it truly is. In reality, this is almost as likely to happen:

team does poorly --> team does better

In fact, how do we know that starting over with a new coach doesn't hurt more often than it helps? Perhaps teams with new coaches improve slightly less than similar teams who keep their coaches. I'm not claiming that all coaches should keep their jobs no matter how poorly their team does. My point is that replacing the head coach is probably not nearly as effective a move as it appears.

The recent parity of team talent levels in the NFL has made the regression effect more prominent. Because teams are generally closer to each other in talent, it is much easier to go from few wins one year to many more the next (and vice versa). Only a slight improvement or decline is required to leverage a large change in a team's fortunes. This may be why the average coaching tenure is getting shorter.

This lesson should extend into all sports, and even day-to-day life. Beware of stories like these: "I had several bad colds last year, then I started taking [herbal product X], and now I don't get sick as often." Or, "My portfolio took a beating last quarter, so I changed my fund manager, and now my investments are making gains again." How about, "Our economy was in a slump, so we borrowed and spent $900 billion to cause it to recover." Or maybe, "My NFL franchise lost too many games last year, so I brought in Bill Parcells, and now it's doing much better."