4th Down Decisions: Part 3 – Which Coaches are the Worst/Best?

Time for the final part of our 4th Down Decision series.  Today we look at individual coaches.  Again, you can follow Jared at @jaredscohen.

Part 3

Decisions by Coach

Now we’re getting into some fun stuff.  Which coaches have the highest pass rate?  We’ve already established that as a whole, coaches are far too conservative.  But are there any who appear to ‘get it’ more than their peers?

Like Moneyball, have any of them figured out that an overlooked (and more aggressive) approach might lead to better performance and more wins?

Let’s take a look.  Below is a table of all the NFL coaches and their 2012 regular reason optimal decision percentage:

Screen Shot 2013-05-22 at 10.42.08 AM

Or, for a more interesting look, click this graphic:

Screen Shot 2013-05-22 at 11.06.07 AM

Now, I know what you’re going to say.

Norv Turner???

Ron Rivera???

Marvin Lewis???

Not exactly a murderer’s row of coaching legends (Andy Reid’s up there too, by the way).        And some of the coaches at the very bottom.  The Seahawks? Packers? Falcons?  They all had good years.

So what’s the deal?  Am I just some crazy idiot from his mom’s basement?

Shockingly, I don’t think so.  I don’t think our data or our conclusions are wrong.  Although when you illustrate it a different way, there are still questions.

Screen Shot 2013-05-22 at 10.43.51 AM

This chart illustrates coaching optimal decision rate (pass rate) against number of regular season wins.  Now, what would be best would probably be point differential or Pythagorean wins or something slightly different, but what’s still interesting is that there appears to be a negative relationship between wins and optimal decision-making.  Finding the correlation gives us a -0.34, so a slight negative relationship.

So, what gives?

Well, first, there’s a question of causality.  Does this data mean that making the ‘optimal’ decisions actually prevents you from winning more games???  Should everyone just punt the living daylights out of the ball on every fourth down?  Well, at least kickers and punters would be happy.

One could interpret it that way, but I think that would be wrong.  I think anyone who thinks that’s the case has actually got their causality backwards.

What I think is a far more likely scenario, is that the worse a team is, the more often it’s playing from behind.  And I think teams who are behind and trying to come back are often more aggressive.  That aggression helps those teams to make ‘more optimal’ decisions during games.  But unfortunately, those teams are behind on the scoreboard for a reason, and most of the time, they lose.

I think the causality works the other way.  Poorer teams are more frequently losing, and therefore more focused on maximizing their total points (e.g., making a comeback).  So they make more optimal decisions.  The teams that are ahead most of the time are more likely to play conservative to hold their lead, so their decisions may be sub-optimal.

I tried to adjust for this by removing plays late in the 2nd/4th quarters and when the score was out of hand, but maybe that wasn’t enough.

Because while I’d like to give Norv and the other guys credit where it’s due, I have to think it’s largely driven by circumstances (but would love to hear other theories as well)

There’s also a way we can check for this.

Decisions by scoring differential 

If we take a look at optimal decision rate based on what the score is, we can see if all coaches behave differently when their team is behind and trying to come back.

If the hypothesis is true, that coaches make ‘more optimal’ decisions when they’re trailing, then we’ll see that in the data, and that could explain why losing coaches (or in Norv’s case, fired coaches) have much higher optimal decision rates.

Screen Shot 2013-05-22 at 10.46.45 AM


When coaches are behind, they make more optimal decisions roughly 10% more than when they’re ahead.  And in the fourth quarter, that gap is even bigger.

Looks like we may have an answer.  Or at least some indication that when you’re behind in a game, you make better calls.

The optimal decisions themselves are more aggressive than most coaches in the NFL.  Teams that are behind are often more aggressive to catch up.  Therefore, teams who are behind more often (and lose more often) will naturally make more ‘optimal’ decisions.

Makes sense to me, but I was all excited to start the Norv Turner is better than Bill Belichick bandwagon.

I’ll have to put that on hold for now.


So what have we learned from all this?

We know that coaches are far too conservative, particularly when faced with short yardage situations in opposing territory.  They kick the ball far too often rather than trying to convert.

Our data is pretty clear on this point.  Coaches only make the right call 15% of the time when they’re supposed to be going for a first down, and when they’re in opposing territory, they make the right call less than half the time.

Well, the natural question, is why?

It can’t be a lack of information.  NFL coaches and management have all kinds of data at their disposal (certainly more than I do), and plenty of talented folks.  And it’s not like these ideas haven’t made it into the mainstream.  High school coaches have made news for never punting, and analysts continue to harp on the conservative tendencies at the NFL level.

At the end of the day, I still maintain that it all comes down to risk aversion.  NFL coaches (and most professionals) have one primary goal.  To stay employed.  And taking a strategy that goes against conventional wisdom exposes you to criticism if the outcomes don’t work out.

I read a quote from Mark Cuban not too long ago where he felt the idea that coaches wouldn’t try anything to win games was laughable.  If I remember correctly, he thinking was that professional sports are so competitive and the need to win so great that of course coaches take every opportunity they can to get better. (and in googling around, I can’t seem to find it, so maybe I’m misremembering)

Cuban certainly has more experience in professional sports than I do, but I think at least in the NFL the data clearly suggests that’s not the case.

Coaching at the NFL level is the highest professional position a football coach can ever get.  It typically takes years and years of work in all kinds of low-paying jobs (what does a quality coach even do?) and moving from city to city with the hope of one day snagging one of those 32 openings.

Oh, and once you get one of those slots, you can’t screw up, because you’ve spent your entire life specializing in a sport with a fixed number of teams and exactly one relevant professional league.  You’ve invested your entire career to get to the top of the pyramid, but there’s nowhere to go but down.  You’ve got to stay up there.  It’s not like you can go coach pro baseball.

So with a lack of transferable skills outside of football, and the inability to create a startup NFL franchise to coach, NFL coaches are in something of a bind.  Unless they have bulletproof job security (and in the long-run, no one has that, right Coach Reid?) they have a clear incentive problem to try such an easily observable strategy.

Why?  Because it’s quite possible the outcomes won’t work out, and people will crucify you if they don’t.  See the criticism Belichick got for his 4th down decision against the Colts some years ago.  He made the right decision, but since the outcome didn’t work out, all of a sudden he made a terrible mistake.

It’s the problem of evaluating decision-making skill based on the outcome.  The equivalent would be telling a poker player he screwed up when he got all his money in with the best hand and someone drew a miracle card to beat him.

If you made all the optimal decisions you could make, and let’s say you were running at a 50% pass rate before, you might do something differently on ~30 plays a year.

Some of those plays might work, but some might not, and the ones people will focus on will be the ones in the situations of highest leverage where the outcome of the game hangs in the balance.

Now, I think making more ‘optimal’ decisions could swing a game or two a year in your favor, but it could also swing a game or two against you (like Belichick and the Colts).  When faced with that possibility, it’s no surprise coaches aren’t chomping at the bit to test out the theory.

For a coach to successfully try this (and it’ll only take one to succeed before others join), I would argue one of three scenarios needs to happen:

–        An edict needs to come down from the owner themselves, mandating the change to football strategy (of course, what kind of coach would want to be in that kind of situation?)  The most likely suspects would be new analytically inclined owners like the Kahn family in Jacksonville, or owners that like to involve themselves in football operations (Jerry Jones, Dan Snyder, the late Al Davis)

–        A coach fresh off a super bowl win uses the capital/credibility from his victory to test it out (you could imagine Belichick doing this, maybe Sean Payton still has enough cred, but I think you pretty much need to be a Harbaugh)

–        A brand new coach in the first year of a contract just let’s it rip and goes for it, putting their NFL credibility at risk because they at least could always go back to college

If you think I’m suggesting Chip Kelly go for it in his first year with the Eagles.  You’d be absolutely correct.


3 thoughts on “4th Down Decisions: Part 3 – Which Coaches are the Worst/Best?

  1. Your causality question is an important, and I feel like you did a great job in demonstrating how game-day positioning can influence the aggression of coaches on 4th down.

    That being said, there is an even larger arc to examine within the coaching statistics; the long-term leash that they have within their position. You touched on this briefly near the end of the piece, but you’ll note that not only were the more aggressive coaches on losing teams, they were on the hot-seat within their current jobs.

    If you evaluated the data on Andy Reid or Norv Turner from earlier in their career (even during losing seasons), I’d be willing to bet they would score much lower. I remember several un-Andy-like decisions happening last season (as the season and his job were clearly slipping away), specifically in regards to 4th down decisions.

    This doesn’t really change the content of part 3 of this piece (which I think is the hardest to examine analytically due to these factors). The only issue I see with the score-card you provided in the earlier posts is the inability to accurately gauge momentum. Unlike blackjack, where cards are unknown but have a previously determined order to come out in, football games are constantly changing from card-to-card. Big plays swing morale, which often snowballs into additional momentum. I think most coaches (obviously dependent upon the scenario), would feel that being stopped on 4th down creates more momentum for their opponent than picking up that 1st down creates for their own offense.

    That being said, I too would love to see Chip Kelly’s seemingly inhuman confidence lead to an analytical approach to dealing with 4th downs.

  2. Great piece again. I agree with most of what you said as well, and I suspect Chip will be one to push the envelope on these decisions. I would be surprised if he doesn’t have one of the highest “pass” rates this year. (I hope you will keep track of this and give us some updates through the year).

    But, I think Bonez really hit it on the head above. Football is not blackjack, and if you double-down on 11 but get a 2, it doesn’t affect your next hand (other than knowing there’s one less 2 out there). If you go for it on 4th and 1 from inside your own 20 yard line and fail, you might be doing what’s right statistically, but you also may seriously swing momentum by energizing the other team and making your own players wonder what the hell you are thinking.

    Another area where I think this same premise could be examined is going for 2 points after a TD. (Not using the current chart to decide when it gets you back in a game, but just to add extra points to your score when you are ahead). As we know, Chip already has a reputation for doing this, although as he describes it, he does this only when he sees a mismatch that increases his odds of success. I remember last year when the Jets signed Tebow that many suggested they should just go for 2 every time since some projected he could score successfully 75% of the time (so for every 4 scores, you net 3 additional points–asssuming 100% PAT kicks). In theory, as long as you convert more than 50% of the time you come out ahead. But, again, since you cannot predict which will be the 50% you are successful, you have to be careful. You could score an additional 3 points in a game you would have won anyway, and then miss the conversion in a game you end up losing by 1.

    I know there is a high school football team that never punts and always onside kicks. They were featured on HBO last year and it was fascinating. http://www.straitpinkie.com/sports/college-sports-sports/pulaski-academy-sports/the-quirky-philosophies-of-pulaski-academy-head-coach-kevin-kelley/. Coincidence that his last name is Kelley?
    Makes you curious how it would work if someone really committed to it. I bet Chip will come closest. Should be an interesting season.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s