Time for the final part of our 4th Down Decision series.  Today we look at individual coaches.  Again, you can follow Jared at @jaredscohen.

Part 3

Decisions by Coach

Now we’re getting into some fun stuff.  Which coaches have the highest pass rate?  We’ve already established that as a whole, coaches are far too conservative.  But are there any who appear to ‘get it’ more than their peers?

Like Moneyball, have any of them figured out that an overlooked (and more aggressive) approach might lead to better performance and more wins?

Let’s take a look.  Below is a table of all the NFL coaches and their 2012 regular reason optimal decision percentage:

Or, for a more interesting look, click this graphic:

Now, I know what you’re going to say.

Norv Turner???

Ron Rivera???

Marvin Lewis???

Not exactly a murderer’s row of coaching legends (Andy Reid’s up there too, by the way).        And some of the coaches at the very bottom.  The Seahawks? Packers? Falcons?  They all had good years.

So what’s the deal?  Am I just some crazy idiot from his mom’s basement?

Shockingly, I don’t think so.  I don’t think our data or our conclusions are wrong.  Although when you illustrate it a different way, there are still questions.

This chart illustrates coaching optimal decision rate (pass rate) against number of regular season wins.  Now, what would be best would probably be point differential or Pythagorean wins or something slightly different, but what’s still interesting is that there appears to be a negative relationship between wins and optimal decision-making.  Finding the correlation gives us a -0.34, so a slight negative relationship.

So, what gives?

Well, first, there’s a question of causality.  Does this data mean that making the ‘optimal’ decisions actually prevents you from winning more games???  Should everyone just punt the living daylights out of the ball on every fourth down?  Well, at least kickers and punters would be happy.

One could interpret it that way, but I think that would be wrong.  I think anyone who thinks that’s the case has actually got their causality backwards.

What I think is a far more likely scenario, is that the worse a team is, the more often it’s playing from behind.  And I think teams who are behind and trying to come back are often more aggressive.  That aggression helps those teams to make ‘more optimal’ decisions during games.  But unfortunately, those teams are behind on the scoreboard for a reason, and most of the time, they lose.

I think the causality works the other way.  Poorer teams are more frequently losing, and therefore more focused on maximizing their total points (e.g., making a comeback).  So they make more optimal decisions.  The teams that are ahead most of the time are more likely to play conservative to hold their lead, so their decisions may be sub-optimal.

I tried to adjust for this by removing plays late in the 2^nd/4^th quarters and when the score was out of hand, but maybe that wasn’t enough.

Because while I’d like to give Norv and the other guys credit where it’s due, I have to think it’s largely driven by circumstances (but would love to hear other theories as well)

There’s also a way we can check for this.

Decisions by scoring differential 

If we take a look at optimal decision rate based on what the score is, we can see if all coaches behave differently when their team is behind and trying to come back.

If the hypothesis is true, that coaches make ‘more optimal’ decisions when they’re trailing, then we’ll see that in the data, and that could explain why losing coaches (or in Norv’s case, fired coaches) have much higher optimal decision rates.

Hmmm…

When coaches are behind, they make more optimal decisions roughly 10% more than when they’re ahead.  And in the fourth quarter, that gap is even bigger.

Looks like we may have an answer.  Or at least some indication that when you’re behind in a game, you make better calls.

The optimal decisions themselves are more aggressive than most coaches in the NFL.  Teams that are behind are often more aggressive to catch up.  Therefore, teams who are behind more often (and lose more often) will naturally make more ‘optimal’ decisions.

Makes sense to me, but I was all excited to start the Norv Turner is better than Bill Belichick bandwagon.

I’ll have to put that on hold for now.

Implications

So what have we learned from all this?

We know that coaches are far too conservative, particularly when faced with short yardage situations in opposing territory.  They kick the ball far too often rather than trying to convert.

Our data is pretty clear on this point.  Coaches only make the right call 15% of the time when they’re supposed to be going for a first down, and when they’re in opposing territory, they make the right call less than half the time.

Well, the natural question, is why?

It can’t be a lack of information.  NFL coaches and management have all kinds of data at their disposal (certainly more than I do), and plenty of talented folks.  And it’s not like these ideas haven’t made it into the mainstream.  High school coaches have made news for never punting, and analysts continue to harp on the conservative tendencies at the NFL level.

At the end of the day, I still maintain that it all comes down to risk aversion.  NFL coaches (and most professionals) have one primary goal.  To stay employed.  And taking a strategy that goes against conventional wisdom exposes you to criticism if the outcomes don’t work out.

I read a quote from Mark Cuban not too long ago where he felt the idea that coaches wouldn’t try anything to win games was laughable.  If I remember correctly, he thinking was that professional sports are so competitive and the need to win so great that of course coaches take every opportunity they can to get better. (and in googling around, I can’t seem to find it, so maybe I’m misremembering)

Cuban certainly has more experience in professional sports than I do, but I think at least in the NFL the data clearly suggests that’s not the case.

Coaching at the NFL level is the highest professional position a football coach can ever get.  It typically takes years and years of work in all kinds of low-paying jobs (what does a quality coach even do?) and moving from city to city with the hope of one day snagging one of those 32 openings.

Oh, and once you get one of those slots, you can’t screw up, because you’ve spent your entire life specializing in a sport with a fixed number of teams and exactly one relevant professional league.  You’ve invested your entire career to get to the top of the pyramid, but there’s nowhere to go but down.  You’ve got to stay up there.  It’s not like you can go coach pro baseball.

So with a lack of transferable skills outside of football, and the inability to create a startup NFL franchise to coach, NFL coaches are in something of a bind.  Unless they have bulletproof job security (and in the long-run, no one has that, right Coach Reid?) they have a clear incentive problem to try such an easily observable strategy.

Why?  Because it’s quite possible the outcomes won’t work out, and people will crucify you if they don’t.  See the criticism Belichick got for his 4^th down decision against the Colts some years ago.  He made the right decision, but since the outcome didn’t work out, all of a sudden he made a terrible mistake.

It’s the problem of evaluating decision-making skill based on the outcome.  The equivalent would be telling a poker player he screwed up when he got all his money in with the best hand and someone drew a miracle card to beat him.

If you made all the optimal decisions you could make, and let’s say you were running at a 50% pass rate before, you might do something differently on ~30 plays a year.

Some of those plays might work, but some might not, and the ones people will focus on will be the ones in the situations of highest leverage where the outcome of the game hangs in the balance.

Now, I think making more ‘optimal’ decisions could swing a game or two a year in your favor, but it could also swing a game or two against you (like Belichick and the Colts).  When faced with that possibility, it’s no surprise coaches aren’t chomping at the bit to test out the theory.

For a coach to successfully try this (and it’ll only take one to succeed before others join), I would argue one of three scenarios needs to happen:

–        An edict needs to come down from the owner themselves, mandating the change to football strategy (of course, what kind of coach would want to be in that kind of situation?)  The most likely suspects would be new analytically inclined owners like the Kahn family in Jacksonville, or owners that like to involve themselves in football operations (Jerry Jones, Dan Snyder, the late Al Davis)

–        A coach fresh off a super bowl win uses the capital/credibility from his victory to test it out (you could imagine Belichick doing this, maybe Sean Payton still has enough cred, but I think you pretty much need to be a Harbaugh)

–        A brand new coach in the first year of a contract just let’s it rip and goes for it, putting their NFL credibility at risk because they at least could always go back to college

If you think I’m suggesting Chip Kelly go for it in his first year with the Eagles.  You’d be absolutely correct.