Time for another guest post from Jared.  You may remember him from:

– One of the Coolest NFL Charts Ever

– The Fourth Down Decision Series and Cheat Sheet (Part 1 of 4)

– When does it make sense to return a kick?

Well today, he’s ready to unveil at least the first iteration of his Hot Seat Index, and attempt to predict which coaches will be fired at or by the end of the season.  You can follow him @jaredscohen and at kebertxela.blogspot.com.  Without further ado:

————

So the Eagles had another especially depressing loss to the Giants this week. And with such a depressing loss, it got me thinking of another depressing outcome, watching your team stink to the point where they fire their coach.

So, this was something I wanted to wait on until much later in the season, but with Greg Schiano working as hard as he can to get fired as soon as possible, I needed to put it out a bit early.

One of the things I hate hearing about as the NFL season progresses is all the discussion around the coaching ‘hot seat’. It’s not that I mind the speculation, but all of the discussion centers on conjecture and the occasional ‘anonymous source’ (the exception being Schiano where the sources have noticeably opted to forgo anonymity!)

As I was thinking about it, I really wanted to see if we could make this a bit more objective and data driven. So with that as the goal, I gathered some data, did some analysis, and am now ready to introduce the NFL Coaching Hot Seat Index!

The Coaching Hot Seat Index is a model that, at any point in the season, will give you the approximate odds of an NFL coach getting fired after the season!

It’s based on a collection of data on all NFL coaching seasons since 1980. I broke the data down and for each coaching season, identified whether a coach was fired or kept their job (with some adjustments for retirements etc.)

With over 900 observations (and ~200 firings!), I started looking for factors which were significant in predicting whether coaches got fired or not. I tried lots of things, number of wins, playoff appearances, having the last name ‘Kotite”, all to see what was really significant in predicting when a coach will get fired.

I ended up with two primary factors, which were by far the most significant in predicting a coach getting kicked to the curb.

– Total point differential (points scored-points given up)
– Change in team wins from prior season (current year wins-prior year wins)

Those shouldn’t come as a huge surprise. (Note: I’m disappointed because I couldn’t examine another factor I wanted to see, the difference between Expected Wins at the Season Start and Actual Wins at season’s end…I wanted to use gambling lines to get at it but didn’t have nearly enough data to look at it. I still think it would be a more compelling variable)

But anyway, with those two factors, and using (or, abusing) a technique called logistic regression, I arrived at an equation to give us the odds a coach will get fired. Logistic regression is basically something you can use to help predict the likelihood of a binary outcome (a coach either gets fired, or he doesn’t) based on some variables (in this case, his team’s point differential and win change from prior season).

In the end, the result is the percentage chance (out of 100%) a coach will be fired after the season. The higher the odds, the more likely the coach is going to be filing some unemployment papers right around the Pro Bowl.

But I couldn’t just create a model and throw it out there without testing it a little. So I created a first draft of it, using only data from 1980-2011, and used that model to ‘predict’ the 2012 season based on its point differential and change in team wins (which, obviously, I already knew). When that passed the sanity check, I updated the model for 2012 and used it on teams from this year.

Those results are included below, with coaches sorted by their ‘odds to be fired’

Not a bad first result. Romeo Crennel, Mike Mularkey, Andy Reid all got canned, and those were the coaches with the highest likelihood. The model obviously isn’t perfect, as the plenty of coaches can still get fired, but at least there’s a structure and some logic here (we can also rank the coaches from safest to least safe!!!)

So this is interesting, and now we can apply it to the current season, and see what 2013 coaches are most likely to be fired!!!

Now, I had to make a couple of assumptions, because this model is based on a full season of performance, which of course we won’t have. But we do have a good sample and some reasonable projections of performance we can use as proxies.

For point differential, we can use a teams’ current point differential and pro rate it out for 16 games. This doesn’t account for changes in either a team’s performance or strength of schedule, but it’s not unreasonable.

For win difference from last year, we can take an estimate of the team’s full season (which I’ve borrowed from Football Outsiders Playoff Odds report, which runs simulations to calculate average team wins) and then just check the difference from 2012 wins.

So, at this point in the season (Week 7, because Week 8 is going on as I write this), which coaches have the highest odds of getting fired?

Well – it’s no surprise to see the Jaguars on the list, but we may have to make an exception for first year coaches as they typically get more than one season to right the ship. We’ll give Gus Bradley a pass (although no one can argue their performance is historically bad…it’s no wonder the odds are higher than anything seen in 2012)

Tom Coughlin checks in at number 2, although we’ll unfortunately need to update that after the Eagles managed to lose to them this week. Their performance has been really bad as well, and any coach without Super Bowl rings would likely be on their way out. But Coughlin may overcome the odds with all the goodwill he’s built over the years (or maybe he changes his mind and “retires”)

Next comes Schiano, where I think we can all agree the odds calculation actually UNDERRATES his odds of being let go. This guy might want to think about booking some tee times in late November if he continues at his current pace.

But the next coaches on the hot seat, Gary Kubiak, Leslie Frazier, and Mike Shanahan, should also probably get their acts together if they want to stay employed.

From an Eagles perspective, Chip Kelly doesn’t seem to be in too much danger (although as I said, we probably should eliminate all first year coaches as a general rule). Of course, this assumes Kelly behaves competently, unlike the absolute sh*t show we just saw against the Giants (which my brother is probably already dissecting)

Of course, from a FORMER Eagles perspective, it looks like Andy Reid has done quite alright for himself in the move to Kansas City.

I’m just saying.

This is perhaps only indirectly related to the strategic post I’ve been looking to expand upon, but it’s important nonetheless, and, in my mind, more immediately relevant.  The topic, as the title suggest, is Expected Points, the concept developed by Brian Burke at AdvancedNFLStats.com.

I’ll spare you the full explanation because I think most readers here are aware of it, but basically it assigns a point value to each down/distance combination to provide a measure of how valuable each situation is.  For example, 1st and 10 at the 50 yard line is worth MORE than 1st and 10 at a team’s own 20 yard line.  Simple enough.  However, many people (myself included) have been a bit to cavalier in using the Expected Points concept to evaluate in-game strategy.

Today, I want to clear that up.  There is a major limitation to the Expected Points concept; that is, it’s an AVERAGE, and it doesn’t account for the relative strengths and weaknesses of each team.  Logically, a certain field position is worth more to the Broncos than it is to the Buccaneers, right?  Similarly, a particular field position is worth LESS against the Chiefs defense than it is against the Eagles.  Unfortunately, that’s not accounted for, limiting the usefulness of the Expected Points analysis.

I’m not saying it’s worthless, far from it in fact; I’m just saying that we need to remember that the EP analysis for a given situation reflects average teams, and therefore must be adjusted when applying it to real-world situations.

Allow me to demonstrate (as you knew I would).

Here is the chart of EP value for 1st downs.

The problem, as I mentioned above, is adjusting for relative strength.  To explain, we need to pick 2 teams.  For the sake of clarity, and consistency, we’ll use Denver and Jacksonville (FO’s best and worst teams by DVOA).

Let’s just look at Denver’s offense.  To adjust, we need to know how the Broncos offense compares to average.  Luckily enough, Football Outsiders provides us with a measure of just that.  Note that, for now, I’m knowingly glossing over the fact that FO’s DVOA might not be the best measure here.

Anyway, Denver, according to FO, ranks 40.9% better than league average.  Well that makes things easy, right?  All we have to do is increase the average EP value of each yard line and re-graph.  Well here’s that graph:

I’m guessing you all see the problem, but if not, hang in there, I’ll get to that in a second.  We ALSO have to account for Jacksonville’s relative strength/weakness on defense.   Just as I did above, I can just use FO’s rating (17.9%) to adjust again, right?

Well here it is:

Perfect….now we have the adjusted EP value for the Broncos versus the Jaguars.  We can do the same calculations as before, using these values, to determine the “optimal” play-call (where optimal means maximizing expected points).

Only…there’s still that problem I mentioned above, which by now EVERYONE has noticed.  Looking at the graph above, it’s pretty clear that we’ve made a mistake in our analysis.  Touchdowns are only worth 6 points, which means no field position can be worth more than that (violated above).  Moreover, since, regardless of relative strength, scoring a TD can never be 100% assured, we shouldn’t even see a value of 6 anywhere on the chart.

Basically, as the mismatch becomes more and more favorable to the offense, the line should approach a limit of 6, but never reach it.  Unfortunately for us, that complicates our plans.  How do we account for this?

I’m not sure, but I do have one potential out.  

We can ignore everything I did above (ugh) and go back to using the average values for EP.  Rather than accounting for the relative strength here, we can instead adjust the expected success rates to account for relative strength.  That raises it’s own issues, but it seems to be more intuitive.

HOWEVER, that only addresses the problem when we’re using a combination of EP and Success Rates to game out the Optimal Value of a certain situation.  It does not address the issue when we are only using EP (which is how many analysts are using it.)

That brings me back to the main point: be careful when using Expected Points to justify in-game strategic decisions.  It can be done (and EP remains the best model for quantifying such situations), but you have to remember that the actual results will be distorted by the relative strength of the teams.

If that reminds you of this E = R ((60 – T) / 60) + C then I’ve done my job.  I apologize for the slap-dash way I’m addressing the overall concept, but it is what it is…I don’t have the luxury of taking the time to plan these things in advance.

I’ll try to return to this soon, hopefully using a real situation from Sunday to game out the options using our adjusted success rates.

Next up, though, we need to talk about Bayes and reconciling New Information (what happens during the game) with what we already “knew” (R before the game starts).

Eventually, I do believe it will be possible to create an algorithm that essentially tells you what the “optimal” decision is in every Go/Kick/Punt situation.  We’re kind of there already, on an average basis.  The key, of course, is to adjust for the teams involved in order to allow actually reliance and use (not that NFL coaches would ever admit that a computer can make better decisions than they can).  At the very least, it’ll provide a valuable guide with which to grade coaches.

This should probably be a permanent end note, but all comments are encouraged.  As I said, I’m trying to develop a useful model here, so suggestions are always welcome.

Let’s return to last week’s post, found here.

I want to start looking at this in smaller chunks, which will hopefully be a little clearer.  First, some overall major takeaways:

– E = R ((60 – T) / 60) + C, the formula, for reference.

– Large underdogs should be extremely aggressive early in games, when R (relative strength) is at its largest.

– Underdogs should attempt to use as much of the clock as possible.  This is more “conventional” and something I didn’t talk about last week, but it’s a logical extension of what I was talking about.  If you have two very mismatched teams, and make them play 100 games, it’s almost certain that the “better” team will win more than it will lose.  The larger the sample, the more likely it is to reflect that actual “relative strength”.  By using up a clock, the underdog is limiting the sample size “# of plays” from which the relative strength advantage can play out.  Using our formula, by bleeding the clock, underdogs are attacking the R value indirectly, using T, instead of going at R itself (scoring points).

– During the game, strategic decisions should incorporate an objective view of how the rest of the game is likely to play out.  For large underdogs, this means they should expect to be outscored, and therefore need to be aggressive in scoring points.

Favorites Strategy

I didn’t discuss how this effects the strategy of the Favorite.  In the most simple reading, it can be assumed that the Favorite should be more “conservative”.  Going back to our Broncos vs. Jaguars example, 3 points is a lot more valuable to Denver than it is to Jacksonville (hence “not all points are equal”).  Therefore, given the same FG opportunity as the one we gave Jacksonville (expected points for FG and going for it are equal, purely a risk/reward play), it should elect the LOW risk option (the FG).

That’s because, as I explained above, at any time T, the favorite can expect to outplay the underdog over the rest of the game, i.e. the R value is advantagous.  As time goes on, this becomes less of a factor (T declining ultimately takes the R half of the equation to 0).

In general, I agree with this.  Large favorites should be content to take whatever points they can get, early in the game.

However, there is a slight wrinkle, one that will appeal to the more aggressive fans.  Let’s go back to our graphic for a moment.  Here is the range of outcomes at the start of the game:

As you can see/remember, if we assume a “random walk” from there, Denver should expect to win a very large percentage of the time.  There is a case to be made for being aggressive, though, and hopefully you can see it.

It goes back to when I explained that you can actually “win” the game before the game is over.  Assume the same EP-Neutral opportunity above, but this time imagine that gaining 7 points is enough to shift the range of outcomes (yellow shade) entirely above the X-Axis.  Would you go for it or kick?  Probably go for it, right?  After all, if you have a chance to “win the game”, with relatively low risk (still have a heavy advantage if you don’t convert), you should do it.

Obviously, I have to note that this is a purely theoretical situation.  During the game, it’s not possible to know EXACTLY where the range of expected outcomes lies.  Therefore, we can’t be sure of where the line between 100% win and 99% win is, even if some of us see that final 1% as extremely valuable.

Still, it implies that there are some situations, even if they are hard to identify, where the Favorite should also be aggressive. In general, though, it should take the lower risk strategic options, because it does NOT want to significantly shift R (outside of the specific scenario I just outlined).

Random Walk

I don’t think I made a big enough point of this model in the post last week.  There are two ways to view the game, ex-ante, and I think one of them is much better than the other.

1) This is the normal model.  Teams start on even ground (Score tied 0 – 0) and we “expect” the course of play to naturally favor one team (the favorite) over the other (the underdog).  During actual play, we project that the difference in skill will gradually manifest itself in the score, and ultimately mean victory for the Favorite.  That’s the usual way of thinking about it.

What I’ve done is to flip that around a bit.

2) Teams start on UNEVEN ground (R value), and from there we expect a random series of events to occur, though they will be within a range of possible outcomes.  This certainly isn’t the “natural” way of thinking about things, but it appeals to me for one very big reason.  Can you guess what that is?

I like it because it forces us to accept and recognize the large role of luck and chance in the outcome of the game.  Future human events are inherently unpredictable, right?  So how do we reconcile that with the first option I outlined above (the normal model)?  Isn’t it explicitly forcing us to predict that which is, by its very nature, unpredictable?

The result of this is that we get ridiculous explanations for unexpected outcomes of games.  For example, take the Giants-Patriots SB (Helmet Catch).  The Patriots were heavy favorites, and yet lost a close game.  Why?

– Is it because Eli Manning is just REALLY clutch?

– Is it because the Patriots “choked”?

– Is it because the Giants have more “heart”?  or “wanted it more”?

Of course not, those are all ridiculous explanations, and yet they’re a natural outgrowth of the way we normally think of games (option 1).

Now let’s look at the “Random Walk / Ex-Ante Relative Strength” model (the name needs work).  Here’s the picture again, just imagine a Patriots logo instead of the Broncos and a Giants logo instead of the Jaguars.

Suddenly there’s no explanation needed for the outcome of the game.  Just look at the picture; you can see there’s a section of the yellow shaded area below the X-Axis.  If we assume that at time T=60, all future game events will take a random path through the yellow area, then it’s obvious that SOME of those infinite paths will end up in the area below the X-Axis.  It just so happens that THIS PARTICULAR run was among those.

Now there’s also obviously some unpredictability in deriving a value for R.  It’s very difficult to know just how good each team is and how they match up against each other.  However, I’d argue that all of the necessary information for getting an accurate R value is theoretically knowable.  Compare that the Normal Model.  It requires us to predict future events, which is NOT POSSIBLE, even in theory.

The upshot of the “Random Walk” is that it forces people to confront a lack of “control”.  It basically boils the game down to a lottery.  That sucks some of the fun out of it, but that doesn’t mean its a less accurate model of analysis.

Similar to last time, I’m going to cut this off prematurely for the sake of time and clarity.  Hopefully you’re still with me.