Today, let’s look at some other teams’ performances from the last decade.

First up, the Patriots.  Remember the yellow highlights are SB winners, the red highlights are SB losers:

An extremely impressive decade all around (obviously).  The defensive performance is perhaps a small surprise, since the offense gets so much attention.  One of the most puzzling aspects of the Brady-Belichick run has been the lack of press attention on the defense.  Notice the Patriot defenses of the last three years don’t come close to approaching the defensive performances of the Super Bowl winning teams from 2003 and 2004.

Also, notice the Patriots 2003 title came with a team that was just 5% better than average on offense.

The Saints (otherwise known as the Drew Brees effect):

I like the Saints’ table because it clearly illustrates just how quickly teams can change their fortunes with a new coach/QB combo.  Look at the offensive side of the table above; can you see when Payton/Brees joined the team?  I thought so.

It’s also interesting to see how badly the defense has trailed the offense.  If the Saints’ defense was just consistently average, the team might have played in another SB or two.

The Giants (sorry):

The Giants’ performance further highlights the relative randomness of the two Super Bowl titles.  In the past ten years, the Giants have had two REALLY good teams, 2005 and 2008.  Naturally, neither of those teams went to the Super Bowl.

The 2007 title team remains perhaps the most puzzling SB Winning team in more than 30 years (maybe ever, but I prefer to stay within my lifetime).  Not only was the 2007 team remarkably average, but they defeated the greatest team in modern NFL history.

If you want a great example of the “any given Sunday” saying, compare the performance ratings of the 2007 Giants with those of the 2007 Patriots (in the table above).

Want to feel better?  Take a look at the Raiders:

No comment needed other than to say the past decade in Oakland has been one of ASTONISHING futility.

Lastly, any time you’re disappointed or frustrated by the Eagles, take a deep breath and say “thank god I’m not a Browns fan”.  Check out the offense:

Went to the U.S. Open practice round yesterday, hence no post.  Biggest takeaway?  Players are going to score, but only those that avoid the rough at all costs.  My wrists hurt just watching players try to get out.

Today’s chart stems from a commenter’s suggestion.  I’ve graphed the offensive and defensive performance (league-average adjusted) for all NFL teams from the last ten years and highlighted the Super Bowl teams in Yellow.  It’s a little tough to read, so to be clear:

The X-Axis (left-right) is OFFENSIVE performance.  So the farther RIGHT a dot is, the better the OFFENSE.

The Y-Axis (up-down) is DEFENSIVE performance.  The HIGHER the dot is, the better the DEFENSE is.

So the best teams are in the upper-right quadrant (the 1st if my high-school geometry memory is correct).  The worst teams are in the lower-left quadrant (the third).

I did not label the Eagles teams, since doing so crowds the graph too much.  However, below is a table showing the Eagles’ performance going back to 2000 (note the graph only goes back to 2003).  The 2004 team is highlighted red because they lost in the Super Bowl.

Looking at the chart above, we can clearly see the teams that won with great offenses (Saints) and those that won with great defenses (Steelers ’08).  We can also see that the 2004 Patriots were the most balanced Champion (best?) by a fairly wide margin.  Also, the 2007 Patriots are the blue dot by itself on the right.  In terms of regular season performance, that was easily one of the greatest professional teams ever, regardless of sport.

While comparing the Super Bowl winners above is interesting (clearly illustrates the relative strength of each team), I’d also like you to take a look at the axis scales.  Notice the offensive axis (left-right) has a much higher maximum than the defensive axis.

No team in the last ten years has finished with a defense more than 40% better than league average (Points Against).  Conversely, over that same time period, 11 teams have finished with an offense at least 40% better than league average (Points For).  Clearly, though, a great offense does not lead to a Super Bowl win with any certainty.

If we go back to the post earlier this week.  We can see the visual illustration of the necessary-condition strategy.  Basically, to win a Super Bowl, you MUST be on the right side of the Y-Axis (average offense or better).  The same rule does not hold for defensive performance.  However, it also appears as though, for Offense, the point of diminishing returns is reached rather quickly (i.e. going from -5% to 5% has a much greater effect on winning than going from 40% to 50%.)

Lastly, here are the Eagles.  I’ve listed them in (X,Y) order so you can easily place them in the graph above.  Remember, Off (X) is right-left, Def (Y) is up-down.

Today we look at the defense.  Here are the best defenses since the 2000 season, adjusted for league average scoring.  Yellow teams are Super Bowl Winners, Red are Losers, and I’ve highlighted the Eagles in Green.

Whoa…a lot more yellow than the offensive table.  Also, we can clearly see the Andy Reid Peak here.  While McNabb was the face of the franchise, the defense is what really carried the best Reid teams.

Again, this isn’t much different from our pervious look (which didn’t adjust for points inflation).  However, it seems pretty clear from that table that a GREAT defense can go a very long way towards winning a Super Bowl.

Notice also that the 2002 Bucs are at the very top of the list.  I’ve said before that the 2002 Eagles were the best of all the Andy Reid teams.  Unfortunately, it ran into a historically great defense that year.  Is it possible that single game led to a Reid overreaction and strategic shift away from the defensive focus that led to so much success?   Yes, yes it is.

After that game, everyone talked about how inept the Eagles offense was.  Meanwhile, it’s quite possible (likely?) that the Bucs that year would have shut down ANY offense.  They were that good.  However, I’ve talked about that before.  Back to today’s topic…

Similar to yesterday’s table, here are the past ten Super Bowl winners and their defensive performances:

For reference, here is the offensive chart for Super Bowl winners:

The performance of teams in both areas is strong (as it should be), but it’s clear the bar for defensive performance is significantly lower.  Not only do we have 3 teams that Won a title with a below-average defense, we have 2 teams that won with a significantly below average defense.

Also interesting to note is the fact that the only Super Bowl winner with a negative offensive performance (the 2008 Steelers) had a historically great defense (37% better than league average).

What about the Super Bowl losers?

Strong performances again. In fact, the average performance of the losing teams is better than the average for the winning teams.  However, we also see that, while defense is clearly important, it is not necessarily mandatory.  The Arizona Cardinals of 2008 were 21% below league average on defense and still managed to come within a Santonio Holmes tip-toe of winning the Super Bowl.

Lastly, here is a chart of the past 10 Super Bowl Winners showing their defensive and offensive performance.

At first glance, it doesn’t look like a huge difference, though average offensive performance is better by 3%.  However, if we look at the minimums on both sides, we get a clearer picture.

After all that, our original thesis stands:

– An average offense (or something extremely close to it) is required to win the Super Bowl.

– An average Defense, while certainly helpful, is NOT required.

THEREFORE

Teams should focus on the offensive side of the ball until they’ve met the average threshold, and then turn towards maximizing defensive performance.

P.S. The table above also makes it extremely clear that the Giants teams that won in 2007 and 2011 are, BY FAR, the weakest winning teams of the last decade.  I’m not sure if that makes me feel better or worse, but it supports my assertion that those teams were largely lucky, and not good (not that it matters).

If you remember back a couple of months, I did a couple posts that went into necessary vs sufficient conditions regarding the construction of a Super Bowl team.  At the time, I looked at every team from the last 10 years and used their PPG and PPA to gauge their relative percentile ranking.  However, since then I’ve gone back and adjusted each team’s performance to account for the league average PPG that season.  This adjustment will help account for the general offensive inflation the league has seen over the past decade.

Today, we’re looking at the offensive side of the ball.

The question is, how good are Super Bowl winning teams on offense?  The idea here is to get an idea of what a truly optimal team construction strategy would look like.  There is a salary cap, and the number of roster spots means every team must sacrifice somewhere in order to improve a different area.  What is the best mix of offense and defense?

My last look came to this general conclusion (ignoring Special Teams for now): Teams should focus on building an above-average offense.  Once that’s assured, the team should focus 100% on developing the best defense possible.

After adjusting each team’s performance, does that still hold?  And if so, what does it mean for the Eagles?

First, let’s revisit the best offenses.  My main data set only goes back to 2003, but for this, I went back to 2000 to make sure I was including some all-time greats (to give us a sense of just how great they were).  You’ve seen this before, but here are the best offenses in recent history:

The teams highlighted Red LOST in the Super Bowl.  Teams in yellow WON the Super Bowl.  We can see the overwhelming dominance of the Patriots, as well as some less-heralded performances, like the 2011 Packers.

However, we also see a lot of red.  Just as in our original look, it seems that a great offense can go a very long way towards winning a Super Bowl, but can’t guarantee a win.

We do have to be careful with the sample size here.  5 of the top 16 offenses since 2000 went to the Super Bowl.  The 1-4 record of those teams in the final game may just be bad luck.

How about all the Super Bowl winners?

Here are the winning teams from the last 10 years:

As you can see, the only team to win a Super Bowl in the last ten years with a below-average offense was the 2008 Steelers, and that team was just 1% off the mark.

Additionally, the average offensive performance of Super Bowl winners is +16%.

It appears as though our previous conclusion, at least on the offensive side of the ball, stands.  An above-average offense is close to a necessary condition for winning the Super Bowl.

That should be good news for Eagles fans, since it explains why the focus of this offseason has really been on Offense, despite the terrible defensive performance of last year.  Given Chip Kelly’s background and skill-set, meeting the league-average offensive threshold SHOULD be close to guaranteed for the Eagles, if not this year, then soon (likely depending on the QB situation).

But wait! What about the Losers?

If there have been Super Bowl LOSING teams that did not have a strong offense, then we may just be over-extrapolating based on a what is likely the result of chance.

Here are the LOSERS from the past 10 years:

Combining this chart with the Winners chart shows us that just 3 teams have even made it to the Super Bowl with below average offense in the past 10 years, and the worst among them was San Francisco this year.  The 49ers were just 4% off the league-average mark.

Tomorrow we’ll look at the defense, but it appears as though our original conclusion not only stands, but looks stronger.  You cannot win the Super Bowl with a bad offense.  Not only that, but you can’t even MAKE IT without a league-average offense.

Now that summer is here, I’ve decided to scale back the posts a bit.  Ideally, this will mean continued daily posts, though of a shorter variety.  There’s less relevant information to discuss, and I’d rather not just ramble every day (I try to make every post interesting/thought-provoking/or in some other way valuable).  However, that may mean an occasional day without a post; I know you’re all heartbroken.  I’ve assembled a lot of data and want to do some higher level things that require more than a few hours work.  On days without posts, you can rest assured that I’m spending some time on these larger projects.

We’ll obviously ramp back up as the season approaches.

————————-

In the meantime, I’m working on my articles for the Almanac.  Here are a few notes to come out of that:

– Eli Manning has a career Passer Rating of just 82.7 and a TD/INT ratio of 1.47.

– Jason Campbell’s career Passer Rating is 82.5 and his TD/INT ratio is 1.46.

– Donovan McNabb’ career Rating is 85.6 and his TD/INT ratio is 2.0.

Eli Manning is likely headed to the Hall of Fame.  You may commence vomiting now…

——–

There is no statistic more important for evaluating college QBs than completion percentage.  I’m straying dangerously close to my Almanac stuff now, but obviously that will be more detailed.  For now, I’ll just give you a chart, with Pro Passer Rating on the Y-axis and College completion % on the X-axis:

The correlation value is a moderate .324.  Given the difficulty of projecting human performance in addition to all the other variables involved, that’s actually an extremely strong indicator.

Kyle Boller had a college completion percentage of just 47.8%.

He was selected #19 overall in the 2003 NFL draft.

27 Pro Bowlers were selected after him (7 more went undrafted, including Tony Romo).

Bonus post before I go on vacation, and this one is awesome.

Remember that area chart I showed you that illustrated how the Eagles playmakers changed over time?  Well I gave Jared (who did the 4th down series) my complete data and he put this together.  Enjoy.  (Sorry you have to click to open it, WordPress apparently doesn’t support Java, so I can’t embed it).

BTW: It’s interactive, play with it…

I hope you all enjoyed the 4th down series.  Thanks again to Jared for doing the research.  Today I wanted to give a few thoughts of my own about the data and its implementation.  (Go read if you haven’t yet, or see the 4th Down tab above for the strategy chart).

It will surprise nobody that I come down on the side of being more aggressive.  The simple fact is that Coaches have been PROVEN to make sub-optimal decisions in certain situations.  While we don’t know for sure why this happens, I agree with Jared that the most likely reason is essentially “groupthink” or a “herd mentality” along with slightly misaligned incentives.

The coach is incentivized to KEEP HIS JOB, not to win.  Normally those things go hand in hand, and it’s very difficult to keep your job if you don’t win.  However, in certain game situations (for example when a team is losing by a lot) coaches clearly make decisions that aren’t aimed at maximizing the odds of winning, like kicking field goals to minimize the margin of loss.  Additionally, the “optimal” decision for coaches is NOT “whatever provides the greatest chance to win”.  It’s more complicated than that.

The “optimal” decision, given the coach’s incentives, is one that achieves TWO goals; win the game, AND minimize criticism of said coach.

Looking at the results, I do not believe all of these coaches are ignorant of the statistically “optimal” decisions.  Some likely are, but given the amount of money at stake and the number of very smart people in league front offices, you can be sure at least a few coaches realize what they’re missing.

The upshot is that this represents a potentially large INEFFICIENCY in the way the game is currently played.  Some day a coach will take advantage of it.  However, note that just because you play the odds correctly doesn’t mean you’ll be rewarded.  This may be another reason for coaches’ reticence.   This “aggressive” strategy WILL WORK, but not every time (as several commenters have noted).  The benefits will only be clear after a LONG time.  Most coaches don’t have the job security to wait that long while being criticized by beat writers for whom anything with a decimal is considered “analytics”.

While I hope (and expect) Chip Kelly to be among the more aggressive coaches in the league, I think it’s EXTREMELY unlikely that he makes a significant departure from what we see now.  At the end of the day, Chip wants to keep his job.  Unfortunately, such incentive misalignment, however slight, inhibits the pace of innovation in the sport (as it does in many industries).

I will certainly keep an eye on Chip’s 4th down strategy and we’ll discuss it here during the season.

I’d also like to address the points made by a few commenters about the overall utility of something like the 4th Down Strategy Chart.

– First, as explained in the first post, each team would, in practice, adjust the chart to account for the relative strength of the opposing defense.  This is not a one-size-fits-all chart.  However, given the HORRIBLE success rates, its pretty clear that team-to-team differences are not accounting for the overall results.

– I would not, though, blindly follow that chart.  The research explicitly excludes end of half and end of game situations.  TIME REMAINING becomes a huge factor in those cases, completely altering optimal strategy.

– I would, however, ALMOST NEVER PUNT with less than 2 yards to gain on 4th down.  It is extremely difficult for a defense to stop the offense from gaining just 1 yard.  I get the sense that many people don’t realize just how small a distance that is.  Today’s homework is to grab a ruler and measure out three feet (a yard).

Also, let’s attack this psychologically.  Think back to last season or picture yourself during a game. Your defense has just forced the offense into 4th and 1.  Are you hoping for a punt? Or are you hoping the offense goes for it, so that your team can stop it and gain “momentum”?

I don’t care where on the field that situation takes place, most people are hoping for a punt (as is the defense!).

In general, if you (as an offense) are doing things the defense WANTS YOU TO DO, you’re doing it wrong!

Several people have mentioned the “momentum” surrendered by going for it on fourth down and not converting, and while I think the concept of “momentum” is largely exaggerated (though not nonexistent), you must also factor in the demoralizing effect that converting has on the opposing defense.

– Even on your own 1 yard line, I’d strongly consider going for it on 4th and 1.  The median NET punting average for the league last year was approximately 40 yards.  Using this number tells us that if you punt from your own goal line, you can expect the other team to start its possession around the 40 yard line.  For some teams, that’s already in field goal range.  For everyone else, it’s just a few yards outside.

So, if you punt from your own goal line you are essentially giving the other team 3 points, with the potential for 7.  If you go for it and fail, in all likelihood you are giving the other team 7 points.  However, if you go for it, you have decent odds of converting, meaning you’ve now add the possibility of scoring 7, scoring 3, and allowing 0 points to the situation.

At a high level, going for it sounds like the better option to me.  Now that I have last seasons play-by-play data (procured last weekend), I will take a look and see if that’s actually the case.  Our 4th Down Chart suggests it is.

– I would ALMOST NEVER punt after crossing midfield. Unless it’s a late-game situation or there are a large number of yards to gain (8-9+), IT DOESNT MAKE ANY SENSE!.  You’re already passed midfield, meaning you’re not guaranteeing the other team any point if you don’t convert.  This is where I expect Chip to be aggressive.  It’s a more “defensible” decision and less likely to immediately back-fire.  That means the reputational risk is minimized, allowing the coach to weigh the “win the game” side of his incentives more strongly.

– Lastly, I completely agree with the chart regarding 4th and 4 or less yards to go situations in field goal range.  Kicking a field goal when you’ve got 4th and 1 is ridiculous (unless its late in the game or time’s running out in the first half).  It’s 1 yard, go get it.  It’s the statistically optimal decision, and 7 points is a LOT more impactful than 3.  For those of you who buy into the “momentum” game, how much does 3 points get you?  Close to none… Kicking a field goal with less than 4 yards to gain is a gutless (and stupid) decision.

That’s all for now.  I’ll be on vacation, starting tomorrow and running through next week.  So probably no posting.  I encourage you to explore the archives though, I’ve tried to make it as easy as possible by giving you tools and shortcuts on the sidebar.

If you missed part one (posted yesterday), I encourage you to read it before moving to today’s continuation.  At the end of yesterday’s post, we arrived at a default 4th Down Strategy chart, essentially a cheat sheet that tells coaches when to go for it and when to punt/kick a FG.  For future reference, I have added the chart as a permanent fixture that can be accessed through one of the menu tabs at the top of the site.  That should make watching the games more fun (or frustrating since you’ll know in real-time when bad decisions are being made).

Today, we move to grading.  Using that chart, how do NFL Coaches perform?  As I mentioned yesterday, this research was done by Jared Cohen, you can follow him on twitter at @jaredscohen.

Fair warning, this is a very detailed analysis (more of a research project) and therefore is longer than the typical blog post.  Please read it when you have some time.  If that’s not possible, feel free to skip to the charts.

Part 2

Methodology

To examine 4^th down coaching decisions, I took the following steps.

1. Download a comprehensive set of all fourth down plays from the 2012 regular season, including a set of key variables I could track and control for, including:
   1. Distance to go for a first down/touchdown
   2. Quarter and clock time of the play (e.g., Q1, 14:30)
   3. Field position (e.g., own 35 yard line)
   4. Scoring margin (e.g., team up by 3 points)
2. Each play was segmented by the choice of its coach as either a punt, FG attempt, or conversion attempt (by rush or pass)
3. Based on the distance for conversion and field position, I compared the fourth down play call to the optimal strategy matrix (the strategy card), to see what the ‘right’ choice would be
4. A play call in which the coach made the optimal decision was termed a ‘Pass’, while a play call that was not (e.g., punting instead of aiming to convert 4^th down) was termed a ‘Fail’

Pretty simple right?

Now, before getting to any data, I should also note that I excluded a number of specific plays, for reasons which I’ll explain.  Remember, the goal of the analysis is to determine whether coaching decisions are optimal under normal circumstances.  The key word here is normal.

–        If  a team was either leading or trailing by more than 14 points (two touchdowns), we excluded the decisions, reasoning that coaches would be making decisions differently than normal behavior (e.g., trying to catch up)

–        If the distance required was longer than 10 yards (e.g., 4^th and 12 yards to go), I excluded it.  I did this largely because those situations usually aren’t decisions for the coach.  It’s a pretty clear field goal attempt or punt depending on where you are, and my major area of focus was on situations where a coach could decide to go for it

–        Plays were also excluded if they occurred in the last 2 minutes of the 2^nd quarter or the last 5 minutes of the 4^th quarter, as coaching behavior will also change significantly.  In the 2^nd quarter, it’s because a team can’t maintain possession.  In the 4^th quarter, it’s because the game is ending and teams will no longer be trying to maximize their total points, they’ll be more focused on gaining/maintaining a lead.

So while the aforementioned decisions could be interesting, they were in situations which are inherently not-normal.  The main goal is to see what coaches do in a typical situation.  Even after subtracting all these conditions, we have over 2,100 fourth down calls to evaluate.  That should be plenty.

So what did we see?

Results

We saw a pretty large number of failures.  I’d never want to play blackjack with these guys.

Below is a chart of the overall grade for NFL coaches’ fourth down decisions, by quarter.

Yes, you’re reading that right.  When making a decision as to what to do on 4^th down, the NFL coaching body as a whole makes the ‘optimal’ decision just slightly over half the time.

Think about that for a minute.  In just about half of all normal 4^th down situations, coaches are making decisions that fail to maximize their number of expected points (and we should expect, actual points).

That seems kind of strange.  And yet it also seems completely believable in a league where principally ALL coaches are far too conservative.

But if we spend some more time peeling back this coaching decision onion, we’ll look at a couple more specific cuts of the data that can give us more insight on exactly where these decisions are happening.

This will include:

–        Decisions by optimal decision (what kinds of decisions are the most frequently screwed up)

–        Decisions by field position (how do decisions vary by where you are on the field)

–        Decisions by yards to go (does optimal decision-making vary by distance)

–        Decisions by coach (which coaches appear to have the highest grades)

–        Decisions by scoring differential (does decision-making change when you’re ahead/behind)

–        Some fun with coaches (looking at specific game decisions to understand exactly what the implications are)

But before we get to that, there are a few caveats to all this analysis, which I want to make clear.  This is to head off complaints and anti-analytics folks who may have already commented about how I live in my mom’s basement.

1. This analysis accepts the illustrated decision matrix as optimal, when in reality, that may not hold completely.  It’s based on my interpretation of Brian Burke’s work, which I think is logical and is the leading model that I’ve seen. (I also ran these numbers with an alternative model generated for college football, and the results were consistent with expectations, which means they were much worse as college teams should kick field goals much less frequently than NFL teams do, hash marks and lower kicking talent level etc.)
2. These optimal decisions do not take into account the talent/performance of the teams in question.  It assumes equal teams are playing each other.  So could a team with a great offense merit different ‘optimal’ choices where they go for it more often?  Of course.  You could also adjust for the defense of your opponent, the skill of your kickers, the opposing punt return man, home field advantage, weather, or any recent lunar eclipse.  This doesn’t have any of those adjustments.
3. When we get into very granular cuts of data (specifically with coaches), we start to run into potential sample size issues.

All of this is to say that yes, there are concerns with this (or any) piece of analysis.  The goal of this isn’t to find incontrovertible proof, or establish new football dogma, it’s to investigate an issue and understand potential implications.

Analytics can serve as a helpful guide and show you where some issues might be, but I’m not going to pretend the conclusions are absolute.

Decisions by decisions (From the Department of Redundancy Department)

One of the reasons I wanted to look into 4^th downs at all was because I’m continually frustrated by coaches kicking field goals or punting.

Coaches are too conservative, just about any research in football has suggested as such, and so when I put this sample together, I wanted to see what my data looked like.

The first thing I did was filter all the fourth downs based on what the ‘optimal decision’ actually was.  Remember, each fourth down, based on our strategy chart, was either ‘Go for it,’ ‘Field Goal,’ or ‘Punt’, based on the action that would maximize your expected points.

So, if we sort our data by what the optimal decision should be, we can see whether NFL coaches are screwing up opportunities to punt, kick field goals, or go for it (hint – my money is on go for it)

As TMQ’s Gregg Easterbrook would exclaim, ‘Ye gods!’

The good news for NFL coaches, is that when they’re supposed to be conservative, they are fantastic about it.  In situations where coaches should be kicking a field goal or punting, they decide to do just that 97% and 99% of the time, respectively.  Amazingly optimal performance!

Of course, such high rates would suggest that our coaches are being extremely conservative, which probably seeps over into fourth downs where they should be trying to convert.  And sure enough, the abysmal 15% pass rate on fourth downs where teams should be going for it is exactly what we’d expect to see from overly conservative managers.

That means that, when faced with a fourth down where the best outcome is to go for it, coaches choose to kick away (punt or FG) about 85% of the time.

Wow.  That seems insanely high.  Making the wrong choice 85% of the time?  I feel like in most jobs those kinds of choices get you fired.  If you make 85% of the burgers wrong at McDonald’s, you’ll most definitely be out of a job.

But to give you a sense of what these failing decisions actually look like, I’ve included a sample from my data set below:

All of these are example ‘fails’ by NFL coaches when the best choice would be to go for a first down.  Some of them seem pretty obvious.  Pete Carroll and the Seahawks had 4^th and 2 from the Arizona 9 yard line and elected to kick a field goal (which by the way, they missed).  Some of them, like Dennis Allen opting for a FG from the Chargers 33 on 4^th and 6, seem a bit more arguable.

But across all those possible decisions, only 15% of them were the ‘optimal’ choice.  Even with my earlier caveats (not adjusted for team ability or game situation), that still seems like something is systematically wrong with NFL coaches.

Decisions by Field Position

So we know what type of decisions coaches mess up.  They’re too conservative and should be going for it more often.

But let’s keep going, and ask ourselves, are these decisions happening all over the field?  Are they happening more in some areas than others?

I broke down the field into four main zones, and looked at the data that way.

1. Own territory – Anywhere between your goal line and the 50 yard line
2. The ‘Maroon Zone’ – A term I borrowed as an homage to TMQ, who has consistently railed on over-conservative coaching for years.  My definition of the Maroon Zone is in opposing territory, but not as far as the opposing 35 yard line.  Too far for a field goal, but surely too close to punt!
3. FG Range – Any position between the opposing 35 yard line and the opposing 20 yard line.  From the 35 yard line a field goal would be 52 yards, which is more or less the regular range of today’s NFL kickers.  We could split hairs and more it back a few more yards, but this was where I decided to draw the line.
4. Red Zone – Anywhere from the opposing 20 yard line to the opposing goal line

So I set these bins and filtered the fourth downs.  Where on the field are teams making more suboptimal decisions?

I should’ve saved the ‘Ye Gods’ for this, huh?

One of the first things that jumps out is that coaches make the most optimal decisions in their own territory.  This makes sense, as we know our coaches are big on punting, and in their own territory, that’s more likely the right decision. (Of course, in an absolute sense, getting only two-thirds of decisions right isn’t exactly fantastic)

The other thing that jumps out is the performance in the Maroon Zone, so opposing territory but a bit too far for most field goals.  Coaches are only making the right decision about 32% of the time here.

Again, some of you might be wondering types of decisions this entails, so I’ve pulled a sampling of Maroon Zone decisions.  This table illustrates eight examples of Maroon Zone decisions from my data set, all from the first quarter of the first week of this season.  It includes the coach, matchup, down and distance, score position, decision (both actual and optimal), and grade.

Let’s take the first row, when Mike Munchak and the Titans, facing a fourth and 1 from the opposing 37 yard line, faced a decision.  You’ll see the Titans elected to pass, and that the optimal decision was to go for it.  For that ‘passing’ decision, they received a ‘1’ grade.

Contrast that with Mike McCarthy and the Packers in their game against the 49ers.  McCarthy had a decision on fourth and 3 from the 49ers 45 yard line.  At the time, the Packers were down three points.  With arguably the best quarterback in football and a stable of talented receivers, did McCarthy choose to go for it?

No, the Packers punted.  Now, you could argue that the 49ers have a great defense and field position is key and blah blah blah.  I’m not saying those arguments have no merit, I’m just saying I would’ve gone for it, and in that situation, going for it is the right move.

You may ask what happened on those plays?  What were the outcomes?  Did the Titans convert the first down?  Did the Packers punt give them great field position?

Well, frankly, I don’t care what happened.  Judging a decision based on the outcomes creates a whole set of biases which I don’t want to influence our analysis.  The goal of this is to understand whether coaches are making the right decisions, not whether those decisions ended up working out.  To me, that means we should keep the outcomes completely outside of this conversation.

Decisions by Yards to Go

So coaches aren’t going for it enough, and although they make the wrong decision most of the time whenever they’re in opposing territory, it’s at its worst when they’re beyond field goal range.

That’s interesting, if not completely unexpected.

But how is their decision-making impacted by the distance required for a first down?  Is there any difference to a coach’s decision making whether its 8 yards to go vs. 3 yards to go?

Again, not a shocking result.

When the optimal decision is a more conservative approach (like punting on a fourth and ten), coaches almost always get it right.

But as the distance to convert shrinks, performance gets remarkably worse, especially around 2-3 yards to go, when coaches are only getting it right one-fifth of the time.

Again, it’s the conservative approach that does them in.  When coaches should be trying for conversions, they’re punting the ball away or attempting a field goal.  What’s interesting is that with only one yard to go, they’re actually a bit better.  I feel like it’s a gap based on aesthetics more than anything else.   One measly yard? We can go get that! The coach may say to himself.  But push it back another three feet and it somehow becomes impossible.

Now we’ve seen just how bad NFL coaches (as a group) are when it comes to optimal 4th down decision-making.  Tomorrow, we’ll look at individual coaches to see who is the best (or least bad as the case may be).  The results are shocking…

Today is the first installment of a series of posts about 4th Down Decision-making in the NFL.  These are going to be a bit long, but extremely interesting for fans interested in the high-level analysis of the game (which should be just about everyone who comes here).

I’ve broken the entire analysis down into 3 parts:

Part 1 (today) will discuss the overall idea behind the analysis and build to a 4th Down Strategy Chart

Part 2 (tomorrow) will use that strategy chart to grade NFL coaches and show us how often the “right” decisions are made

Part 3 (Wednesday) will show the results for individual coaches and discuss potential implications of the entire analysis.

This was authored by Jared Cohen, with limited editing/input from me.  You might remember Jared from his previous posts here, specifically his timely explanation, shortly before the Super Bowl, of why running a kick out from the back of the end zone is usually a GOOD decision.  Of course, soon after that analysis, Jacoby Jones ran a kick out from the back of the end zone in the SB, resulting in a 109 yard touchdown…

Also, by way of qualifications, Jared is a:

– Two-time Jeopardy champion, and the man who brought Family Guy to life in Final Jeopardy.  You can buy his e-book about the whole experience for just $0.99 here.

– MBA from the Booth School of Business at the University of Chicago, now a management consultant at Booz & Co.

– Play-charter for Football Outsiders for the 2012 season

You can follow him on twitter at @jaredscohen.

Part 1

Always double down on eleven.

It’s a rule of thumb any blackjack player will tell you.  Applicable in almost every situation, at any casino, when your cards come up eleven and the dealer isn’t showing an Ace, you double down.  No matter how much you’re betting, no matter how bad your luck has been, no matter which random foreign country your expressionless automaton of a dealer is from, you double down on eleven.

Why?

Because it gives you the best chance at winning.

And how do we know that doubling down gives you the best chance at winning?

Because people have researched it.  People have done their homework, complete with statistics programs, random number generators and complicated looking equations (or as its known in some parts of the country, “witchcraft”)

 They even have a card which tells you what to do.  Since Blackjack is a game with well-defined rules and structure, there can be a clear strategy that, if executed, will help you win as much as possible (or, to be technically accurate, help you lose as little as possible).

While you don’t have to do what the card tells you, it’s definitely more right than your ‘gut’ instincts or homebrewed system.  It’s based on data.

Yet, some players will insist on hitting their 12 against a dealer 6.  They’ll refuse to double down on 11.  They’ll split 10s!

Simply put, they’re doing it wrong.

So how does this relate to football?  Well, I was wondering if NFL coaches are doing the exact same thing, eschewing data in favor of their own gut.

What I found was a little surprising.

Now, football is a complex game, with a lot of situations and decisions.  For this analysis, I decided to focus on what has long been a frustrating aspect of many Sundays spent with the Red Zone channel…Fourth Downs.

That shouldn’t come as a shock.  Fourth downs are the most direct view into a coach’s risk tolerance and personal philosophy.

And in thinking back to my blackjack discussion, I wondered what NFL coaches would look like if there were an optimal strategy for fourth downs.  There’s already a chart for when you’re supposed to go for a two point conversion.  If there were a basic strategy 4^th down card, how would coaches perform when compared them to it?  How close would they be to the best strategy?

That is to say, do these coaches know to double down on eleven? Or are they the guy sitting at the table refusing to split their pair of eights?

In these articles, I’ll describe the methodology behind the analysis, illustrate some results, and discuss some of the implications.

Context and Expected Points

To start things off, I wondered if anyone had already done research into optimal fourth down strategy.  I know lots of smart football fans out there have already spent time on it, so I figured it would already be established in the literature (and would save me the time of having to establish it).

Turns out it has been, and we’ll get into it in some detail, but before we get to that, we need to just establish the foundation for such an optimal strategy chart.  And that means some background in the idea of expected points.

Originally conceived back in the 1980’s, expected points has gained a fair amount of traction with the football analytics community.  Even ESPN uses it.  Here’s a little bit of their explanation, which I’m cribbing because it’s clear.

Based on statistical analysis of 10 years of NFL play-by-play data, ESPN has created a formula that assigns an “expected points” value to the team with the ball at the start of each play based on the game situation. Expected points (EP) accounts for factors such as down, distance to go, field position, home-field advantage and time remaining.

The value it puts out is on a scale from about minus-3 to 7, and it basically represents “which team is likely to score next, and how many points?” It represents the likely points not just on the current drive but also on the next drive or any subsequent drive until the score changes or the half ends. A lower value indicates a more favorable situation for the defense (i.e. fourth-and-20 from your own 1-yard line could be close to minus-3 EP), and a higher value represents a more favorable situation for the offense (i.e. first-and-goal is generally worth 6 EP).

Essentially, expected points is a way to evaluate football situations against each other.  Because each play is based on its situation (what down it is, field position, and how many yards to convert a first down), expected points serves as a normalized metric for the value of possession in any situation.  What’s more, is that after a play, you can compare the expected points values from the old and new situations, and determine how valuable the play was.  Here’s an example from ESPN’s explanation:

From your own 20-yard line, an 8-yard gain on third-and-10 is worth about minus-0.2 EPA because you don’t get a first down; the same 8 yards on third-and-7 is worth 1.4 EPA for converting a long third down and keeping the drive alive. EPA knows that not all yards are created equal.

So, using expected points as our basis, we can compare the best option (what would create the most expected points increase) for any given fourth down situation.  And when I say we, I mean Brian Burke.

Burke has already done a fair amount of research on the optimal fourth down strategy, which I’ve leveraged as the base for this analysis.

Using expected points values and historical data, Burke has done some prior research on what the ‘optimal’ fourth down decision should be (FG, punt, or going for it).  Depending on your field position, and yards to go, he put together a view of what decision would be best for a given fourth down.  His chart of that strategy is below:

Depending on where your given fourth down situation falls, the best option is illustrated here.  For example, if you had a fourth down and two yards to go on your opponent’s 10 yard line, the recommended option would be to go for it.  Alternatively, if you faced  a fourth down and had eight yards to go to convert it from your opponent’s 10 yard line, you should kick a field goal.

Make sense?

This chart can serve as the basis for our basic fourth down strategy chart.

Put another way, based on my rough transcription, it would look something like this.  Apologies for the lack of labels, but:

– The columns show yards to go (so 1 = 4th and 1)

– The rows show yard-line on the field, from your own end zone (so 5 = own 5 yard line, 95 = opponents 5 yard line).

– Together, cell 2 x 2 means shows the correct decision when faced with 4th and 2 from your own 2 yard line.

Looks just like a blackjack strategy card, doesn’t it? 

Note that there are a few strange suggestions in the card which are most likely just statistical anomalies that will disappear with more data.  The most obvious example of this is cell 2 x 98 (or what to do when faced with a 4th and 2 from the opponents 2 yard line).  That chart says kick the field goal, but it makes very little sense to kick from the 2 yard line while going for it from the 1 and 3-6 yard lines.

Also, in practice, teams could adjust the card each week to account for the relative strength of the opposing defense.  If you’re playing a very weak defense, you’re odds of converting 4th downs goes up, so you’d see a few more Green blocks above.

Now that we have an “optimal strategy”, or at least a “default”, we can compare it to real life decisions and see how often coaches make the right call.  Come back tomorrow for the results in Part 2.