Unfortunately, I haven’t yet found the necessary data to complete the type of 3rd down examination I’m aiming for. Yesterday, I mentioned that many commentators are overlooking the fact that you have to be a very good offense in order to run a lot of plays (you need to gain a lot of first downs). That might sound obvious, but it’s important to consider when thinking about Chip Kelly’s stated preference for “pace”. Just wanting to push the pace won’t mean much if the team can’t gain first downs.
Obviously, that train of thought needs more research. For today, though, I figured I’d highlight something I stumbled across while searching for data. See this site:
I have no idea where it came from, but I’ve downloaded the data its supposedly based on and it seems legit. It’s useful for illustrating an area of the game that I believe is potentially the largest untapped resource of analytical advantage: Run/Pass play selection.
At a very basic level, we can look at play selection through what I will call the Tecmo Bowl Model (or Tecmo Super Bowl if you prefer the upgrade).
What’s the Tecmo Bowl Model?
I’m making this up as I type, but in my head, it’s an incredibly simple construct for visualizing the importance of Game Theory in play selection. For those who don’t know the game (I’d be very surprised if any readers don’t, but just in case), Tecmo Bowl was a groundbreaking football game for the original Nintendo system. I won’t go into any more detail than they that, but if you haven’t played it, you should go find an emulator and change that. The important part is the play selection screen, shown here:
Basically, for both offense and defense, a selection of 4 plays are shown. The player picks one of these plays by pressing the corresponding button combination (shown below each play). Here is the key part:
If the Defensive player picks the same play as the Offensive player, then as soon as the ball is hiked, pretty much every offensive lineman is pancaked, resulting in a likely sack or a throw from the QB that’s no more than a jump-ball.
Without getting into any higher-order strategy, the defense has a 25% chance of “picking” the offenses play.
Despite its relative simplicity, I believe this game has a LOT to tell us about modern (and real) football. In essence, the defense is continuously trying to “pick” the offense’s play. The results aren’t as dramatic as in the game above, but the fundamentals are the same.
Is the offense going to run or pass? Short or deep? Maybe play-action?
These are all questions defensive play-callers have to ask themselves. Each subsequent decision is largely determined by both the defense’s overall strategy/strengths AND the subject down/distance/score/time scenario.
Just as in the video game, if the defense successfully “picks” the offense’s play, the result SHOULD be a “win” for the defense.
So what?
This has obvious reprecussions for offensive strategy. It is not enough to just pick a good play; you also need to account for what the defense will be expecting. Correlating to the statement above: if the offense picks a play that the defense is NOT expecting, it has a distinct advantage and will be more likely to “win” that down.
Let’s take a look at an example. From the site I referenced at the top:
The Scenario: 3rd Down, 1-5 yards to go, All Scores, Any Time.
The Results:
For this scenario, “success” is defined as a 1st down or a touchdown. What can we infer from the data above?
The output makes it very easy to see the relevant discrepancies. In this situation, Run plays were 14% MORE successful than Pass plays.
Despite this, Pass plays were called nearly TWICE as often.
There are two leading explanations for this difference:
– It’s a true inefficiency.
– It’s simply the result of the type of Game Theory I described in the Tecmo Bowl example above. Basically, Run plays are more successful BECAUSE they are called less often. Pass plays are called 67% of the time, meaning the defense is most likely EXPECTING PASS, leaving them more susceptible to a RUN.
The big question is, has this game (long-term play selection) reach its Nash Equilibrium?
The Nash Equilibrium (very basic explanation) is the point at which both sides know the other’s strategy, but neither side is incentivized to change its own in response to that knowledge.
This situation is slightly different, since we’re talking about individual occurences within a long term game, but I believe the overall point holds. More importantly, I do NOT think the NFL has reached the point of equilibrium (within this particular scenario at least).
Here’s where I need some help. It seems to me that the point of equilibrium would naturally occur around the Run/Pass split that equates to a roughly equal chance of success for each. Note that because of the Tecmo Bowl Model explanation above, this does NOT necessarily equal a 50/50 split in play calling. Regardless, I might be mistaken in my reasoning here. If you have any other predictions/analysis, please let me know.
Also, lest we fall victim to a small sample, here is the output when we include every season from 2002-2012:
The numbers are almost identical, which means I’m either missing something in my prediction/analysis (very possible), or NFL play-callers are operating at a very inefficient clip.
Obviously, if it’s the latter, it has HUGE implications for in-game strategy.
Eagles Rewind 
I think it becomes deeper when you compare the potential yardage for run vs pass play.
On 3rd & 3 you can:
1) dial up a run which you can expect at least 3 yards
2) dial up a quick pass to try and pick up the 3 yards
3) try to spread the field and go for a longer pass play
The difference I see is that there’s really no different play call on offense for a 3 yard run vs a 30 yard run.
In reality it could be closer to a true 33% chance for each play type?
I see you’re point, but I think the consequences of it are more likely to hit at a higher level.
My immediate response would be: the offense, on 3rd down, should be HEAVILY incentivized to hit the point at which they have the greatest odds of converting. The goal should be (in most cases) to gain the minimal amount of acceptable yardage, which presumably carries the lowest risk of not converting.
Overall, offenses target points, not first downs, so it makes sense that the optimal point will lean towards passing (higher ceiling), but it looks like the split should be a lot more even.
It may be that offenses just aren’t valuing 1st downs as highly as I expect them to. If it were me, the ONLY goal on third down would be to get a 1st down (aside from special situations). I expect, though, that some coaches see the 3rd and short as an opportunity to “take a shot” downfield. However, i’d be surprised if that happened often enough to account for the difference we see here.
“It seems to me that the point of equilibrium would naturally occur around the Run/Pass split that equates to a roughly equal chance of success for each.”
This is not necessarily true. What we care about is overall success, which is weighted by the frequency of run and pass plays. From the data, it might look like it would be better for an offense to call a higher frequency of run plays in this situation, but we don’t know if it actually is better. More called run plays would cause the optimal defense to adjust, and the run success rate would probably go down while the pass success rate goes up. Whether the overall success rate increases, though, depends on the relative strength of those effects.
So the league ratios could indeed be at or close to the Nash equilibrium. That’s not to say that Chip Kelly couldn’t surprise opponents and run more on third and short to gain a temporary advantage until defenses learned to adjust.
Thats exactly what Im getting at. I don’t expect the Nash point to occur with an equal Pass/Run spit precisely for the reason you stated. As teams run more, defenses will expect it more, leading to diminshing returns for each marginal increase in % runs.
While it’s definitely possible we’ve hit that point (hence the lack of movement in the long term averages), I find if unlikely that the equilibrium point is so tilted.
There are a lot of complicating factors here, but it certainly seems to me like teams should be running more in 3rd down and 5 or less yards to go. Also, we haven’t even begun to incorporate the 4th down options, which (if you look at the 4th down play chart) would likely tilt the advantage even more towards running the ball (i.e. gaining 4 yards on 4th and 5 is actually a success in many cases, since gaining just 1 yard with an additional run is likely).
Sounds like we’re on the same page then. Your last point is especially important, I think. The league’s play selection and results on 2nd down with 1-5 yards to go bears this out: http://football.10flow.com/#2002|2012|0|0|0|0|2|4| . In that case, runs are called 60% of the time and the success rate is roughly equal for runs and passes.
To the last point:
I should have made this clear, but any inefficiency here is definitely short term, since the league will adjust.
I think its interesting when you split the data to 1 yard to go, 2 yards to go only, and then 3-5 yards to go. Using the 2002-2012 sample size, the run/pass split is almost exactly a 75/25 percent with 1 yard to go. However just with 2 yards pass becomes greatly preferred with a 38/61 percent split and for 3-5 yards its an overwhelming 17/82 split in favor of pass. Success rate for runs also falls significantly the further you get away from a first down 71/59/48 while pass plants remain remarkably constant 55/51/46.
It seems that between 3-5 yards the success rates are almost the same meeting your definition of “equilibrium”. I think you can pretty safely say that teams are passing too much with 2 yards to go, but it seems switching strategies provides only slight gains and isn’t indicative of huge inefficiency. The bulk of the discrepancy comes from 25% of the time when a pass play is called with 1 yard to go but there is a 16% difference in success rate. So I think the conversation should be centered around what is unique about 3rd down decision making with 1 yard to go that causes coaches to call a pass play.
The next step is definitely to examine the data at smaller intervals. Unfortunately, that site isn’t set up for looking at every potential yardage, but you’ve started to do the work already, and you’re point about 3rd and 1 looks like a very important one.
Hi Brent. You are one of my favorite writers every time i read anything you write.
I think you may be missing one thing here, which goes back to that whole statistical thing about correlation not implying causation. There clearly is a link, but the explanation could be anything. the tricky part is that the explanation could be a third variable.
For example.. perhaps the success of run plays was not a result of being less expected and also not a result of run plays working better. Perhaps they were more successful because they were used in more specific situations where run plays work.
For an extreme example, lets say all of those run plays came in situations where the quarterback was planning a pass play and called an audible because he found a weakness in the defense. Clearly then switching to more run plays would not be a solution.
For another example, i could show a link between 3 point shooters and nba field goal efficiency, but that doesnt mean every player should shoot 3s, because that only works for certain guys.
In any case, I’m not disagreeing with your conclusion, but showing a correlation does not imply that the correlation will continue in other circumstances nor say anything about the cause.
i agree with your outlook on how the game works and i like the tecmo bowl example. I also think you have a great outlook on the data. However I’m just suggesting that lets not take this as fact that any team should just run more often, without further analysis, if that makes sense.
Makes a lot of sense. This definitely needs more work before we can say “all teams should run more”, which I dont believe is the case. I’ve got a follow-up post for today that will go further and clear some of it up.
Pingback: Nash Equilibrium and 3rd Down Strategy | Eagles Rewind