Hopefully everyone has enjoyed the round-by-round breakdown.  While there are obviously a number of variables that can’t be controlled for, the pure statistical look at each position group has already provided some interesting insight.

Before I get to the big chart, let me just clarify exactly what I think this type of analysis is good for, then dig into an Eagles example.  Feel free to skip to the chart and come back.

The purpose of this is NOT to arrive at a set of rules by which teams should draft players.  Instead, it is meant to provide a general guideline, or a “default draft position”.  For example, yesterday I said that selecting DEs in the 3rd round is a terrible value proposition according to the data I’ve collected.  That doesn’t mean drafting a DE in the 3rd round is ALWAYS a terrible decision, it just means that for a team to make that decision, it must see or know something about the subject player that makes him CLEARLY much better than any other prospect available at that position.

The biggest flaw in NFL draft strategy, as far as I can tell, is each team’s confidence in its own ability to evaluate talent.  Regardless of the general manager, every team has historically had a very large margin of error when it comes to talent evaluation.  For instance, in the 1st round less than 6 in 10 LBs selected from 1999-2011 had or are having significantly productive careers (according to our definition.)

So what does that mean?  It means teams, in general, should be mostly focused on value during the draft, as opposed to parsing prospects.  This is perhaps never more apparent than when a team decides to trade up to select someone.  Let’s use the Eagles as an example.  NOTE: This is a very rough example, with numbers pulled from my ass, and is only meant to illustrate a larger point.

When the team traded up for Brandon Graham, it swapped 1st round picks with Denver and gave up two 3rd rounders as well (moving up from 24 to 13).

At the 13th pick, no DEs had been taken.  Between picks 13 and 24, 3 were taken, including Graham.  So that means the Eagles, in their analysis, decided that they had to take a DE (will not argue that decision here, though I was mad they passed on Early Thomas, and can produce witnesses that will verify I said that when it happened).

The only way the trade made sense was if the Eagles, in their DE analysis, decided that the odds of Graham becoming a stud DE were MUCH higher than the odds of JPP, Morgan, or Hughes becoming a stud DE.  Here is where the “margin of evaluation error” comes into play.

Using our historical draft data, we can calculate the odds of getting a starting DE with a 1st round pick and two 3rd round picks (I realize they were hoping more than an average starter, but stay with me).  Using the table below, we can calculate those odds to be 81.5%.  Using the Pro Bowl percentages from the earlier tables, we arrive at 37% for the odds of getting a Pro Bowl DE if you select DEs with a 1st round pick and 2 third round picks.  So here is the breakdown for “generic DEs”:

– 1st Round Pick – 24% chance of Pro Bowl, 67% chance of starting

– 1st Round Pick and 2 third round picks – 37% Pro Bowl, 81.5% starting.

Please note that this DOES NOT mean the Eagles made a bad decision.  Obviously the odds should be better for the 13th pick than for the 24th pick (we’ll get to that another time).  It DOES MEAN, however, that the Eagles, in their evaluation of Brandon Graham, should have been almost certain that he was more than 37% likely to be Pro-Bowl caliber, and more than 81.5% likely to start.

I have no doubt that they believed this, BUT, if they had applied a margin of error to their own analysis (as any good team should) prior to making the trade, they would have been unlikely to go through with it.  Let’s be extremely generous and assume the Eagles front office could peg these odds with a MOE of +/- 15% (WARNING: overly simplified statistics).   That means if they estimated Graham had a 90% chance of starting, his true odds of starting were almost definitely between 75-100%.

Immediately we can see an issue.  Even giving Graham incredibly high odds of starting (90%, which is more bullish than any team should be with any players outside the top few picks) and the Eagles a very generous MOE (+/-15%), the resulting range still does not exclude the 81.5% starting odds for the generic position of a first and two thirds (although it is near the bottom of that range).

That means the Eagles really should NOT have been (though I’m sure they were) confident that Graham would be better than just taking whichever DEs were available at #24 and in the third round, and hence, should not have made the trade.

I don’t mean to suggest that trading up is never a good idea, simply that the evaluative bar for whichever prospect is the target must be EXTREMELY high, and much higher than the standard currently being applied by most teams.

My final point:  Teams do not appear to take a probabilistic approach to drafting (which they almost certainly should), and I would guess that they do not actively overlay a margin of error onto their evaluations.  This is very much a “new-school vs. old-school” issue, similar to the statistical revolution in baseball, but IT IS NOT THE SAME.  It is largely a matter of GMs being willing to recognize and account for their own shortcomings and cognitive biases.  The franchises that can apply this will, in the long run, be more successful than those that don’t.  (Looking into which teams might be using this type of strategy will be another day’s post)

Ok, enough talk.  Here is the chart with every round included. I removed the All-Pro and Pro Bowl columns to make it easier to compare.  Dig into it and see what you think.  Later this week we’ll mine it for an “optimal default strategy”.