I posted last week about how we can compare this year’s prospect class to previous years and illustrated the system I will use going forward to do that.  However, it occurred to me that it would be of some value to modify the current system so that we do have a true apples-to-apples comparison with previous years.

I’ve done that and will use those results to illustrate the main idea of today’s post: accurately setting expectations for the draft.

Unfortunately, the draft structure lends itself very well to year-to-year comparisons (i.e. looking at the success of previous #4 picks).  However, in reality it doesn’t work like that.  Each draft class is distinct and wholly unrelated to the previous classes, meaning the quality of player available at #4 will differ greatly depending on the year.  But how greatly, and what does that mean for this year?

To answer that question, I first eliminated the NFP ratings from this year’s data (I don’t have it for previous years).  I then added the 2012 and 2010 prospects to the database and ran all the prospects through the same TPR formula.  Note I did not include 2011 because, for some ridiculous reason I can’t imagine, NFL.com’s draft tracker does not work for that year, meaning I don’t have the prospect ratings for the Cam Newton, A.J. Green, Patrick Peterson draft class.

Anyway, now we’ve got this year’s prospects compared side-by-side with the top prospects from 2012 and 2010, graded using the same scouting ratings and formula.  Here are the results, with current prospects highlighted yellow:

Notice the ratings and order of this year’s class is different from the current TPR system (due to the absence of NFP’s scouting data).  Also, while I will only be discussing the TPR rankings here, I’ve included a chart at the end of the post that sorts by average rating (so even if you think TPR is bullshit you can still find something interesting here).

As you can see, Luck, Bradford, and Griffin are far and away the best prospects from these three years.  Also fairly obvious is the fact that Luke Joeckel, this year’s #1 ranked player, is just the 9th best prospect here.

That’s what everyone means when they say there are no “elite” players in this year’s draft.

However, after the top subset of prospects, this year’s class takes over the list, meaning that there are a LOT of very good players here.  I know this point has been made repeatedly, but I wanted to show a very clear picture of exactly how this draft compares to previous years.

So what does that mean for the Eagles?

– First and foremost, it means trade back.  Almost regardless of compensation, if the Eagles can move back a few picks (staying in the top 10), the team should and likely will take that opportunity.

– The Eagles should still get a very good player.  Here’s where it will get tricky for fans.  If the Eagles stay at #4, whomever they select will be expected (by fans) to be a star from day one (or soon after).  It’s pretty clear, though, that such expectations would be unfair for just about all of these players.

Please, everyone must understand:

Luke Joeckel is not Joe Thomas

Star Lotulelei is not Gerald McCoy or Ndamukong Suh

Dee Milliner is not Patrick Peterson

It sucks (for the Eagles), but it’s true.  Anyone expecting that level of talent from the Eagles first pick is likely to be sorely disappointed.  However, take a close look at the chart above and it should be clear that the Eagles can still come out of this draft in good shape.  A number of prospects in this class are rated more highly than guys like Eric Berry, Joe Haden, Earl Thomas, Anthony Davis, etc…

Adding a player of that caliber is still a big upgrade and something we should all be excited about.

Other notes:

– By this measure, Brandon Graham was a better prospect than anyone the Eagles will be able to select this year…

– Eagles fans should be a bit higher on Dee Milliner.  Not perfect, but he’s a very good CB prospect.

– Picking Geno Smith at #4 would be crazy…and I think all the rumors are just misinformation.

– It’s possible that the Derrick Morgan, JPP, Brandon Graham DE run is among the best we’ve ever seen at that position.  Obviously depends on Graham’s transition, but all three of those players were steals on draft day.

– Not related to this post, but Tyler Wilson should be getting a lot more respect.

– Don’t sleep on Ziggy…with so much misinformation out there and rumors that the Eagles really like EVERYONE, I’m more intrigued by the LACK of reported interested in Ansah.

Finally, here is a chart of the same prospect set sorted by average scouting rating.  Similar story, though the names have shifted a bit:

While I posted the full TPR rankings earlier this week, I didn’t go through them in any detail.   Today I’ll look at the players most effected by the formula.

First the fallers.  These are players whose consensus scouting ranking is HIGHER than their ranking in the TPR system.  As a result, these are players we expect to be OVERDRAFTED.  Now we need to be careful about the relative magnitude here.  For example, a ranking difference (as shown in the TPR Effect column) of just a few spots isn’t a big deal.  We’re mostly concerned with the larger movements, as they indicate a wide divergence between perceived value and actual value (as we’re attempting to quantify it here).

Here are the top fallers among the top 60 overall prospects:

For Eagles fans, the first guy on the list (Cyprien) should be of interest.  Cyprien is a definite potential target for the team in the 2nd round.  By these rankings, taking him at #35 overall would be a slight reach, but nothing to get too upset over.  HOWEVER, it also shows that we shouldn’t be surprised if he goes towards the end of the first round.

We should all hope that somebody takes Matt Elam in the first round (where he’s been listed in several mock drafts).  Elam ranks as just the #5 overall safety by the TPR rankings, FAR below Cyprien, Eric Reid, and just behind D.J. Swearinger.

Other notes:

– Jarvis Jones and Sylvester Williams both drop from the mid-first round to the end of the first round or beginning of the second.  Both of these players have relatively large scouting rating standard deviations, meaning ESPN, NFL.com, and NFP have largely divergent opinions of each player.  Neither seem to be in play for the Eagles, but these guys are both  wild cards for draft day, in that they can go anywhere from the top 15 to the top half of the second round.

– Dion Jordan falls 4 spots.  While that’s not a huge amount on an absolute basis, it’s very meaningful for him, since he began ranked 4th overall.  Though he’s been a popular mock pick for the Eagles at #4, the rankings here show that would NOT be a good pick.

Now let’s look at the risers.  These are guys who are likely to be UNDERDRAFTED, and therefore may represent good value in the first two rounds of the draft.

Quarterbacks, obviously, feature prominently on this list. Despite a relatively low hit rate, the QB position still receives the largest benefit from this system.  The reward of hitting on one is so much larger than the reward of any other position that QBs tend to fly up the list (which we see in real life as well).

What’s interesting here is that Tyler Wilson gains the most and becomes the SECOND ranked QB, just ahead of Matt Barkley.  Most QB discussion has centered on Geno Smith, Ryan Nassib, Matt Barkley, and EJ Manuel.  According to these rankings though, Tyler Wilson should be getting a lot more attention.

– Towards the lower end, Ezekiel Ansah benefits, moving from #8 to the top 5.  Still not sure if he can be a 5-tech DE (I think he can), but he remains a dark hose candidate for the Eagles pick.  His ranking here suggests that wouldn’t be a stretch.

– Tank Carradine jumps 10 spots, from the end of the first round to the middle.  He is a player I’ve been secretly hoping would drop to the Eagles at #35, but his listing here suggests that’s unlikely.  If he does, it’d be an early candidate for steal of the draft.

So now we have relative rankings for all of the top prospects in this year’s draft, but there are a couple of issues with the scores, namely:

– Each player is scored on a different scale (due to positional risk and impact modifiers)

– We can’t compare players from one year to the next, since those modifiers will change as the league changes.

Neither of those presents any problems for what the ranking’s main purpose is, but there is a way to address it, which I will now explain.  This will allow us to answer the question of “how good is a prospect?”, rather than just “who is the best prospect?”.

First let’s look at the scales.  As I mentioned, each position carries a different maximum score, depending on the amount of inherent risk at that position and its potential impact.  Here are the maximum sores by position:

Note that these numbers are rounded for the sake of simplicity.  The only surprise for me is at the Safety position, which ranks near the bottom.  Basically, this is telling us that:

According to our Risk/Impact measures, the PERFECT QB prospect is better than the PERFECT DE prospect, and so on.  Again, the ranking of safeties surprises me, but that’s why I use the data, rather than my own rankings.

Now that we have the maximum scores for each position, we can take another step.  That is, we can find divide each player’s individual TPR grade (TPR seems more accurate than PVM at this point) by the total available points at his position.  For example:

Ziggy Ansah has a TPR grade of 93.6, out of a possible 103 points, giving him a % score of 91.3%.

This method of scoring has a couple of advantages.  It solves the problem of comparing prospects from different years while accounting for the fact that the league evolves.  It enables us to easily back out the positional impact and risk modifiers, while keeping individual risk. It also is a bit easier to understand, since it’s out of 100 and every player is on the same scale.

By looking at the % score, we get a better question of the ABSOLUTE value of each prospect, rather than just the RELATIVE value (though for the draft’s purposes RELATIVE is far more important).

So now let’s look at the top of the board again, this time with the % scores included. First I’ll show the top players sorted by TPR rank.  Then I’ll show them sorted by % score (“Pos % Score”).

This second chart will help to answer the question of “How strong is this draft?”.  Unfortunately, I do not have the complete data set for the past several years (can’t find NFP’s ratings).  With only 2 sets of ratings, the system loses a fair amount of value.  Additionally, I do not have the salary information from 2011-2012, which would result in different positional impact scores.  However, we’ll do the best we can with what we have, since I want to show an example of what we’ll be able to do from now on.  My guess is that, if included, NFP’s ratings would INCREASE the standard deviation of most players, LOWERING their overall score.

Here are the top 25 picks from last year:

Again, since we don’t have NFP’s ratings, this isn’t an exact apples-to-apples comparison.  Regardless, we can see at the top of the board that Andrew Luck was a far better prospect than anyone available this year (not that we didn’t know that).

HOWEVER, after Luck, Griffin, and Kalil, this year’s draft class looks pretty good by comparison.  In fact, the average Positional % score of the top 25 players THIS year is slightly above the average for last year, even though last year includes Andrew Luck as a high outlier.

Other notes of interest:

– Dee Milliner appears to be a BETTER prospect than Morris Claiborne was last year.  This has been talked about a lot (Milliner not being a “top” CB prospect), but he scores better than Claiborne, who was drafted 6th overall last year.  Consequently, it really should be a surprise if Milliner goes top 5 this year, despite not having elite return skills.

– All three top DTs this year (Star, Floyd, and Richardson) score better than Fletcher Cox did last year.

– Ryan Tannehill would indeed be the top QB prospect if he was in this year’s draft.

– Andrew Luck is as close to a perfect prospect as we are ever likely to see.

– RG3 was a better QB prospect than Chance Warmack is a G prospect.

Tomorrow we’ll take a look at the overall rankings and next week I’ll see if I can do some side-by-side big board comparisons so we can see where guys like Kiper diverge from our board.

If you haven’t seen the new Total Prospect Rankings, see either yesterday’s post or the PVM Rankings tab at the top.

There are a number of ways we can slice and dice this information, and over the next couple weeks I’ll look at a bunch of them.  Today, we’ll look at the “riskiest” prospects this year.  Note we are only looking at the top subsection of players (137 total, of which I posted 130), so we aren’t including players who are likely to be late round picks.

As I discussed yesterday, “risk” in the draft is composed of two components, the individual player’s risk and the larger positional risk, as quantified by the historic success of players drafted at each position.

First let’s look at the individual component.  Here is a chart showing the prospects with the highest standard deviations in their consensus ratings.  Note that the measure here is HALF the true standard deviation.  Also, for this chart I’ve only used Top 60 prospects.  The worse prospects tend to have higher standard deviations, so using the whole sample results in a bunch of guys that aren’t really worth talking about just yet.

The columns should be self-explanatory, but: “Consensus” is the average scouting score.  “.5 SD” is the measure we are looking at.  “Pos Miss Rate” is the historical failure rate for each position.  “Risk Adj” is the combined risk component, as described yesterday.

Immediately, the two QBs at the top jump out.  EJ Manuel and Ryan Nassib, both potential targets for the Eagles in round two (or late round 1) carry very high “individual risk rates”.  The three scouting groups I’ve used have widely different opinions of these players.  Therefore, they are riskier choices.

Other notable inclusions on this list are:

– Sylvester Williams, DT.  Ranked 22nd overall by the Consensus Scores, he drops to 33 in the TPR system (due in part to his placement above).  He carries the 6th highest individual uncertainty score of all the top prospects.

– John Cyprien, S.  He is a popular target for the Eagles 2nd round pick, but does not fair well in the TPR system.  There is relatively wide disagreement among scouts about his ability, and while he ranks 30th in the Consensus Scores, he drops to 43rd in the TPR rankings.  Note that the Eagles 2nd round pick is nearly halfway between those two rankings, meaning he remains a viable option for the Eagles at pick #35.

– Towards the bottom, we can see Sharrif Floyd carries the highest individual uncertainty among Top 10 prospects.  This is part of the reason I am firmly in the Star>Sharrif camp.

Now here is the chart showing the players with the LOWEST individual risk grades:

This is perhaps the more interesting chart for Eagles fans, as it includes many of the potential targets for the 4th overall pick.

– Chance Warmack remains a pain in the ass.  I originally conceived of the positional adjustments in an effort to explain why Chance is a terrible use of resources in the Top 10. However, through two iterations he remains near the top of the TPR draft board.

Here I have to stress two things about Warmack that people aren’t mentioning enough:

1) He is the highest rated prospect by Consensus Score.

2) He has the lowest standard deviation, suggesting there is widespread agreement that he is the best prospect.

I’m close to the point of saying that our (and MY) preconceived notions about positional value may be blinding us to what should be an obvious assertion, namely: Chance Warmack should be a top pick.

I’m not there yet, but the entire idea here was to take subjectivity and bias out of the equation, and on that basis, Chance Warmack looks pretty good.

– Dee Milliner, Dion Jordan, and Star Lotulelei all place near the top of the board.  That means, on an individual basis, we should be more confident in their scouting grades then in the grades of players lower on the list.

How about Total Risk?

Now let’s look at the total risk component.  As described yesterday, the total risk component is weighted heavily to Positional Evaluative Error, with individual uncertainty included, but with less impact.

This is one area where I am not 100% comfortable with the model.  Should individual uncertainty carry more weight?  I don’t know, but maybe, and it’s something I’m thinking about as we get closer to the draft.

Anyway, here is a chart showing the Top 60 prospects with the HIGHEST risk components:

Nobody should be surprised to see EJ Manuel and Ryan Nassib at the top of the list.  A lot of individual uncertainty + a large positional risk rate = a big risk component.  These guys may become good/great players, but given where they will likely be drafted, they represent the riskiest picks in the entire draft.

Others of note:

– We already discussed Williams, Cyprien, Floyd.  For obvious reasons, they are included above.

– Keenan Allen is another one to note. Ranked 16th overall by Consensus Score, he is one of the “riskiest” top prospects, despite the fact that the TPR system only drops him to 19th overall.

While the guys above probably don’t seem familiar to most of you, they are all players that we’ll be seeing/talking about towards the bottom of the 1st round and through the 2nd round of the draft.  Might as well start familiarizing yourself with them now.

Now for the LEAST RISKY top prospects:

Not surprisingly, we see mostly offensive linemen at the top of the list.  The biggest name for Eagles fans is Eric Fisher, who is probably the most likely pick for the Eagles at #4.  Based on the current measure of risk that I’m using, Fisher is the 3rd SAFEST top prospect.

Some other things to keep in mind:

– Pay close attention to the CBs listed near the top of the list.  Guys like Rhodes and Taylor could be great pick-ups for the Eagles if they fall to the second round.

– Menelik Watson is another guy to keep an eye on.  It’s possible (though I think unlikely) that he’s available at #35 and, should the Eagles pass on Fisher at #4, would be a prime target for the team.

As I said, I’ll be looking at this info in a number of different ways, but please feel free to suggest anything in particular that you’d really like to see, either through twitter, email, or the comments.

As promised, I have completed a relatively dramatic overhaul of the PVM system.  Just as I did in the original post, I’ll explain the methodology and theory first (and changes) and then list the updated rankings.  Feel free to skip to the bottom if you don’t care how the system works.

Please note that this is still far from a finished product.  There are a number of issues (mostly calibration) that I will only be able to address after collecting several years of data.  Additionally, I’m posting it now so that I can incorporate and comments/criticism prior to the draft.

The Theory

The original PVM system was meant to function as a better determination of BPA (Best Player Available) rankings.  In essence, a player’s scouting grade is not enough.  A complete system will account for positional value and positional risk IN ADDITION to the scouting grade.

Basically, this system is an attempt to provide a “Default BPA Ranking”, against which all team drafts can be compared/graded.  The original system included two components, scouting grades and a positional value multiplier (hence the PVM name).  Today, we’ll add a measure of risk to the formula.

Methodology 

As mentioned above, the system is composed of three components:

Consensus Scouting Grade – Player/Positional Risk Adjustment + Positional Impact Bonus

Consensus Scouting Grade

This is the easiest part.  I compiled prospect ratings from 3 4 reputable sources: Scouts Inc. (ESPN), NFL.com, the National Football Post, and Draft Ace.  The NFP’s scale is different, so those number were adjusted for an apples-to-apples comparison.  Additionally, NFP includes letter grades/modifiers that were not incorporated here.  Each source is weighted equally, with the average score taken as the “Consensus Rating”.

Player/Positional Risk Adjustment

This is the new addition to the system.  Obviously, some players/positions carry more risk than others.  I’ve attempted to quantify that using two measures: standard deviation of the scouting ratings and historical positional miss rates.

Positional Miss Rates:  To quantify the positional risk, I looked at every player drafted in the 1st or 2nd Round between 1999 and 2011.  Basically, players selected in the 1st and 2nd round are expected to be starters, regardless of position.  The miss rate is an attempt to see which positions have larger margins of evaluative error.  Here are the rates I arrived at:

A player counts as a “miss” if they did not start for at least 5 seasons (according to Pro-Football-Reference.com) or, if they have not been in the league that long, have not started at least half the time. Note that my historical data does not break out LBs by outside/inside, hence the same miss rate.

As you can see, there is a very wide disparity in success rates.  As expected, offensive linemen are “safer” picks, while offensive skill positions are the riskiest.

Here’s where calibration comes into play.  The scouting grades are out of 100 points.  To arrive at a positional risk measure, I multiplied the historic miss rates by 100, then took the square root of that.  For example, the risk adjustment for the TE position would be 100 * .28 = 28.  SQRT(28) = 5.27.

That’s the first half of the risk measure.

I then took each player’s rating standard deviation and multiplied it by .5.  Basically, the idea here is that wide agreement among scouts should equate to more certainty and vise-versa.  Therefore,  a high standard deviation means we are less certain about the consensus rating.  A low standard deviation means we are more certain.  I took half of the SD measure to avoid overly penalizing players with high consensus grades but high standard deviations.  On the pure SD measure, players with lower grades across the board can jump players with higher grades across the board, by shear virtue of small deviation.

I then added this to the positional measure explained above to arrive at the Risk Adjustment.  The Risk Adjustment is subtracted from the consensus prospect rating.

For Example: Sharrif Floyd’s consensus rating is 92.98.  His standard deviation multiplied by .5 = 1.66.  His positional risk measure (DT) equals 6.76.

6.76 + 1.66 = 8.42

So Floyd’s Risk Adjustment is 8.42, his score after accounting for risk is 84.56.

Now we need to add our last component, positional value.

Positional Value Multiplier

The overall idea here remains the same as the original system.  However, I’ve changed both the sources of data and the subset included.

Previously, we used the average cap hit for each position across the entire league.  Since the NFL has a salary cap, the relative allocations should tell us how teams, in general, value different positions.

However, those measures included rookie contracts and contracts for players not expected to contribute at their position (like a special teams ace).

So here’s what I did instead:

First I went to Pro Football Focus and counted how many players played more than 50% of their team’s snaps at each position.  I did this to account for positional scarcity (e.g. there are a lot more WRs than Cs).  I then took 25% of those measures so that we will only be looking at players considered well above average at their position (at least by pay).

Then I looked up the top _ contracts at each position and found the average value at each position.

I used those averages to create new positional multiplier values.  Here are the new measures:

Compared to the original measures, there is a wider disparity.  For instance, the QB measure was previously 9.07%, significantly less than the new measure.

Just as we did before, I used these measures to inflate the Risk Adjusted Prospect score for each player.

To recap: We start with the Consensus Ratings, Subtract a Risk Measure to account for player rating uncertainty and positional scouting error, then inflate according to the PVM numbers to account for positional impact.

Overall, the biggest flaw right now is the calibration of the Risk Measure and PVMs, their subjective.  However, I attempted to balance the relative magnitude of both of them.  Additionally, if we step back and look at the overall measure, we see that relative importance is as follows:

– Consensus ratings (by far the most important)

– Positional Value and Risk

– Individual player rating uncertainty

That looks right to me, as far as what I’m going for.  As I mentioned above, the only way to arrive at a “correct” calibration is to use the system for several years, then analyze the rankings to see which parts of the formula are over-weighted and which are under-weighted.

By all means, if you have thoughts as to how to improve the system, let me know.  We still have a couple of weeks to make revisions/improvements.

Total Prospect Rankings

Here they are, we’ll talk about them in more detail starting tomorrow.