As promised, I have completed a relatively dramatic overhaul of the PVM system.  Just as I did in the original post, I’ll explain the methodology and theory first (and changes) and then list the updated rankings.  Feel free to skip to the bottom if you don’t care how the system works.

Please note that this is still far from a finished product.  There are a number of issues (mostly calibration) that I will only be able to address after collecting several years of data.  Additionally, I’m posting it now so that I can incorporate and comments/criticism prior to the draft.

The Theory

The original PVM system was meant to function as a better determination of BPA (Best Player Available) rankings.  In essence, a player’s scouting grade is not enough.  A complete system will account for positional value and positional risk IN ADDITION to the scouting grade.

Basically, this system is an attempt to provide a “Default BPA Ranking”, against which all team drafts can be compared/graded.  The original system included two components, scouting grades and a positional value multiplier (hence the PVM name).  Today, we’ll add a measure of risk to the formula.

Methodology 

As mentioned above, the system is composed of three components:

Consensus Scouting Grade – Player/Positional Risk Adjustment + Positional Impact Bonus

Consensus Scouting Grade

This is the easiest part.  I compiled prospect ratings from 3 4 reputable sources: Scouts Inc. (ESPN), NFL.com, the National Football Post, and Draft Ace.  The NFP’s scale is different, so those number were adjusted for an apples-to-apples comparison.  Additionally, NFP includes letter grades/modifiers that were not incorporated here.  Each source is weighted equally, with the average score taken as the “Consensus Rating”.

Player/Positional Risk Adjustment

This is the new addition to the system.  Obviously, some players/positions carry more risk than others.  I’ve attempted to quantify that using two measures: standard deviation of the scouting ratings and historical positional miss rates.

Positional Miss Rates:  To quantify the positional risk, I looked at every player drafted in the 1st or 2nd Round between 1999 and 2011.  Basically, players selected in the 1st and 2nd round are expected to be starters, regardless of position.  The miss rate is an attempt to see which positions have larger margins of evaluative error.  Here are the rates I arrived at:

A player counts as a “miss” if they did not start for at least 5 seasons (according to Pro-Football-Reference.com) or, if they have not been in the league that long, have not started at least half the time. Note that my historical data does not break out LBs by outside/inside, hence the same miss rate.

As you can see, there is a very wide disparity in success rates.  As expected, offensive linemen are “safer” picks, while offensive skill positions are the riskiest.

Here’s where calibration comes into play.  The scouting grades are out of 100 points.  To arrive at a positional risk measure, I multiplied the historic miss rates by 100, then took the square root of that.  For example, the risk adjustment for the TE position would be 100 * .28 = 28.  SQRT(28) = 5.27.

That’s the first half of the risk measure.

I then took each player’s rating standard deviation and multiplied it by .5.  Basically, the idea here is that wide agreement among scouts should equate to more certainty and vise-versa.  Therefore,  a high standard deviation means we are less certain about the consensus rating.  A low standard deviation means we are more certain.  I took half of the SD measure to avoid overly penalizing players with high consensus grades but high standard deviations.  On the pure SD measure, players with lower grades across the board can jump players with higher grades across the board, by shear virtue of small deviation.

I then added this to the positional measure explained above to arrive at the Risk Adjustment.  The Risk Adjustment is subtracted from the consensus prospect rating.

For Example: Sharrif Floyd’s consensus rating is 92.98.  His standard deviation multiplied by .5 = 1.66.  His positional risk measure (DT) equals 6.76.

6.76 + 1.66 = 8.42

So Floyd’s Risk Adjustment is 8.42, his score after accounting for risk is 84.56.

Now we need to add our last component, positional value.

Positional Value Multiplier

The overall idea here remains the same as the original system.  However, I’ve changed both the sources of data and the subset included.

Previously, we used the average cap hit for each position across the entire league.  Since the NFL has a salary cap, the relative allocations should tell us how teams, in general, value different positions.

However, those measures included rookie contracts and contracts for players not expected to contribute at their position (like a special teams ace).

So here’s what I did instead:

First I went to Pro Football Focus and counted how many players played more than 50% of their team’s snaps at each position.  I did this to account for positional scarcity (e.g. there are a lot more WRs than Cs).  I then took 25% of those measures so that we will only be looking at players considered well above average at their position (at least by pay).

Then I looked up the top _ contracts at each position and found the average value at each position.

I used those averages to create new positional multiplier values.  Here are the new measures:

Compared to the original measures, there is a wider disparity.  For instance, the QB measure was previously 9.07%, significantly less than the new measure.

Just as we did before, I used these measures to inflate the Risk Adjusted Prospect score for each player.

To recap: We start with the Consensus Ratings, Subtract a Risk Measure to account for player rating uncertainty and positional scouting error, then inflate according to the PVM numbers to account for positional impact.

Overall, the biggest flaw right now is the calibration of the Risk Measure and PVMs, their subjective.  However, I attempted to balance the relative magnitude of both of them.  Additionally, if we step back and look at the overall measure, we see that relative importance is as follows:

– Consensus ratings (by far the most important)

– Positional Value and Risk

– Individual player rating uncertainty

That looks right to me, as far as what I’m going for.  As I mentioned above, the only way to arrive at a “correct” calibration is to use the system for several years, then analyze the rankings to see which parts of the formula are over-weighted and which are under-weighted.

By all means, if you have thoughts as to how to improve the system, let me know.  We still have a couple of weeks to make revisions/improvements.

Total Prospect Rankings

Here they are, we’ll talk about them in more detail starting tomorrow.

A couple of people asked for a more detailed breakdown of the 1st round, so we’ll do that today.  First, however, I need to apologize for a fairly large mistake.  As I was putting together today’s data, I noticed a mistake in the Strategy Chart I posted yesterday (and have posted before).  The upshot?

– 1st round LBs are much more successful than I had listed them.

Obviously this affects the Dion Jordan comments yesterday and will affect my draft analysis going forwards.  Sorry for the misinformation, just a typo in excel that I missed.  Thankfully I caught it before we start incorporating the data into the PVM rankings.

Now back to the subject:

If you haven’t seen the post showing the top 15 breakdown, please read that here:

Top 15 Breakdown

That post illustrates pretty clearly that there isn’t nearly as big a difference between, say the #5 pick and the #12 pick, as some people believe.  However, there is a big drop-off after #15.  In general, the real “first round” is picks 1-15.  Hence my point about the advantages of trading down from #4 but remaining in the top 15.  There is some positional loss from trading down (no QBs in general), but overall it appears to be a good strategy.

For those wondering, I also did a positional breakdown post for the Top 15 picks here.  That’s also a must-read for anyone interested in the draft, as it gets to the “prospect tiers” aspect of strategy.  Please note that both of those use a slightly different time-frame than we will use today.

Now let’s look at success rates for the first round.  Here is the complete chart, with the first round broken into three segments.  Following is each segment by itself to make it easier to read.  The data does not include this past season, so pro bowlers/all-pros from this past year are not included.

Before we start analyzing, I want to warn everyone about the danger of small sample sizes.  The reason I haven’t typically broken the draft into such small sections (1-5) is that when you do that and combine it with a positional breakdown, the samples get way too small to draw any conclusions from.  For example, there have been just 2 safeties drafted in the top 5, Sean Taylor and Eric Berry.  Both were/are very good players and each made the Pro Bowl.  However, it’d be wrong to suggest that the odds of a top 5 safety making the pro bowl are 100%.  In light of that, try view this data in a proper context.

Now the takeaways-

– Notice that the odds of getting an All-Pro caliber player fall significantly after the top 15.

– QBs, as we’ve discussed, are very risky throughout the first round, but especially so in the second half of it.  Just 45% of QBs drafted between 16-32 became “starters”.   Just one of the 11 became a Pro Bowler (Aaron Rodgers).

– The charts certainly support the point I made earlier: The “first round” of the draft should really be viewed as just the top 15 picks.  Odds of success fall dramatically after that, so it’s unreasonable to hold all 1st round selections to the same standard.

– Obviously positional distribution is a major factor here.  Since the best QBs are typically taken high in the first round, we can assume that those selected towards the end of the round are lesser prospects.  Compare that to positions like G or TE, which are not typically taken high.  As a result, it makes sense that those positions would carry greater odds of success towards the end of the first round.

– In all, this highlights the importance of BPA discipline.  Just because you need a QB doesn’t mean you should take the best one available.  As we can see above, taking the best prospect, regardless of position, is a much better strategy.  Note I still believe in adjusting for positional value, but BPA with that adjustment remains, by far, the best draft strategy.

– RBs are really terrible value picks and should be taken in the 1st round much less often.

– Regarding the Eagles, the data shows that, regardless of which position the team selects, it has good odds of getting an elite player.  With several “needs”, the Eagles main focus should be on minimizing risk.  Blowing the #4 pick because you picked “potential” over “performance” or were focused on one position would be a HUGE missed opportunity.  I’m not directing that comment at any prospect in particular, just trying to emphasize how important it is for the Eagles to maintain BPA discipline.

Sorry again for the bad LB data. I’ll be updating the full chart and reposting next week.

I’m on the road the rest of this week, so NO POSTS TILL MONDAY.  I may be able to respond to comments, but full posts will not be possible.

Thanks for reading.  I appreciate all the feedback and hope you continue to find this blog interesting.

Today let’s take another look at our draft strategy chart, this time using it to talk about a few prospects.  Some of you have seen this, newer readers have not.  I apologize if its tough to see without zooming in, but I like to get a comprehensive look with this chart.  Note that the 6th and 7th round columns were included in yesterday’s post.  The sample is all players drafted from 1999-2011.

The different color backgrounds reflect my quick attempts to eliminate sub-optimal round/position choices (in black).

Between now and the draft, I am hoping to mine this in a little more detail and ultimately combine it with the PVM system.  However, for today, let’s talk about specific players.

First, a note on value.  Obviously applying this chart to individual players is a misuse of the data.  There is no guarantee that the sample is representative, and there are potentially a lot of other factors at work.  In general, I believe the value of this type of data is to give us a sense of which positions are easier/harder to evaluate.  For example, LBs seem to have a much larger margin of evaluative error than OTs.  Therefore, a LB with similar ratings to an OT will carry more inherent risk by virtue of the fact that, historically, LB projections are less accurate.

Hopefully that made sense.  We’ll be revisiting this a lot in the next few weeks.  Now let’s get to the players.

Dion Jordan – He immediately pops out because A) he’s been frequently “mocked” to the Eagles, and B) plays a position that carries the worst 1st round hit rate.  This doesn’t mean Jordan won’t be a good player, only that LBs are very hard to project coming out of college.  This is a major reason why I do not like him for the #4 pick.  Added risk, no additional reward.  With a lot of similarly rated prospects, no reason to take on any additional risk.  Also, notice that the odds of finding a starting LB in the second round are almost as good as in the first round.

Star Lotulelei – I’ve been a big proponent of Star for as long as this blog has been up (I may have actually been the first one to peg him to the Eagles).  I still believe he is among the best potential picks for the Eagles.  He’s a run stopper and a natural pivot on the d-line, able to hold his ground against a double-team, allowing the LBs  to fly to the ball.  Heart condition notwithstanding, Star would immediately become the Eagles 5-tech and could play NT as well if Spooky doesn’t perform.  However, the chart above is a bit of a red flag.  DTs carry some of the lowest “elite player” odds (all-pro, pro bowl) as well as relatively low starter odds.  Additionally, the odds of finding a starting DT in the second round are good.

Once I figure out how to factor this in to the PVM model, I expect to see Star move down the board a couple of spots.  For now though, he remains a personal favorite.  In a perfect world, the Eagles would trade down to #7 or #8 and pick him up there.

Joeckel/Fisher – The Eric Fisher train has picked up a bit of speed recently, especially with the Eagles sitting out the OT FA market (for the moment).  I’m not a huge Fisher fan, but the odds clearly submit him as a “low-risk” pick.  It is common knowledge at this point that OTs are safer picks in the 1st round than any other position, and the data definitely bears this out (im not counting Cs due to sample size).  HOWEVER, that is an overly simplistic way of looking at it.  You also have to account for the fact that OT is one of the EASIEST positions in which to find contributors later in the draft.  You won’t get an elite OT after the first round, but notice you still have 25% chance to get one as late as the 4th round (and 13% in the 7th).

Let’s play a quick probability game.  Assume that this data was perfectly representative of the upcoming draft.  The Eagles have four 7th round picks.  If the team were to use all 4 of them on OTs, what would the odds of finding a starter be?

Well the chances of missing, according to our table, are 87%.  The chances of missing with all 4 picks are .87^4, or 57%.  Therefore, the odds of NOT MISSING all of them would be 43%.

A 43% chance isn’t great, but it’s pretty significant, and remember this is in the 7th round.

In the second round, 70% of OTs in our sample ended up as “starters”.

This is a long way of saying that the Eagles resources, according to the current prospect rankings and team needs, are probably better spent elsewhere than on an OT in the 1st round.

Dee Milliner – Another guy who has picked up steam (the 40 time helped a lot).  He could go as high as #1, though I think that’s unlikely.  I like him because he is ranked #1 by PVM, though we’ll have to wait and see if he holds onto that position as ratings are updated.  The major pushback on him is that he doesn’t rate as a “shutdown” corner.  Obviously, if you are using the #4 pick on a CB, you want him to be an “island” sort of player.  However, this is a pretty ridiculous read of the situation.  Essentially, draftniks are comparing Milliner to CBs in OTHER drafts.  Obviously there have been better CB prospects.  The fact remains, though, that in THIS draft, Milliner is by far the best CB prospect, and is also one of the best prospects PERIOD.

You can chalk this up to bad timing for the Eagles, in that they got a high pick in a relatively weak draft (projected), but at the end of the day you have to play the cards you’re dealt.  The table above says DBs are pretty safe in the first round.  For those asking, CBs and Ss are not separated because the often switch positions coming out of college or in their first few years (or last).  I’m doing some work to fix this, but keep in mind that this is also a potential advantage.  If a CB fails in the NFL, he may be able to switch to Safety and become a productive player.  If a WR or QB fails (or most other positions), there is no outlet  to salvage that prospect.  Consequently, DBs are lower risk.

In case you didn’t realize it, this is a big thumbs UP for Milliner.

Geno Smith – This chart definitely throws the brakes on the Geno hype.  QBs have a very low success rate, even in the first round (relatively).  That would make me wary of drafting a QB with more than a couple small flaws in his game.  I still think Geno goes high (top 5), but I wouldn’t want to be the team that takes him.

The chart illustrates just how hard it is to draft quarterbacks.  Remember that it isn’t a “reach” just because a QB went higher than his rating suggests.  Hitting on a QB offers a MUCH greater reward than hitting on any other position (hence the PVM adjustment).  HOWEVER, with that reward obviously comes greater risk.  For a team like the Eagles, which has a bunch of holes and some existing talent at QB, a high-risk/high-reward play like Geno Smith doesn’t make much sense.

Was asked to post the 2010 2nd round table, so here it is.  Still waiting for FA news.  I think it’s best to ignore the “rumors”, hence no speculation here.  In general though, my FA plan would be:

– Add depth (everywhere) with low-priced veterans on 1-2 year deals.

– Add a NT. Doesn’t have to be a great one (not many of those in the NFL), but a huge need if the team is moving to a 3-4.  This wouldn’t preclude taking one in the draft, but even then you need a back-up and it would be nice to not be overly reliant on Dixon.

– MAYBE add one marquee guy, as long as he is relatively young (<26-27).  Plenty of cap space, so if the team loves a guy like Smith or Long then take a shot.  Key is to pick the one they really like and let the others go.

– Don’t tie up cap space beyond this year.  This is a massive transition for the Eagles, and the fact is that Howie/Chip themselves don’t know how it’s going to shake out.  They key is to bolster the roster while maintaining cap flexibility for the next couple years.  With so many moving parts, it’s impossible to say who fits and who doesn’t, so throwing big money around is very risky.

Conversely, if you preserve space until the rest of the foundation is together, you have a much clearer picture of where your needs are and which impact FAs fit the team best.

Patience is the key, though it remains to be seen if the Eagles new front office has any.

Oh, and I wouldn’t consider, even for a moment, giving up the #4 pick for Revis.

Now the 2010 draft table:

As noted by a commenter yesterday, TJ Ward has had a pretty good start to his career and was drafted 1 spot after Nate Allen.  This system has Ward as one of the biggest reaches; he had one of the lowest included prospect ratings.

– Taylor Mays presents an interesting case.  His aggregate scouting rating was pretty good, hence the high rating here.  However, I remember a LOT of commentators downgrading him.  He may be a good case of why the system needs more sets of ratings.  My guess is there were a lot or scouts who did not score him as highly as ESPN or NFL.com.

– Torell Troup looks like a big mistake by the Bills, though injuries have wrecked his career, so its hard to judge him.  It’s worth noting that Terrance Cody and Linval Joseph were both ranked significantly higher and available at the Bills pick.

– Regarding Cody (since he came up yesterday as well), he’s been pretty inconsistent, and was supplanted by Kemoeatu in the starting line-up, but PFF actually graded him better last year than Kemoeatu.

Also, I realized I didn’t do a good job of showing the big picture.  No individual player’s ranking will be perfect in any system.  The goal is to create a system that, overall, does a better job of valuing players.  Here is a table showing the actual first round of 2010 with the PVM top 32.  We’ll delve deeper into this type of comparison some other time, for now you can analyze/compare them and make up your own mind.

One last note:  This system is by no means a finished product.  To that end, if you have an idea for improving it, please let me know.  No pride of ownership here, I just want to create the best system possible.

As of this writing, the Eagles haven’t made any FA news, so I’ll continue with our draft talk. Today we will look at the 2010 NFL Draft (B-Graham year).  For those of you who noticed, I’ve skipped 2011 for two reasons:  2010 is more interesting, and I can only find 1 set of prospect ratings for 2011.

I haven’t really emphasized this yet, but a MAJOR part of the PVM ranking system is the consensus prospect ratings.  The average rating is far more important than the PVM adjustment.  As a result, if I have only one set of ratings the rankings will not be nearly as valuable.  To that end, if any of you know where I can find 2011 prospect grades/ratings (numerical), please either email me, tweet me, or respond via comments.  I still hope to find another set to work with (right now I only have ESPN’s).

In the meantime, here are the top 32 prospects for 2010 via the PVM system.  For these, I used ESPN and NFL.com ratings, though the NFL.com ratings had to be adjusted to a 100 points scale.  Remember, the right-most column is the player’s actual draft pick minus his PVM Rank.  So players with a negative number were drafted HIGHER than their PVM Ranking.  Only players chosen in the top 2 rounds are included in the analysis.

I’ll break this down so it’s easier to see in a second, but a couple of big notes first:

– Jimmy Clausen shoots WAY up the board.  While the PVM adjustment helped, this is mainly due to the fact that his average rating was 92.27.

– CJ Spiller jumps into the top 5.

– Brandon Graham, regardless of the trade involved, appears to have been taken right where he should have.  However, both JPP and Derrick Morgan rank higher and were on the board when the Eagles picked.

To be fair, all three players were taken by the 16th pick (Eagles originally has the 24th), so had the team not traded up there was a strong chance they would not have been able to choose any of them.

– The biggest “overdraft” in the top ten belongs to Rolando McClain, narrowly beating Trent Williams.  Needless to say I feel pretty good about the PVM system here…

Now let’s take a narrower look.  Here are the most OVERDRAFTED:

Really like this one for obvious reasons.

– Tyson Alualu and Tim Tebow stand out as the worst “value” picks in the first round.  Tebow, even with the positional impact bump, rated as the 44th prospect (he was taken 25th).  The Jaguars, picking Aluala, were derided (correctly) at the time.  If I remember correctly, the Jags justified it by saying he was the best guy on their board.  However, it was pretty clear at the time that they could have traded down and still got him.

It’s a valuable reminder that there is a lot more to “winning the draft” than just setting your board more accurately than other teams (better scouting).  As I’ve explained, the “skill” portion of the draft involves moving around so you get the guys you like at draft spots where they offer maximum value.

– Nate Allen’s here as well, drafted 12 spots ahead of where the PVM ranking has him.  This was a 100% “need pick” and clearly hasn’t worked out the way the Eagles hoped.

– Conversely, Anthony Davis has worked out for the 49ers, despite being taken 12 spots ahead of where he was ranked.

One big note here that, although obvious, I feel compelled to explain:  Every team will have their own individual position values, which means each team’s PVM Board (if they made one) will look different from the one above.  For that reason, it’s tough to grade teams on individual picks because we don’t know what their internal positional ratings are.  It’s possible that the 49ers place a higher premium on OT than the rest of the league (likely in fact).  In that scenario, their PVM board (with no regard for scouting), may have had Anthony Davis (and Iupati) ranked higher, which might justify taking both OL well ahead of their rankings above.

Prior to this year’s draft, I hope to compile individual PVM ratings for each team.  It will probably be very “noisy” due to various contract issues, but it also might help us infer what teams might do.  Perhaps I’ll even put an “ideal” mock draft up, showing what each team should do with their picks under this system.

Now back to 2010.  Here are the most UNDERDRAFTED players:

Some BIG hits here as well as some BIG misses.

– Clausen I mentioned above.  Though he appears to be a miss, the idea here is that at the time of the draft, his potential upside warranted a much higher draft choice than 48 overall.  That said, it’s conceivable that he has just been lost behind Cam Newton and can still be a productive player on the right team (though I’m not holding my breath).

– Sean Lee and Dez Bryant both fly up the board here.  Bryant ranked #10 overall by PVM.

– At the bottom of that chart we can see JPP and Derrick Morgan, who both ranked as top 10 prospects by PVM.

– Charles Brown jumps a full round by PVM, though it’s still unclear how his career will pan out (injured his knee this year).

– Terrence Cody and Sergio Kindle both rank high, and coincidentally or not, were both selected by the Ravens.  Though neither player has played up to their projections, it’s interesting to note the Ravens’ multiple selections. I’ll be keeping my eye on which teams show up more/less on the over/under-draft lists.  In theory, teams that are applying a system of this type should find their way onto the under list with some frequency.

That’s all for today.  Hopefully we’ll have some Eagles FA news to discuss soon.  Again, if anyone knows anywhere to get past prospect rankings, please let me know.