- Joined
- Apr 7, 2013
- Messages
- 123,401
Wharton professor studies NFL draft, NFL folly
Mike Jensen, Inquirer Sports Writer
Posted: Tuesday, September 24, 2013, 2:02 AM
One of Wharton School professor Cade Massey's recent research projects started as a consulting deal with a National Football League team.
The franchise asked Massey, "Who is best at the draft? Who should we be paying attention to?"
His answer was surprising.
"I went out and looked at the data," said Massey, sitting in his office on the fifth floor of Huntsman Hall at Penn.
"It turned out, there are no differences in teams' abilities to draft."
There are clearly "huge differences" in outcomes, Massey was quick to point out.
"Some teams have great years, other teams have bad years - and it matters," Massey said. "But those differences aren't persistent year-to-year, which tells me that they are chance driven. Something between 95 and 100 percent - I'm not exaggerating - of team differences in the draft is driven by chance."
There is skill involved in selecting players, the professor said.
"It's just that teams are equally skilled, in a very uncertain environment," Massey said.
That eye-opening conclusion probably won't be received with a lot of "darn rights" within the NFL. But Massey, who previously worked at Duke and Yale, and coauthors NFL power rankings that were published in the Wall Street Journal, has punctured NFL balloons before.
An earlier paper he cowrote, titled "Loser's Curse: Overconfidence vs. Market Efficiency in the NFL Draft," basically flipped established beliefs about the worth of NFL draft choices. It got plenty of attention across sports.
On a basic level, this all makes sense. How much credit do the Patriots deserve for drafting Tom Brady in the sixth round? If they'd known Brady was going to be half as good as he turned out to be, they obviously wouldn't have let everyone else have a crack at him for five-plus rounds.
But Massey's work goes deeper than that, into the real value of picks. For years, NFL teams based the worth of draft choices on something that came to be known as The Chart, and gave a value to each pick.
Except, according to research data collected by Massey and coauthor Richard Thaler, it was all wrong. Top choices were overvalued once you factored in their salaries.
"The genesis of that paper is the '99 draft," Massey said. The consensus was that Kentucky quarterback Tim Couch was the top pick, as he turned out to be. Of course, Couch didn't turn out to be the top QB. That was the year Donovan McNabb was drafted.
"We thought when you looked at '83, the lesson should have been, you don't know which of these quarterbacks are going to be the good ones," Massey said. "[Dan] Marino was the sixth [QB] taken. [John] Elway was first, but there's [Tony] Eason and [Todd] Blackledge in between."
For the football-history impaired, Marino was a bit better than Blackledge or Eason. But Massey's point isn't about scouting ability. (Alleged character issues were the reason for Marino's fall in the draft.) It's about overconfidence. "They're too sure that they can predict the future," he said.
That's Massey's real interest. The NFL draft happens to provide the data.
"This isn't about the NFL draft," he said.
His field, Massey explained, has shown very clearly that people don't appreciate how much uncertainty there is in life. And now he's trying to come up with strategies that "are optimal when the environment is more uncertain."
Massey worked as a long-term consultant for an NFL team that didn't want to be identified, he said. (Not the Eagles.) He is friends with plenty of analytic types, including new 76ers general manager Sam Hinkie, whose brains and creativity he raves about.
Massey is offering ideas that aren't wedded to sports alone.
"You should play lottery environments very differently than math-problem environments," Massey said. "With math problems, there's a right answer. Work hard enough and you'll get the right answer. If you can't get it, find the smartest guy in the room and he'll have the right answer."
And lotteries?
"You don't spend any time trying to get the right number because it doesn't make any difference," he said. "You try to get as many draws as possible or spend as little money as possible."
One of Massey's central tenets: People tend to think of life as more math problem than lottery. There's some mix, obviously. But they've got the ratio wrong. Given the strong element of chance, he said, you don't throw up your hands and give up.
The prescriptions are very pedestrian, he added.
"As many draws as possible. Minimize cost. Value the process," he said.
That last one is interesting. If there is more uncertainty, Massey believes, the process should actually be valued more.
"You talk to some teams - these guys put in months and months, the scouts do, building the profiles of all the players," Massey said. "They build the draft boards. And then some guys, some owners, some GMs, will come in late in the process, even draft day, and shuffle some of these boards, just completely violating the process."
That would be fine, Massey said, if there were a right answer.
"But in a situation where there is no right answer, you're completely flouting a process, which is going to screw up the organization," Massey said.
-------------
Study: NFL Teams Have No Idea What They're Doing In The Draft
Today the Philadelphia Inquirer profiles Cade Massey, a professor at Penn's Wharton School of business. Already with a study under his belt arguing that the conventional wisdom of the Draft Value Chart is all wrong, Massey was contracted by an unnamed NFL team to study the history of the draft for market inequalities. He discovered something that won't come as a surprise to football fans: the draft is kind of a crap shoot.
There is skill in making individual picks, Massey says, but the fact that draft success isn't sustainable points to the conclusion that every team is fairly evenly matched. What seem to be indicators that drafting is a talent, like the Lions' drought or the Patriots' boom of a decade or so ago, are statistically expected aberrations.
"Some teams have great years, other teams have bad years - and it matters," Massey said. "But those differences aren't persistent year-to-year, which tells me that they are chance driven. Something between 95 and 100 percent - I'm not exaggerating - of team differences in the draft is driven by chance."
If you take issue with that, you'll have to math it out with the math guy; I'm just passing things along. But Massey's field of study seems perfectly designed to tackle the NFL draft— according to his site, his expertise in psychology and economics hones in on "judgment under uncertainty, with a focus on optimism, overconfidence, and learning."
Last year, he co-authored a study with the University of Chicago's Richard Thaler. Entitled "Loser's Curse: Overconfidence vs. Market Efficiency in the NFL Draft," it aimed to determine whether there are any patterns in how front offices value draft picks, and if those patterns expose an opportunity for greater value.
The study is embedded below, but here's a talking point: teams overvalue higher picks in part because they overvalue their own judgment in evaluating players, and that "overconfidence is exacerbated by information." The more front offices know about a prospect, the more they think they know, and they also assume other teams value that prospect as highly, creating a feedback loop that pushes players higher up the draft board than they may deserve.
In terms of practical takeaways, the study says to toss out that hoary chestnut, the Draft Value Chart. You know the one: teams have consulted it for decades to put a rough value on any trade including multiple picks. (Example: the second overall pick would be exactly equal to a sixth and a 16th.)
Massey says the chart is wrong, because it doesn't take into account salaries in a salary cap league. His study looks at "surplus value," or what a player actually gives you compared to what would be expected for his contract worth. Because rookie contracts keep salaries artificially low, the surplus values of draft picks are nearly always positive. But some are more valuable than others.
That treasured first pick in the draft is, according to this analysis, actually the least valuable pick in the first round! To be clear, the player taken with the first pick does have the highest expected performance, but he also has the highest salary, and in terms of performance per dollar, is less valuable than most players taken in the second round.
Here's the chart showing the surplus value of picks from drafts between 1994-2008 (when, it must be noted, the lack of a draft slotting system severely inflated the top picks' contracts). According to the research, teams gained more value from drafting late in the first round than they did early in the first.
Massey and Thaler have advice for GMs: only suckers trade up. Over those 14 years of drafts, they calculated the outcomes of every possible 2-for-1 trade for a first rounder using the Draft Value Chart, and found "overwhelming evidence that a team would do better in the draft by trading down." The team that would have traded down would have gained an average of 5.4 man-starts per season, with roughly the same amount of Pro Bowl appearances, at a cheaper cost.
The study was naturally controversial, in part due to misreadings. It deals only in probabilities, not in individual picks. If you trade up and land a player who turns out to be a superstar, it was a good trade. The study merely says that in most cases, that possibility doesn't justify the risk.
It also has little chance of making an impact in actual front office behavior, because of the very psychological barriers it cites. In terms of "impact"—tickets sold, media coverage garnered, general excitement—a top pick is always going to be desirable. Even more important is the corollary: the fear of missing out on a superstar. That's the kind of thing that costs GMs their jobs, and leads to moves that look sexy in the short term but don't work out over time.
Mike Jensen, Inquirer Sports Writer
Posted: Tuesday, September 24, 2013, 2:02 AM
One of Wharton School professor Cade Massey's recent research projects started as a consulting deal with a National Football League team.
The franchise asked Massey, "Who is best at the draft? Who should we be paying attention to?"
His answer was surprising.
"I went out and looked at the data," said Massey, sitting in his office on the fifth floor of Huntsman Hall at Penn.
"It turned out, there are no differences in teams' abilities to draft."
There are clearly "huge differences" in outcomes, Massey was quick to point out.
"Some teams have great years, other teams have bad years - and it matters," Massey said. "But those differences aren't persistent year-to-year, which tells me that they are chance driven. Something between 95 and 100 percent - I'm not exaggerating - of team differences in the draft is driven by chance."
There is skill involved in selecting players, the professor said.
"It's just that teams are equally skilled, in a very uncertain environment," Massey said.
That eye-opening conclusion probably won't be received with a lot of "darn rights" within the NFL. But Massey, who previously worked at Duke and Yale, and coauthors NFL power rankings that were published in the Wall Street Journal, has punctured NFL balloons before.
An earlier paper he cowrote, titled "Loser's Curse: Overconfidence vs. Market Efficiency in the NFL Draft," basically flipped established beliefs about the worth of NFL draft choices. It got plenty of attention across sports.
On a basic level, this all makes sense. How much credit do the Patriots deserve for drafting Tom Brady in the sixth round? If they'd known Brady was going to be half as good as he turned out to be, they obviously wouldn't have let everyone else have a crack at him for five-plus rounds.
But Massey's work goes deeper than that, into the real value of picks. For years, NFL teams based the worth of draft choices on something that came to be known as The Chart, and gave a value to each pick.
Except, according to research data collected by Massey and coauthor Richard Thaler, it was all wrong. Top choices were overvalued once you factored in their salaries.
"The genesis of that paper is the '99 draft," Massey said. The consensus was that Kentucky quarterback Tim Couch was the top pick, as he turned out to be. Of course, Couch didn't turn out to be the top QB. That was the year Donovan McNabb was drafted.
"We thought when you looked at '83, the lesson should have been, you don't know which of these quarterbacks are going to be the good ones," Massey said. "[Dan] Marino was the sixth [QB] taken. [John] Elway was first, but there's [Tony] Eason and [Todd] Blackledge in between."
For the football-history impaired, Marino was a bit better than Blackledge or Eason. But Massey's point isn't about scouting ability. (Alleged character issues were the reason for Marino's fall in the draft.) It's about overconfidence. "They're too sure that they can predict the future," he said.
That's Massey's real interest. The NFL draft happens to provide the data.
"This isn't about the NFL draft," he said.
His field, Massey explained, has shown very clearly that people don't appreciate how much uncertainty there is in life. And now he's trying to come up with strategies that "are optimal when the environment is more uncertain."
Massey worked as a long-term consultant for an NFL team that didn't want to be identified, he said. (Not the Eagles.) He is friends with plenty of analytic types, including new 76ers general manager Sam Hinkie, whose brains and creativity he raves about.
Massey is offering ideas that aren't wedded to sports alone.
"You should play lottery environments very differently than math-problem environments," Massey said. "With math problems, there's a right answer. Work hard enough and you'll get the right answer. If you can't get it, find the smartest guy in the room and he'll have the right answer."
And lotteries?
"You don't spend any time trying to get the right number because it doesn't make any difference," he said. "You try to get as many draws as possible or spend as little money as possible."
One of Massey's central tenets: People tend to think of life as more math problem than lottery. There's some mix, obviously. But they've got the ratio wrong. Given the strong element of chance, he said, you don't throw up your hands and give up.
The prescriptions are very pedestrian, he added.
"As many draws as possible. Minimize cost. Value the process," he said.
That last one is interesting. If there is more uncertainty, Massey believes, the process should actually be valued more.
"You talk to some teams - these guys put in months and months, the scouts do, building the profiles of all the players," Massey said. "They build the draft boards. And then some guys, some owners, some GMs, will come in late in the process, even draft day, and shuffle some of these boards, just completely violating the process."
That would be fine, Massey said, if there were a right answer.
"But in a situation where there is no right answer, you're completely flouting a process, which is going to screw up the organization," Massey said.
-------------
Study: NFL Teams Have No Idea What They're Doing In The Draft
Today the Philadelphia Inquirer profiles Cade Massey, a professor at Penn's Wharton School of business. Already with a study under his belt arguing that the conventional wisdom of the Draft Value Chart is all wrong, Massey was contracted by an unnamed NFL team to study the history of the draft for market inequalities. He discovered something that won't come as a surprise to football fans: the draft is kind of a crap shoot.
There is skill in making individual picks, Massey says, but the fact that draft success isn't sustainable points to the conclusion that every team is fairly evenly matched. What seem to be indicators that drafting is a talent, like the Lions' drought or the Patriots' boom of a decade or so ago, are statistically expected aberrations.
"Some teams have great years, other teams have bad years - and it matters," Massey said. "But those differences aren't persistent year-to-year, which tells me that they are chance driven. Something between 95 and 100 percent - I'm not exaggerating - of team differences in the draft is driven by chance."
If you take issue with that, you'll have to math it out with the math guy; I'm just passing things along. But Massey's field of study seems perfectly designed to tackle the NFL draft— according to his site, his expertise in psychology and economics hones in on "judgment under uncertainty, with a focus on optimism, overconfidence, and learning."
Last year, he co-authored a study with the University of Chicago's Richard Thaler. Entitled "Loser's Curse: Overconfidence vs. Market Efficiency in the NFL Draft," it aimed to determine whether there are any patterns in how front offices value draft picks, and if those patterns expose an opportunity for greater value.
The study is embedded below, but here's a talking point: teams overvalue higher picks in part because they overvalue their own judgment in evaluating players, and that "overconfidence is exacerbated by information." The more front offices know about a prospect, the more they think they know, and they also assume other teams value that prospect as highly, creating a feedback loop that pushes players higher up the draft board than they may deserve.
In terms of practical takeaways, the study says to toss out that hoary chestnut, the Draft Value Chart. You know the one: teams have consulted it for decades to put a rough value on any trade including multiple picks. (Example: the second overall pick would be exactly equal to a sixth and a 16th.)
Massey says the chart is wrong, because it doesn't take into account salaries in a salary cap league. His study looks at "surplus value," or what a player actually gives you compared to what would be expected for his contract worth. Because rookie contracts keep salaries artificially low, the surplus values of draft picks are nearly always positive. But some are more valuable than others.
That treasured first pick in the draft is, according to this analysis, actually the least valuable pick in the first round! To be clear, the player taken with the first pick does have the highest expected performance, but he also has the highest salary, and in terms of performance per dollar, is less valuable than most players taken in the second round.
Here's the chart showing the surplus value of picks from drafts between 1994-2008 (when, it must be noted, the lack of a draft slotting system severely inflated the top picks' contracts). According to the research, teams gained more value from drafting late in the first round than they did early in the first.
Massey and Thaler have advice for GMs: only suckers trade up. Over those 14 years of drafts, they calculated the outcomes of every possible 2-for-1 trade for a first rounder using the Draft Value Chart, and found "overwhelming evidence that a team would do better in the draft by trading down." The team that would have traded down would have gained an average of 5.4 man-starts per season, with roughly the same amount of Pro Bowl appearances, at a cheaper cost.
The study was naturally controversial, in part due to misreadings. It deals only in probabilities, not in individual picks. If you trade up and land a player who turns out to be a superstar, it was a good trade. The study merely says that in most cases, that possibility doesn't justify the risk.
It also has little chance of making an impact in actual front office behavior, because of the very psychological barriers it cites. In terms of "impact"—tickets sold, media coverage garnered, general excitement—a top pick is always going to be desirable. Even more important is the corollary: the fear of missing out on a superstar. That's the kind of thing that costs GMs their jobs, and leads to moves that look sexy in the short term but don't work out over time.
