The Volokh Conspiracy - Evaluating Billy Beane and Moneyball:

This analysis of Oakland A's general manager Billy Beane's famous "Moneyball" picks in the 2002 major league draft concludes that the Oakland GM's use of statistical methods to evaluate players has had mixed results. In my view, evaluating Moneyball based on the performance of just seven draft picks is a weak methodology, given the small numbers involved and the great influence of chance factors, such as fluke injuries, on individual player performance. Moreover, focusing on just one year's draft picks ignores the fact that Beane has actually been GM of the A's for 7 years. It also ignores the reality that he uses statistical analysis to evaluate free agent signings as well as draft picks.

A better measure of the Beane/Moneyball record is how well the A's have done overall during his tenure relative to the available financial resources. This approach considers the full impact of all of Beane's player acquisition decisions during his time in Oakland. Here, I compare the A's record during Beane's 7 full seasons as A's GM (1999-2005), to that of the New York Yankees and Boston Red Sox, usually regarded as the 2 most successful American League teams of the era.

From 1999 to 2005, Yankees won an average of 97 (of 162) games per season, for a total winning percentage of .602. The A's won 94 games per year for a winning percentage of .581. The Red Sox, in turn, won 92 games per year, for a .567 winning percentage. The A's made 4 playoff appearances during that time and narrowly missed the playoffs in two other seasons, exactly the same as the Red Sox; the Yankees reached the playoffs in all 7 years. Thus, the A's record under Beane is slighly worse than that of the Yankees and slightly better than that of the Red Sox. Quite impressive for the A's considering the strength of the comparison group.

But here's the kicker: During the period in question the Yankees payroll averaged $138 million per year, the Red Sox $103 million, and the A's a mere $42 million (figures calculated from data available here). The A's achieved roughly the same results as the other two teams, while spending only one third as much as the Yankees and about 40% of what the Red Sox shelled out. To put it another way, the A's during Beane's tenure spent $450,000 per win, while the Yankees spent $1.42 million and the Red Sox spent $1.12 million. Given that the Yankees and Red Sox are both regarded as well-run franchises among the best in baseball, Beane's vastly superior use of resources compared to them is extremely impressive.

To be sure, the above analysis is oversimplified. Ideally, we should compare Beane's A's to all other AL teams and not just to the Yankees and Red Sox. I do not have time to do all the calculations right now, but I highly doubt that any of the other AL franchises even comes close to the A's performance in wins relative to spending (a quick eyeball analysis suggests that only the White Sox, Angels, Twins, and possibly Indians could even approach the Yankees and Red Sox). National League teams are not a good comparison group because they rarely face AL teams in regular season games and so are not playing against the same competition.

Another potential methodological flaw is the implicit assumption that the relationship between increased spending and wins is linear. In reality, doubling spending will not double your wins even if the money is spent optimally. Nonetheless, there is no doubt that Beane got far greater bang from his bucks than the Yankees and Red Sox did from theirs, even if we take nonlinearity into account.

It is also true that not all the A's success can be attributed to Beane. However, Beane did pick nearly all the players, and the few left over from the prior regime were picked by his predecessor Sandy Alderson, who had begun to use statistical analysis himself and chose Beane as his successor in part because of the latter's interest in the new methods.

Finally, naysayers will point to the A's relatively weaker playoff performances during this time period. Both the Red Sox and Yankees did much better in the playoffs than the A's, who lost all four of their first round series. However, as Billy Beane himself pointed out in Moneyball, playoff series outcomes are heavily influenced by chance because of the small number of games involved and the high degree of randomness in baseball. The A's lost all four series by narrow 3-2 margins and several times were victimized by very bad luck. Had they been slightly luckier, their playoff record might have been much better.

	[Ilya Somin, August 20, 2006 at 7:53am] Trackbacks Evaluating Billy Beane and Moneyball: This analysis of Oakland A's general manager Billy Beane's famous "Moneyball" picks in the 2002 major league draft concludes that the Oakland GM's use of statistical methods to evaluate players has had mixed results. In my view, evaluating Moneyball based on the performance of just seven draft picks is a weak methodology, given the small numbers involved and the great influence of chance factors, such as fluke injuries, on individual player performance. Moreover, focusing on just one year's draft picks ignores the fact that Beane has actually been GM of the A's for 7 years. It also ignores the reality that he uses statistical analysis to evaluate free agent signings as well as draft picks. A better measure of the Beane/Moneyball record is how well the A's have done overall during his tenure relative to the available financial resources. This approach considers the full impact of all of Beane's player acquisition decisions during his time in Oakland. Here, I compare the A's record during Beane's 7 full seasons as A's GM (1999-2005), to that of the New York Yankees and Boston Red Sox, usually regarded as the 2 most successful American League teams of the era. From 1999 to 2005, Yankees won an average of 97 (of 162) games per season, for a total winning percentage of .602. The A's won 94 games per year for a winning percentage of .581. The Red Sox, in turn, won 92 games per year, for a .567 winning percentage. The A's made 4 playoff appearances during that time and narrowly missed the playoffs in two other seasons, exactly the same as the Red Sox; the Yankees reached the playoffs in all 7 years. Thus, the A's record under Beane is slighly worse than that of the Yankees and slightly better than that of the Red Sox. Quite impressive for the A's considering the strength of the comparison group. But here's the kicker: During the period in question the Yankees payroll averaged $138 million per year, the Red Sox $103 million, and the A's a mere $42 million (figures calculated from data available here). The A's achieved roughly the same results as the other two teams, while spending only one third as much as the Yankees and about 40% of what the Red Sox shelled out. To put it another way, the A's during Beane's tenure spent $450,000 per win, while the Yankees spent $1.42 million and the Red Sox spent $1.12 million. Given that the Yankees and Red Sox are both regarded as well-run franchises among the best in baseball, Beane's vastly superior use of resources compared to them is extremely impressive. To be sure, the above analysis is oversimplified. Ideally, we should compare Beane's A's to all other AL teams and not just to the Yankees and Red Sox. I do not have time to do all the calculations right now, but I highly doubt that any of the other AL franchises even comes close to the A's performance in wins relative to spending (a quick eyeball analysis suggests that only the White Sox, Angels, Twins, and possibly Indians could even approach the Yankees and Red Sox). National League teams are not a good comparison group because they rarely face AL teams in regular season games and so are not playing against the same competition. Another potential methodological flaw is the implicit assumption that the relationship between increased spending and wins is linear. In reality, doubling spending will not double your wins even if the money is spent optimally. Nonetheless, there is no doubt that Beane got far greater bang from his bucks than the Yankees and Red Sox did from theirs, even if we take nonlinearity into account. It is also true that not all the A's success can be attributed to Beane. However, Beane did pick nearly all the players, and the few left over from the prior regime were picked by his predecessor Sandy Alderson, who had begun to use statistical analysis himself and chose Beane as his successor in part because of the latter's interest in the new methods. Finally, naysayers will point to the A's relatively weaker playoff performances during this time period. Both the Red Sox and Yankees did much better in the playoffs than the A's, who lost all four of their first round series. However, as Billy Beane himself pointed out in Moneyball, playoff series outcomes are heavily influenced by chance because of the small number of games involved and the high degree of randomness in baseball. The A's lost all four series by narrow 3-2 margins and several times were victimized by very bad luck. Had they been slightly luckier, their playoff record might have been much better.