This analysis of Oakland A's general manager Billy Beane's famous "Moneyball" picks in the 2002 major league draft concludes that the Oakland GM's use of statistical methods to evaluate players has had mixed results. In my view, evaluating Moneyball based on the performance of just seven draft picks is a weak methodology, given the small numbers involved and the great influence of chance factors, such as fluke injuries, on individual player performance. Moreover, focusing on just one year's draft picks ignores the fact that Beane has actually been GM of the A's for 7 years. It also ignores the reality that he uses statistical analysis to evaluate free agent signings as well as draft picks.
A better measure of the Beane/Moneyball record is how well the A's have done overall during his tenure relative to the available financial resources. This approach considers the full impact of all of Beane's player acquisition decisions during his time in Oakland. Here, I compare the A's record during Beane's 7 full seasons as A's GM (1999-2005), to that of the New York Yankees and Boston Red Sox, usually regarded as the 2 most successful American League teams of the era.
From 1999 to 2005, Yankees won an average of 97 (of 162) games per season, for a total winning percentage of .602. The A's won 94 games per year for a winning percentage of .581. The Red Sox, in turn, won 92 games per year, for a .567 winning percentage. The A's made 4 playoff appearances during that time and narrowly missed the playoffs in two other seasons, exactly the same as the Red Sox; the Yankees reached the playoffs in all 7 years. Thus, the A's record under Beane is slighly worse than that of the Yankees and slightly better than that of the Red Sox. Quite impressive for the A's considering the strength of the comparison group.
But here's the kicker: During the period in question the Yankees payroll averaged $138 million per year, the Red Sox $103 million, and the A's a mere $42 million (figures calculated from data available here). The A's achieved roughly the same results as the other two teams, while spending only one third as much as the Yankees and about 40% of what the Red Sox shelled out. To put it another way, the A's during Beane's tenure spent $450,000 per win, while the Yankees spent $1.42 million and the Red Sox spent $1.12 million. Given that the Yankees and Red Sox are both regarded as well-run franchises among the best in baseball, Beane's vastly superior use of resources compared to them is extremely impressive.
To be sure, the above analysis is oversimplified. Ideally, we should compare Beane's A's to all other AL teams and not just to the Yankees and Red Sox. I do not have time to do all the calculations right now, but I highly doubt that any of the other AL franchises even comes close to the A's performance in wins relative to spending (a quick eyeball analysis suggests that only the White Sox, Angels, Twins, and possibly Indians could even approach the Yankees and Red Sox). National League teams are not a good comparison group because they rarely face AL teams in regular season games and so are not playing against the same competition.
Another potential methodological flaw is the implicit assumption that the relationship between increased spending and wins is linear. In reality, doubling spending will not double your wins even if the money is spent optimally. Nonetheless, there is no doubt that Beane got far greater bang from his bucks than the Yankees and Red Sox did from theirs, even if we take nonlinearity into account.
It is also true that not all the A's success can be attributed to Beane. However, Beane did pick nearly all the players, and the few left over from the prior regime were picked by his predecessor Sandy Alderson, who had begun to use statistical analysis himself and chose Beane as his successor in part because of the latter's interest in the new methods.
Finally, naysayers will point to the A's relatively weaker playoff performances during this time period. Both the Red Sox and Yankees did much better in the playoffs than the A's, who lost all four of their first round series. However, as Billy Beane himself pointed out in Moneyball, playoff series outcomes are heavily influenced by chance because of the small number of games involved and the high degree of randomness in baseball. The A's lost all four series by narrow 3-2 margins and several times were victimized by very bad luck. Had they been slightly luckier, their playoff record might have been much better.
In my view, evaluating Moneyball based on the performance of just seven draft picks is a weak methodology, given the small numbers involved and the great influence of chance factors, such as fluke injuries, on individual player performance.
Moreover, there's no control: how well did other teams' first-round draft picks do (most teams' first-round draft picks have mixed records) and how much did it cost them.
Regardless of the success of the draft, Beane's draft seemed to flunk some basic precepts of game theory and economics. A draft is a Dutch auction, and spending a first-round pick on someone no other team is going to pick in the first five (or even ten) rounds is overpaying, even if only in terms of opportunity cost, even if the player turns out to be a hidden-gem success.
Beane's teams have performed fantastically for their circumstances, however. Yet I wince when I see their unfortunate playoff outcomes described as victimization by bad luck. In school we just used the phrase "sample size" (162 >>> 5).
-- the A's have not been as successful as either the Yankees or RedSux during this period (just just their respective AL and World Series pennants). That is the definition of success in any sport, not average games won (just ask the 2001 Seattle Mariners how much historical weight goes with the title of "most wins by a team that did not win its league's pennant.")
-- the A's spend much of the seasons playing the Mariners, Angels and Texas, which no one can compare with 19 games between the Yanks and Sux (in the season series against each other, the Yanks and Sux usually end up close to even or over/under by about 2 games).
-- By the "average wins per dolar spent" standard, I am sure that there are other teams, including losing teams, that could celebrate their frugality. For example, how much did Atlanta spend per win during this period, and are they the mark for success in the NL (despite the fact that another team has been the NL representative in the World Series in each of those years)? How about Minnesota, inicluding its success in reaching the post-season during these years?
-- Lastly, the above comparions is based at least in part on the assumption that is qualitively better to spend little for 7 years and end up without zero championships than to spend a lot in a couple of those 7 years and win a championship (see 2001 Diamondbacks). Is one better than the other for the fans? How about the team?
The bottom line is that Beaneball should be evaluated based on the usual measures for success, not a lower "we were only planning for the 162 game season" measure (and, to be fair, the Yankees spending vs. World Series should also be judged on the same measure-if they win, well spent, if not, failure).
Oakland has a much lower dollars per marginal win than Atlanta.
Once upon a time baseball valued the 162-game season more than the exhibition playoff afterwards. The wildcard changed that. A five-game series between two very good teams is essentially a coin-toss that no general manager can hope to influence. While Oakland hasn't had much luck in the coin tosses, they've been almost as good as any team in baseball in the last decade, and much better than their peers with comparable budgets.
It seems like the only division which is fair to criticize, is the AL Central, which at times seems like it has 2 .350 level teams and 2-3 .600 level teams, which is probably a result to some degree of the divisional play. The AL West especially has the most bunching of all the divisions (no team is effectively out of it yet) and all the teams are fairly strong.
Ilya was working off the assumption that the Yankees and Red Sox were likely the two best teams, and his conclusion was that the A's were comparable. The win-loss record (if it shows anything) would support that. If we then accept that these are the three best teams in the AL, then we must face the fact that this season the Yankees will play 28 games against the best teams, the Red Sox will play 29 games against the best teams, and Oakland will merely play 19 games against the best teams. The advantage garnered by that (I make no claims to how great it is) might indeed be balanced by the paucity of games against the Devil Rays, but it's unlikely we're going to be able to figure it out.
What we can do is look at these teams' records outside their own division, but then we're going to be relying on a very small number of games against the AL Central in any given season. To get a reasonable sample we'll have to aggregate over several seasons, which forces us to assume that a win against the Twins in season X is equal to a win against the Twins in season Y.
So no, I'm not certain the Yankees and Red Sox have been better over the last seven years than the A's. I have to rely on my own subjective impressions, and reluctantly admit that I can't back them up with all the statistical evidence I'd hope. And I'm pretty sure I made Rob Neyer cry, too.
Also, I'm sure there's some big moral debate to be had regarding how much one should value the regular season vs. the playoffs, like Ted Frank and El Blogero may end up having. But surely we can all agree that the measure of Beane's success as a manager can't be his ability to foresee which guys happen to play better in October than in April? Building a team that's most likely to win any given game during the season is going to give one a pretty good chance at getting a ring at the end of it. With the playoffs so short, that's pretty much all a manager can do...improve the odds his team will happen to win four games out of seven.
Prince Fielder, Mil
Jeff Francis, Col
Jeremy Hermida, Fla
Joe Saunders, LAA
Khalil Greene, SD
Scott Kazmir, NYM--now plays for TB
Nick Swicher, Oak
Cole Hamels, Phi
Jeff Franceur, ATL
Joe Blanton, OAK
Matt Cain, SF
Mark Teahen, OAK--now for KC
Others have been up in the majors, but the 12 above meet the somewhat subjective definition of guys who will be MLB players for the foreseeable future. Of the 12, three were drafted by the Oakland A’s with their seven picks. That means the other MLB teams have thus far gone 9/34. Or about 26 percent. Oakland went 3/7, or 42 percent. Even judging Beane by one round of one draft, he did better with his seven picks than did the other 29 teams with their 34 picks, on average.
I had to laugh out loud that Jeremy Brown is considered a strike against Beane. It’s been a while since I read Moneyball, but my recollection is that, as the article notes, the scouts were simply incredulous that the short, slow, fat Brown had been drafted in the first round. No other MLB team had him anywhere near the first round on its draft board. He was, if memory serves, a guy expected to be drafted no earlier than the 10th round or so. Now, one can debate whether Beane should have maximized the value of his pick by trading down, but it’s worth noting that drafting baseball players is a really, really tough business. There are first round busts, several of them, every year. Jeremy Brown was considered a marginal prospect by nearly every team but the A’s, and four years later, he has played in the major leagues. That’s nothing to sniff at.
The Twins have also been around $450,000 per win from that time period. The Marlins have been around $470,000. The Brewers, nobody's have been at $580,000, while the Royals have been around $525,000.
I think it's problematic to compare people along one axis ($ per win) when they're actually competing along another axis (wins/pennants).
Brian Cashman is not being asked to do his best on a budget--if he were, he may well do just as well as Billy Beane. On the other hand, if Beane were handed the keys to the Yankees there's no guarantee that his techniques would scale. Look at Dan Duquette of the Expos and Duquette of the BoSox.
In short, if you enjoyed Moneyball, you'll enjoy MoneyLaw.
For example, Lewis referred to the A's 2001 draft as "an expensive disaster." But the A's two 2001 first rounders -- Bobby Crosby and Jeremy Bonderman -- are both very good major leaguers, better than any of the 7 first rounders from the "Moneyball" draft. And the two players with the most success from the "Moneyball" draft -- Nick Swisher and Joe Blanton -- are guys that the scouts liked, as well.
Lewis wrote an entertaining book, but, like some of his other works, he tended to too often paint things in stark black and white, rather than recognize the shades of grey.
I think Moneyball is really just about identifying and exploiting undervalued commodities. Some teams are better at it than others, and evaluating results requires a bit of context.
But then, when it comes time to evaluate the A's playoff performace, we're told that we should not be so harsh because they have been victims of bad luck.
Your post compelled an entire post in response. I hope to see you and your readers over at MoneyLaw.
Jim
Here's the properly functioning link.
No, it's not a Dutch auction. The budget that Beane was given wasn't anywhere near enough to pay million-dollar signing bonuses for even one prospect, let alone several; he HAD to find players that other teams thought were not valuable. And, as mentioned in the book, he used a high draft choice as his way of telling a seemingly unlikely prospect that the A's really believed in him, as well as bestowing prestige on the player.
I don't know about the facts you cover, but your general approach is definitely the right one to take.
It would be best to use some objective criteria for "making it": e.g., for hitters: games played OR at bats OR total number of bases hit; for pitchers: innings pitched.
Well, it might be hard to figure out if it weren't such a blowout. Boston is really famous and everything, but since 1999 the AL East's overall winning percentage is .502; the AL West has gone .531. That's including the Yankees and the A's; if you excludde them the records are .478 and .498. Tampa Bay's *best* record in that time is 70-91; all AL West teams combined have been worse three times in seven years. The AL West has had a better overall record every year since 1999, including this year. Yeah, I think all those Tampa games might make up for it.
The A's outperformed the Red Sox by much greater margins before the Sox adopted Moneyball principles starting in 2003. I did not break down the stats by pre- and post-Moneyball Red Sox performance, but in 1999-2002, the A's greatly outperformed the Sox in every year except 1999 (Billy Beane's first as GM). The Red Sox won 94, 85, 82, and 93 games during those 4 years. The A's 87, 91, 102, and 103. On average the A's outdid the Sox by over 7 games per year during Boston's pre-Moneyball days. Since 2003, the Red Sox have slightly outperformed the A's in terms of wins. In both periods, of course, the Sox spent vastly more money than Oakland.
I'm less concerned by the criticism that the A's built so much on their Big 3 at this time, in that this may be more consistent with Moneyball thinking. Given the A's budget constraint, Beane basically chose to allocate resources to starting pitching over relievers, based on the idea that almost any decent pitcher could be turned into a capable reliever. So by refusing to overpay for relief, he was able to afford to keep his starters (and I think he made some trades in there too), and by appropriately valuing starting pitching to avoid trading starters for other lower-valued investments. For position players the tradeoff at the margin tells us first, to trade off hitting for fielding and second, OBP for BA.
So I think the real criteria for measuring the thesis is in the counterfactual of where the team would otherwise have finished--holding budget constant.
The most mysterious part of the story to me is how the A's have been so dominant in August every year (best month record of any team in any month over the past five years), but this hasn't carried over to September and October. I could see a story that their post-season woes were casued by a lack of depth--i.e., that budget pinches not in your starting lineup, but with respect to bench strength and middle relief. This is where it seems to me that the Yankees had been able overwhelm their opponents with money is by being deeper, and so being able to deal with injuries and single game/batter matchups better in a short series. But if it were merely depth, then it is hard to see why the A's consistently play so well in August.
*******
This is a good point. On those rare recent occasions when Beane is given permission and a "blank check" to go get a high priced free agent in the off-season, he seems to whiff pretty badly (Kendall, Loaiza), but still does very well at the sluggers flea market (F. Thomas, Jaha, even Durazo for a while).
On the other hand, Cashman/Steinbrenner (never sure who's calling the shots) blew WAAAAAY too much cash/prospects on Pavano and R. Johnson but still can find a few useful cheap retreads that no one else wanted (Chacon, Small, e.g. in 2005).