pageok
pageok
pageok
[Michael Abramowicz, guest-blogging, January 29, 2008 at 4:04pm] Trackbacks
A Quick-and-Dirty Empirical Defense of Prediction Markets:

From the comments, it seems that some readers would find empirical evidence more persuasive than theoretical arguments. Consider the figure below. It aggregates a total of 145,388 trades on Major League Baseball games in 2005. Each contract would pay off $10.00 if the specified team won the game.

The x-axis represents different transaction prices, and the y-axis shows, for all trades at that price or up to 10 cents higher, the probability that the team in fact won the game. Note that the points conform fairly closely to a 45-degree line.

This doesn't prove that prediction markets are perfect predictors. Maybe some other prediction mechanism could be calibrated in this way yet have more predictions toward the ends of the probability continuum. A savvy trader might still be able to profit by running statistical tests to find systematic biases in prices.

And of course, we can't necessarily extrapolate from baseball to other types of forecasting environments. Some organizations (like insurance companies) may already have very sophisticated procedures for developing probabilistic forecasts, and this doesn't prove that prediction markets would enhance those.

Nonetheless, they ought to be good enough for government work. And most corporate work too.

statfan (mail):
I want to see not a comparison of prediction markets against actual data, but against other prediction methods. Say, the vegas odds, for sports.
1.29.2008 4:44pm
Sasha Volokh (mail) (www):
I don't understand what's on the y-axis. What's "the probability that the team in fact won the game"?
1.29.2008 4:48pm
Justin (mail):
I think what he means is if n is the number of stocks that were sold atthat point, x/n, where x the number of stocks that expired at 100.
1.29.2008 4:53pm
Nathan_M (mail):
Isn't the y-axis the number of games the contract paid off, divided by the total number of games at that particular price?

So if there were, say, 600 games where the contract price was $5, and the team being traded won 300 of them, the y-value at $5 would be 0.5.
1.29.2008 4:58pm
Curt Fischer:
I agree with Sasha; the y axis is confusing. If Nathan_M is right, a much better term than "probability" for the y-axis is "frequency".
1.29.2008 5:06pm
Daniel Chapman (mail):
Ok I read your response on the first post and the article it linked to. I'm still not satisfied that a "self-deciding prediction market" is any more than a "vote." Instead of asking the question "Should Corporation X take this action?" You're asking "Do you think it is probable that Corporation X will take this action based on the information at your disposal?"

Since the people buying "shares" in the "market" can directly influence the eventual decision, I can direct the Corporation to make a particular decision by buying up all available shares in the prediction market, right?

Sounds like a shareholder vote to me. So what?
1.29.2008 5:11pm
Daniel Chapman (mail):
I should also point out that as long as you have the resources to control the market (IOW: You don't end up on the losing end of the vote), your "investment" will pay off. Go ahead and make that bet that it's 99.9% probable that the decision will be made... since the prediction market controls the outcome, it's guaranteed to be a good bet!
1.29.2008 5:13pm
byomtov (mail):
I don't understand what's on the y-axis. What's "the probability that the team in fact won the game"?

I think Nathan_M is correct. It's not really a probability, it's the percentage of games in fact won. So if you look at $7 on the x-axis and find it matches with 72%, say, then that means the $7 bet was actuarially fair, historically.

Of course, this is pretty weak evidence. Baseball games, indeed most high level sports, on average, are highly predictable. I suspect that if you followed a simple rule like: add the winning percentages of the two teams and pay $x for a contract on team X, where x is the ratio of X's percentage to the sum of the two percenatges you would get a very good graph.

Alternatively, you could just graph the Vegas odds. I bet they are at least as good as the prediction market. If they are significantly worse that would be evidence.
1.29.2008 5:15pm
byomtov (mail):
Almost actuarially fair, I should have said.
1.29.2008 5:16pm
Sasha Volokh (mail) (www):
I understand now, thanks!
1.29.2008 5:38pm
Michael Abramowicz (mail):
All: Yes, the label on the y-axis isn't ideal, but you all have figured it out.

Daniel: Assume the market forecasts what the corporation's stock price will be if a factory is built (contract 1) and is not built (contract 2). If you drive contract 1's price up (forecasting big stock price increase), you make it more likely that the factory will be built. If the factory is built but the stock price doesn't go up, you lose money. If other traders anticipate this, they will bet against you (as long as there is still any possibility that the factory will still be built). So, note, the market isn't just assessing the probability that the action will be taken, but also the effect of that action.
1.29.2008 5:46pm
Vish (mail):
Sports prediction markets are not a good example, because of the existence of Vegas odds information with huge amounts of money being spent on them. tradesports tends to hew closely to Vegas odds, probably because there are arbitrage opportunities otherwise.
1.29.2008 5:50pm
noto (mail):
Interestingly, all of the points in the upper-right lie above the 45-degree line. I have long had the suspicion that very strong favorites win more often than the prediction-market-implied-odds would suggest (and I've won some money betting on that belief). Would be interesting to try to test this theory more formally.
1.29.2008 6:01pm
Curt Fischer:

Of course, this is pretty weak evidence. Baseball games, indeed most high level sports, on average, are highly predictable.


I don't understand your criticism. Surely prediction markets, like just about any prediction scheme, will be more accurate when using easily predictable outcomes than when not.
1.29.2008 6:07pm
Daniel Chapman (mail):
Every prediction market I've ever seen has allowed people to "bet" on the probability that a particular event will happen... now we're talking about a prediction market that predicts stock price? So for example, we might have the following markets:

"Stock price of Corporation X to increase if Factory Y is built"
"Stock price of Corporation X to decrease if Factory Y is built"
"Stock price of Corporation X to increase if Factory Y is not built"
"Stock price of corporation X to decrease if Factory Y is not built"

Ok, I'll accept that these are valid prediction markets in theory, but how would Corporation X actually use the data? Choose whatever option the participants in the prediction market think will maximize stock price? Again... why not just have a shareholder vote?

Could we please have some examples of basing an actual decision on a prediction market that DOESN'T turn the prediction market into a self-fulfilling prophecy where people are simply rewarded for betting with the majority? Sports examples are not helpful. They show the accuracy of pooled information in a prediction market, but it's obvious that the result is not dependent on the market.
1.29.2008 6:09pm
Elliot123 (mail):
"Ok I read your response on the first post and the article it linked to. I'm still not satisfied that a "self-deciding prediction market" is any more than a "vote." Instead of asking the question "Should Corporation X take this action?" You're asking "Do you think it is probable that Corporation X will take this action based on the information at your disposal?"

A vote actually decides the outcome of the event in question. A prediction market lacks that power. Someone can buy as many contracts as he wants in a predictions market and it will have no effect on the actual event. Not so with a vote. The vote decides.
1.29.2008 6:11pm
Daniel Chapman (mail):
That's exactly what he's talking about, actually... This isn't just a glorification of the accuracy of prediction markets, he's talking about using them to make policy decisions.
1.29.2008 6:18pm
Brian K (mail):
noto,

i assume that results from people making contrary bets. even though the likelihood of payoff is small, the cost is small and the payoff is large. i've made some money overall using this strategy, albeit in stocks and not prediction markets. although i have no idea if this is a sound strategy in general or only works when you have strong suspicion that the market as overreacted.
1.29.2008 6:23pm
Elliot123 (mail):
"Could we please have some examples of basing an actual decision on a prediction market that DOESN'T turn the prediction market into a self-fulfilling prophecy where people are simply rewarded for betting with the majority?"

HiTax and LoTax are runing for president. I want no change in my cash flow. I buy contracts on HiTax. If HiTax wins, I pay more in tax, but that is covered by my win in the market. If LoTax wins, I lose in the market, but gain in a tax reduction. In both cases my cash flow remains the same.

I don't care about predictions, all I want is a fixed cash flow and I use the market to get that. Someone else may look at the market to use it as a prediction.
1.29.2008 6:24pm
Daniel Chapman (mail):
Why did you quote me there? The outcome of the prediction market in your example isn't even a decision, much less a decision made based on the predictions of the market.

Re-work your example so that the winner of the election is determined by the prediction market, and you'll see my point.
1.29.2008 6:27pm
byomtov (mail):
I don't understand your criticism. Surely prediction markets, like just about any prediction scheme, will be more accurate when using easily predictable outcomes than when not.

My criticism is that the prediction market adds little or nothing to our ability to predict the outcome of games. The idea of the market is that it aggregates a lot of information to improve our predictions. But in this case I think the market predictions are no better than existing ones. So why bother? And why claim accuracy when the prediction market's results just echo widely available forecasts and are likely, as Vish says, to behevily influenced by them.
1.29.2008 6:36pm
Nathan_M (mail):
Mr. Abramowicz's claims the baseball experiment suggests that prediction markets "ought to be good enough for government work. And most corporate work too." I don't think that is supported by the evidence he has presented.

All this data shows is that in this context prediction markets have a particular level of accuracy. That is interesting on its own, but the claim that prediction markets are useful seems to me to require showing that they are somehow more accurate than other methods of prediction, which this study can't possibly do.

Ultimately Mr. Abramowicz's is claiming that prediction markets are as good or better than other ways of predicting something (why else would they be good enough?), but this data provides no basis for that sort of comparison.

Other people have suggested in their comments that it would be interesting to compare the results of prediction markets with the Vegas odds. I agree that could provide clear evidence that prediction markets are good.

I would suggest another interesting experiment would be to look at the ex post accuracy of the Vegas odds before and after prediction markets were introduced. If either (1) prediction markets out performed the bookmakers in Vegas or, (2) the bookmakers did better once they had access to prediction markets, then that would be evidence prediction markets are useful.

(Parenthetically, I would add that in a way bookmakers already operate as a sort of prediction market, since they adjust their odds based on the bets people place, so even if prediction markets did not outperform them that would not be evidence prediction markets are never helpful.)
1.29.2008 6:51pm
Elliot123 (mail):
"Why did you quote me there?"

Because you asked for an example where the decision doesn't depend on self-fulfilling prophecy. My decision doesn't. I don't even care about the outcome.

"The outcome of the prediction market in your example isn't even a decision, much less a decision made based on the predictions of the market."

There really is no outcome to a market. There is an outcome to an election. A market has a changing state at any point in time. I agree the outcome isn't a decision. In my example, my decision is made long before the contract expiration.

"Re-work your example so that the winner of the election is determined by the prediction market, and you'll see my point."

The winner is not determined by the market. The winner is determined by the voters.
1.29.2008 6:52pm
Daniel Chapman (mail):
Wow... you misunderstand me every time you read me, it seems... I can't be THAT ambiguous...

First, when I asked the author of the post for a "decision based on the prediction market," I was looking for an example where the decision is, in fact, the thing that the market is supposed to be predicting. Like in the factory example we've been discussing, Corporation X is deciding whether to build the factory or not based on the market's predictions.

Second, when I said "The outcome of the decision market," I meant the outcome that the market is predicting. That's why I said the decision is based on the market's predictions, not the market's outcome. Sorry if I'm making up vocabulary as I go, but, well... I have to.

Finally, OBVIOUSLY the winner of the election is not determined by the market! THAT IS MY POINT! If you read the original posts at all, you'd understand that he is proposing that corporations or governments should use prediction markets as decision making tools! That's why I said the election example, like sports games, is unhelpful! If I actually have the power to make a decision, like building a factory or appointing a judge to the bench, how can I use a prediction market to make that decision without reducing the market to a mere vote?

I really can't be any more clear here...
1.29.2008 7:08pm
Guest 1L:
noto,

People are funny about high probability/low probability events, but I wouldn't get too excited about your hypothesis until you found out what the fee structure was. I believe generally, there is a more or less fixed contract fee. That would mean that if you're betting on a highly probable event, the ratio of possible gains to the fee would be high. This would naturally depress prices.

Also, note that the pattern isn't repeated toward the origin. Logically the probability/price stuff going on there is the same as for the upper right except short is long and long is short. If the market worked perfectly, Price = Probability * ($10 - Fee). That would explain an appreciable distance from 45 degrees in the upper right and the lack of an appreciable difference toward the origin.
1.29.2008 7:17pm
Guest 1L:
I meant, of course, that the ratio of probable gains to fee would be low with high probability events.
1.29.2008 7:21pm
Elliot123 (mail):
There are thousands of users of futures markets, and they use the information from the market for all kinds of things. What you refer to as the outcome would be the settlement price at expiration. Since almost all traders have finished using the market prior to that time, it attracts little attention. I don't think there is any existing market that is used for a single decision.

When a market is established, nobody can determine what it will be used for. The usefulness will be determined by the volume of trade and open interest. Maybe someone will use the factory market to determine if they should build, but it will not survive unless it performs many more functions which will attract traders.

In the factory example, a fast food operator may use the info to determine if he should acquire a lot near the proposed plant. He may think that the company wants a high stck price, so he figures a high prediction market price will induce a build decision. He buys the lot, but sells the factory contract to mitigate some of his risk.

It's instructive to look at open interest in expiring futures contracts. Only a very, very, very small amount of the total contract life trading volume is ever delivered. The outcome is an afterthought.
1.29.2008 7:37pm
Daniel Chapman (mail):
You're talking about real life. I get it. I know how intrade works, so please stop trying to explain it to me.

We're talking about something completely different over here, though... so you're not responding to my questions. What if, in some HYPOTHETICAL UNIVERSE, a corporation actually decided to build a factory based on the predictions of a prediction market? Would it be a good thing? A bad thing? A paradox?

"Corporation X to build Factory Y" stabilizes at 80%... Based on this "market," Corporation X actually DECIDES to build Factory Y! Get it?

(Or, as Mr. Abramowicz suggests, "Stock Value of Corporation X to increase if Factory Y is built")
1.29.2008 7:46pm
Elliot123 (mail):
Of couse I'm talking about the real world, not InTrade, but the futures markets in Chicago, New York, London, etc. It seems our real world experience should be a good guide in discussing new contracts.

I have no idea if the decision to build would be a good thing. Likewise, I have no idea if a decision by Sara Lee to buy 1,000 contracts of December wheat is a good thing, bad thing, or paradox. The info from the prediction market would be one of many factors. If such contracts were traded, their success wuld probably hinge on whether the companies actually followed their indicators. If not, traders would have no reason to use them.

I actually don't know how a contract could settle if it were, "Stock price increases IF factory built." If the factory was not built, the contract couldn't settle. I couldn't find the original on it.

However, one could have a pretty good contract on, "Apple sells 2 million iPods in December." That might turn a decsion to build a new plant.
1.29.2008 8:34pm
Daniel Chapman (mail):
It might, but it's not what we're talking about.

"Daniel: Assume the market forecasts what the corporation's stock price will be if a factory is built (contract 1) and is not built (contract 2). If you drive contract 1's price up (forecasting big stock price increase), you make it more likely that the factory will be built."

This discussion has context, you know... I'm not the one making the rules.
1.29.2008 8:48pm
Daniel Chapman (mail):
Whatever... you know what? I started posting this with a genuine question about the poster's theory, and I got sidetracked in some sort of non-argument with another commenter. Sorry for wasting everyone's bandwidth.

I'm just going to forget about it, and stick with my theory that this guy's theory puts the cart before the horse.
1.29.2008 8:55pm
cathyf:
We're talking about something completely different over here, though... so you're not responding to my questions. What if, in some HYPOTHETICAL UNIVERSE, a corporation actually decided to build a factory based on the predictions of a prediction market? Would it be a good thing? A bad thing? A paradox?

"Corporation X to build Factory Y" stabilizes at 80%... Based on this "market," Corporation X actually DECIDES to build Factory Y! Get it?
I'm interested in the effect of insider information.

Suppose the company observes that the prediction market puts an 80% probability on the building of the factory. So the company buys contracts up the wazoo, until it drives the price up to 99%. Then it stops buying, and announces its decision to build the factory. The factory which it partially finances from the trading profits it earned from "betting" in the prediction markets.

Notice that the lower the probability (price) in the prediction market, the higher the trading profits earned, so the lower the net cost of the factory. So the lower the prediction market's "probability", the higher the real probability of the factory being built...

Or, as I said over in the dead thread... Suppose I run for president. Then I decide to pull out. I sell my contracts up the wazoo, then announce my decision. And take my handsome trading profits. All I need to do to get my profits is to get my contracts up high enough above zero to cover the transaction costs.

Real markets have rules against insider trading, kerb trading, athletes betting on their games. (Those rules take various forms -- securities laws, exchange rules, and mob guys who will come break your legs if you cheat them.) Because only idiots show up to be fleeced by people who have an information advantage. What structures do the prediction markets have?
1.29.2008 9:02pm
Wayne Jarvis:
D.C. Why do you assume that the prediction market must predict ultimate question to be decided?

If that were the case, then you would be correct that the prediction market would be a self-fulfilling prophesy. But that would just mean that the particular market was poorly designed. In practice nobody would base its decision on a poorly designed market like that.

Example: Will Old McDonald plant corn or soybeans? The market predicts soy beans so that is what Old McDonald does. Not very useful.

But if the market were to predict whether is will be a wet or dry growing season (and for the sake of this hypothetical a wet season favors soybeans) that would be very useful market. Prof. Abramowicz's example is also a great example. (BTW, "why not just have shareholders vote" is not really an argument that the prediction market wouldn't work).
1.29.2008 9:22pm
Daniel Chapman (mail):
Thanks for answering my question. Is that what you were talking about, Mr. Abramowicz?
1.29.2008 9:27pm
Mr. Liberal:

Nonetheless, they ought to be good enough for government work. And most corporate work too.


Umm... not really.

I think you need empirical evidence in the political context.

My problem is that baseball is too much like rolling dice. The events are predictable. A particular pitcher is likely to throw a ball in a predictable manner, a particular batter is likely to respond well to that manner well or not in a systematic way. I think it is probably much easier to develop mathematical models to predict a baseball game than an election. Such models could be used by the more sophisticated traders, thus making the prediction market rather accurate.

Elections are another matter. You need empirical evidence dealing with that. Then, you need to somehow compare it to the alternative, (i.e. the conventional wisdom) because most of the time, prediction markets and the conventional wisdom are likely to be in sync.

The interesting question is this. When prediction markets vary from the conventional wisdom about elections, who is usually right? In what circumstances?

I think prediction markets are a fascinating area for research. But, I think there is too much hype.
1.29.2008 9:32pm
Mr. Liberal:
One more comment. The phrase "good enough for government work" ought to banned. Government work should be held to high standards.

This has nothing to do with prediction markets, I am just saying. =)
1.29.2008 9:34pm
Elliot123 (mail):
"Example: Will Old McDonald plant corn or soybeans? The market predicts soy beans so that is what Old McDonald does. Not very useful."

Old McDonald is a canny old guy, and has been planting beans and corn for a long time. If beans are high enough, he is going to short the beans, and lock in a price. The market is doing more than predicting; it is contracting with him for a fixed quantity of beans at a fixed price. It's very useful for him.

Suppose he doesn't short the beans, and thinks they will continue to increase in price? Then he can manage risk of yield per acre by trading on the weather futures market for the wet or dry season.
1.29.2008 10:05pm
Elliot123 (mail):
One of the things to consider in testing the accuracy of a predictive market is that it is a continuous curve over time. Each day, there is a different set of prices and predictions. So, for a presidential election, there are about 90 days of predictions, from convention to election.

HiTax wins the election. If a market had HiTax at 70% for 80 days, then he drops to 20% for the last 10, did that market correctly predict? If it is at 30% for 80 days, then goes to 70% for the last 10 days, did it correctly predict?
1.29.2008 10:19pm
Michael Abramowicz (mail):
Daniel -- I'd rather predict the actual stock price level or change than "increase" or "decrease." Then it would be straightforward for the corporation to use the data. And it's not self-fulfilling prophecy because the market is predicting the effect of the decision, not just whether the decision will occur. But I think we're close to being on the same page now.

Nathan -- I agree we need more comparative analysis of prediction markets vs. other approaches, but it's not so easy to do this. And showing prediction markets are better than Vegas odds would still leave us vulnerable to the criticism that Vegas odds aren't good enough.

Elliot123 -- Several more good points, but I do think you haven't factored into some of your explanations (e.g., the market "will not survive unless it performs many more functions which will attract traders") that subsidies can themselves attract traders.

Mr. Liberal: Your interesting question is who is right when "prediction markets vary from the conventional wisdom." Of course, we have to measure the conventional wisdom, and our study will only compare prediction markets to that form of measurement. But more studies like this would be useful.

Elliot123: The election markets are most accurate right before the election, but at least some data suggests they beat polls also well before the election. The key, of course, is to look at lots of data over time.
1.30.2008 2:22pm
Elliot123 (mail):
"Elliot123 -- Several more good points, but I do think you haven't factored into some of your explanations (e.g., the market "will not survive unless it performs many more functions which will attract traders") that subsidies can themselves attract traders."

Sure, subsidies are certainly an attraction. But I think they need other factors as a catalyst. It seems a bit like the situation with scalpers in a futures market. The opportunity to scalp profits will attract them, but that opportunity depends on the order flow to work. On a day with poor orderflow, the scalpers just stand around scratching and betting on basketball.

The successful market has a number of mutually reinforcing phenomena at work. A subsidy would certainly be one. It's a bit like the payment NASDAQ gives to firms for placing orders. They do encourage trade, and some people make the majority of their profit from those payments.
1.30.2008 4:57pm