A quick post script to my previous post on David Wessel's economics column in today's WSJ concerning the stimulus. Wessel quotes Barney Frank in passing:
Not for the first time, as an elected official, I envy economists. Economists have available to them, in an analytical approach, the counterfactual. Economists can explain that a given decision was the best one that could be made, because they can show what would have happened in the counterfactual situation. They can contrast what happened to what would have happened. No one has ever gotten reelected where the bumper sticker said, "It would have been worse without me." You probably can get tenure with that. But you can't win office.
I didn't include this in my discussion, as I found it a bit tangential. However, when I saw that Professor Mankiw had linked to it, I decided to revisit it.
Leave aside as too ad hominem whether Congressman Frank's own statements regarding his role in Fannie and Freddie have any element of counterfactual. I understand that what some people, certainly in his Congressional district, find clever, I find too clever by half, but okay - that is really not my interest here, and please, not in the comments.
Is it really true, as a general proposition, that politicians have no recourse to counterfactuals? What about, for example, the famous claim of "saved or created" four million jobs? As accounts of the President's February 9, 2009 press conference had it:
Question: The American people have seen hundreds of billions of dollars spent already, and still the economy continues to free-fall. Beyond avoiding the national catastrophe that you've warned about, once all the legs of your stool are in place, how can the American people gauge whether or not your programs are working? Can they — should they be looking at the metric of the stock market, home foreclosures, unemployment? What metric should they use? When? And how will they know if it's working, or whether or not we need to go to a plan B?
Answer: I think my initial measure of success is creating or saving 4 million jobs. That's bottom line No. 1, because if people are working, then they've got enough confidence to make purchases, to make investments. Businesses start seeing that consumers are out there with a little more confidence, and they start making investments, which means they start hiring workers. So step No. 1, job creation.
Jobs "created" is a factual. Jobs "saved," on the other hand, poses a counterfactual. Professor Mankiw has an excellent quick discussion of why:
The expression "create or save," which has been used regularly by the President and his economic team, is an act of political genius. You can measure how many jobs are created between two points in time. But there is no way to measure how many jobs are saved. Even if things get much, much worse, the President can say that there would have been 4 million fewer jobs without the stimulus.
An actual answer to the question "What metric?" could have taken the form: "If the unemployment rate on [insert date] is below [insert threshold], I will judge the plan to be a success." Given the uncertainties inherent in the economy, however, no sensible politician would hold himself to such a measurable standard. But the President also wanted to avoid sounding like he was avoiding accountability. So he gave us a non-measurable metric. A clear and specific benchmark, without any way of ever knowing whether it has been reached.
A completely honest (but perhaps politically ill-advised) response to the question would have been, "Geez. I am only President of the United States. I cannot be held responsible for everything that what happens with the economy!" If he had said that, I would have agreed with him.
---- Update: A regular reader of this blog (who deserves anonymity) misinterpreted my meaning, so let me clarify: The 4 million job number is a counterfactual policy simulation of what the stimulus will do based on a particular model of the economy. As such, I have no objection to someone citing it in a policy discussion. In fact, macroeconomists use models to generate figures like this all the time. I have even done it myself.
But as an answer to the question "how can the American people gauge whether or not your programs are working?... What metric should they use?", citing the 4 million job figure is a non sequitur, or more likely a diversion. A metric has to be measurable, and the actual number of jobs "created or saved" by the policy will never be measurable from any data source.
The more I think about it, the more I think, contra Frank, that politicians of all ideological stripes get elected on counterfactuals all the time. Am I right about that - not about Barney Frank, please - or not? Other examples of politicians offering counterfactuals? And are counterfactual arguments always bad, while we're taking up the abstract category? When and how, or not?
Update: While I am at it, let me ask what the abstract nature of counterfactual argument is. What makes a counterfactual argument a counterfactual? I usually think of them as "but for" arguments - but maybe that is too quick. What is the abstract structure of a counterfactual? And what, in that case, makes for a good as opposed to bad counterfactual argument? Finally, does this bear any relationship to argument from analogy, and if so, how?
Dan Schmutter
Isn't there still a counterfactual when you discuss how many jobs were "created"? Let's say that by 2012 the economy has 4 million more jobs than when Obama took office. Sure, we can "measure" that there are more jobs, but we can't really measure how many of those jobs wouldn't be there had Obama done nothing, or had Obama done what his opponent suggested.
Whether you're estimating the number of jobs "saved" or the number of jobs "created," you're still speculating as to the number of jobs influenced by the President's actions.
Let me just be clear about the general issue - it's not that I'm opposed to counterfactual arguments. On the contrary, we can't do real policy without them. But I don't think it's the case that politicians cannot get elected making them, whether defensible counterfactual propositions or not.
... and its corollary, "fighting them over there so we won't have to fight them here."
Whenever good things happen (like jobs being created), supporters of the President tend to argue that they happened because of the President's policies, and opponents tend to argue that they would have happened anyway. It's probably not coincidence that the arguments tend to play out this way. But even if ten million jobs came into existence between now and next year, no one would have to concede that they were "created" by the stimulus; they could still argue that the stimulus was completely ineffective. So I don't see "jobs created" as any more of a hard fact than "jobs saved."
The first part represents the "created" portion of the Obama plan while the second represents the "saved" portion. Only when jobs are created AND saved (or created in excess of those not-saved) is actual job growth, and therefore perceived economic recovery, realized. The big deception of Obama's statement is the word "or".
The question of "How many jobs are created" is just as factually untenable as "how many jobs are saved," at least when your sole metric is unemployment rate.
We can measure the number of jobs saved by speaking directly to employers and asking them how many jobs they planned on shedding in the '10 fiscal year before the stimulus package. Then compare that to the number of actual jobs each employer shed in the '10 fiscal year. It represents a reasonable measure of the number of jobs "created or saved" we can attribute to the stimulus package.
Therefore, the rationale for "jobs created" or "jobs saved" is directly related to actual, measurable experiences.
That said, the fact that unemployment exceeded the administration's expectations should rationally lead to the conclusion that rather than "creating or saving" jobs, the stimulus has actually reduced the number of jobs.
Long story short, I disagree with the idea that jobs "created or saved" cannot be measured. Comparing projected unemployment with actual unemployment gives an excellent metric by which to measure "created or saved" jobs.
It just turns out that the Obama administration has failed to do either.
1) ID a problem
2) gather information
3) propose solutions
4) compare and contrast proposed solutions
5) make a selection.
This process is used in science, engineering, consulting, and military decisionmaking.
Counterfactuals are all the unselected alternatives.
Good counterfactuals are plausible unselected alternatives.
Bad counterfactuals are implausible unselected alternatives (in the speak of the above process, perhaps a proposed solution that was determined to be unrealistic, or a solution that wasn't even considered-and thus unstudied).
the problem with the political process is it doesn't follow the scientific decisionmaking process*, and thus multiple solutions aren't studied, or its a complicated enough problems that the the consequences of solutions are impossible to predict. Thus, counterfactuals are alternative solutions to a problem that hasn't been adequately defined, or solutions that haven't been compared or contrasted, or for which information wasn't gathered. thus, it becomes an information-poor 'sense' of whether a counterfactual (an alternative solution) is reasonable or not.
Note that the above formalized decisionmaking process, I bet, was created by some psychologist in the 60's (perhaps to improve business decisions, perhaps to define the scientific method, I don't know). I have encountered it in the military (the Military Decisionmaking Process, or MDMP), as well as in engineering and governmental policy. If you research it, you will probably find one source for it sometime within the last 40 years.
Sk
*I'm not saying the political process should be a scientific process. I'm saying the problem with identifying plausible conterfactuals in the political process is due tot he fact that its not a scientific process.
The counterfactual in those cases is often relatively easy to support.
"If we hadn't expanded the factory we would not have been able to meet the surprisingly strong demand for our new widgets."
"If we hadn't introduced New Coke we would have saved [some huge amount] and avoided a major embarrassment."
The problem is that while economics does its best to predict things, on the macro level you can't really run the same experiment twice the way you could in the laboratory.
I agree, but only if the voters are afraid the counterfactual may still happen, e.g., a terrorist attack. The economic apocalypse we avoided isn't a pressing concern any more, making Obama a victim of his success. Our pre-occupation with economic matters is more like what happens in all cyclical recessions, albeit a very bad one. And that raises a second factor: how clear is the contrast between current conditions and what could have been? For Bush, the contrast between 9/11 and no new 9/11's was clear, even if part of his "cure," the Iraq war, was worse than the disease. For Obama, the contrast is murkier for a couple of reasons: first, we don't have any fresh memory of an economic collapse comparable to what we just avoided; second, for people who lost their jobs, businesses, houses and other investments, the recession we got may seem indistinguishable from the depression we skirted. It would be hard for many of them to believe they're better off than they could have been. It's like a doctor telling an accident victim he saved his life but had to amputate the legs. The patient may appreciate what the doctor did for him someday, but probably not in this election cycle.
This is because a great deal of the job savings occurred as the result of states forgoing budget cuts that, had they gone through, would have resulted in state gov't workers losing their jobs. (Of course, many conservatives don't consider state jobs actual "jobs," or, alternatively, value them highly, but that's another story.)
We can figure out how many jobs were saved by (1) examining state budget proposals brought up before the stimulus passed and the number of cuts they called for, or (2) in the absence of specific budget proposals, estimating how many state jobs would have been lost had states passed non-stimulus-supported budgets. The latter may not be that difficult, since most states have balanced budget amendments.
I realize that this would not be perfect, but the causal link between avoiding gov't layoffs and stimulus money seems stronger than the causal link between the stimulus and the creation of private sector jobs.
It's not. It's not. It's impossible to credibly estimate the number of "jobs saved" when your butt is on the line based on that number. So if I go to my boss and say, "look, the department is operating 10% more efficiently since you hired me, so I saved you alot of money", he might or might not believe that the 10% is because of me, but if I go to him and say, "well, I've estimated that the department would have operated 10% less efficent if you hadn't hired me, but we're operating at the same level, so I saved you money" he's going to think I'm a weasel.
When President Obama claims to "saved or created" X number of jobs, he does so in calm confidence that there is no parallel US, identical except that his policies are not applied, against which we could compare our employment figures -- his statement is therefore not falsifiable. It may be true, it may be false, we'll never know.
By contrast, someone who claims that if a country "stays out of the Middle East", it would be safe from terrorism, apparently believes that either no country has ever stayed out of the Middle East or that every country that has done so has been spared terrorism. (Spain? Germany? Indonesia? Sri Lanka? India?)
What the government can say is two fold. First, and maybe less helpful, is that there haven't been any major terrorists attacks since 9/11/01. But that may be just because the terrorists shot their wad then.
But they can also say that Y attacks were thwarted, and likely also that Z policy allowed them to thwart them. How? Let us take the TSP (Terrorist Surveillance Program), where international calls were sometimes tapped w/o a warrant. They may be able to say, well, we got 100 suspicious calls, investigated them, found 20 suspicious people engaging in terrorist activities, and arrested them. Or maybe they caught an arms shipment as a result. That sort of thing. Identifiable instances of the policy resulting in a terrorist attack being foiled.
But then, coming back to the first situation, about there not being any major terrorist attacks since 9/11/01, when combined with thwarting Y attacks using Z policy, maybe one can say that the lack of major terrorist attacks was at least partially due to using Z policy.
Another key difference here though is that since there have been no major terrorist attacks since 9/11/1, there is no countrafactual. There is no possible examples of other attacks succeeding because of the policy, while others were thwarted.
But I've been down this path before, when I rather gently chastised Ezra Klein on this blog for talking about the 'incredible lightness' of Greg Mankiw, and took untold grief for it in the blogosphere for challenging the purveyor of Journolist.
If you have a job, hang on to it.
I don’t care about Mankiw, there are many like him. What irritates me on this site and I guess will continue to irritate me is that supposed libertarians support the federal reserve. There is nothing about the federal reserve in its creation, track record, or current machinations that deserves kudos from a libertarian. At best you are stuck with the discordant "Ignoring what got us into this mess(the fed), THEY are best suited to leading us out by doing what they did before. But MORE OF IT!!!" Whatever.
Change we can believe in!
Donald Rubin, "Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studes", J Educ Psych 1974
This is a non-mathematical discussion of the idea of counterfactuals and experimentally or observationally determining causal effects from counterfactuals.
Paul Holland, "Statistics and Causal Inference", J Am Stat Assoc 1986
A little more statistical, with some elementary probability theory built in, but also more wide coverage of various approaches to understanding counterfactuals and causality.
If you have a comment about spelling, typos, or format errors, please e-mail the poster directly rather than posting a comment.
Comment Policy: We reserve the right to edit or delete comments, and in extreme cases to ban commenters, at our discretion. Comments must be relevant and civil (and, especially, free of name-calling). We think of comment threads like dinner parties at our homes. If you make the party unpleasant for us or for others, we'd rather you went elsewhere. We're happy to see a wide range of viewpoints, but we want all of them to be expressed as politely as possible.
We realize that such a comment policy can never be evenly enforced, because we can't possibly monitor every comment equally well. Hundreds of comments are posted every day here, and we don't read them all. Those we read, we read with different degrees of attention, and in different moods. We try to be fair, but we make no promises.
And remember, it's a big Internet. If you think we were mistaken in removing your post (or, in extreme cases, in removing you) -- or if you prefer a more free-for-all approach -- there are surely plenty of ways you can still get your views out.