In 2006, the Lancet published a controversial study finding a substantial, continuing Iraqi death toll in the years following the 2003 U.S. invasion. The study bolstered critics of the Iraq war and prompted substantial debate, online and elsewhere.
Neil Munro revisits the Lancet study in the new issue of National Journal.
In the ensuing year, numerous skeptics have identified various weaknesses with the study's methodology and conclusions. Political blogs and academic journals have registered and responded to the objections in a debate that has been simultaneously arcane and predictable. The arguments are arcane because that is the nature of statistical analysis. They are predictable because that is the nature of today's polarized political discourse, with liberals defending the Lancet study and conservatives contesting it.I did not follow the debate closely enough to reach a conclusion about the merits of the study or its critics. The Munro article provides a convenient overviewof the controversy for those of us without the time or patience to wade into the depths of the debate. Munro is not entirely neutral, however, as he concludes there are potential problems with the initial study.How to explain the enormous discrepancy between The Lancet's estimation of Iraqi war deaths and those from studies that used other methodologies? For starters, the authors of the Lancet study followed a model that ensured that even minor components of the data, when extrapolated over the whole population, would yield huge differences in the death toll. Skeptical commentators have highlighted questionable assumptions, implausible data, and ideological leanings among the authors, Gilbert Burnham, Riyadh Lafta, and Les Roberts.
Over the past several months, National Journal has examined the 2006 Lancet article, and another [PDF] that some of the same authors published in 2004; probed the problems of estimating wartime mortality rates; and interviewed the authors and their critics. NJ has identified potential problems with the research that fall under three broad headings: 1) possible flaws in the design and execution of the study; 2) a lack of transparency in the data, which has raised suspicions of fraud; and 3) political preferences held by the authors and the funders, which include George Soros's Open Society Institute.Of these critiques, I find the political preferences of the authors and their funders to be the least persuasive. Political bias of this sort could certainly explain problems with the study, such as a failure to scrutinize sources and ensure their reliability, but I do not think that the authors' ideological predispositions (or those of the funders) should, in and of themselves, case doubts on the study's findings. The Lancet study's conclusions should stand or fall on their own. In this regard, it is interesting that Munro reports the Lancet editors are less confident of the analysis than they once had been.
Today, the journal's editor tacitly concedes discomfort with the Iraqi death estimates. "Anything [the authors] can do to strengthen the credibility of the Lancet paper," Horton told NJ, "would be very welcome." If clear evidence of misconduct is presented to The Lancet, "we would be happy to go ask the authors and the institution for an official inquiry, and we would then abide by the conclusion of that inquiry."
More than the number killed in the American Civil War on BOTH SIDES?
In fact, that was done. The comparison was of mortality rates before the war and after, IIRC.
The apparent lack of data availability troubles me. While it's not uncommon for those who have collected so much data to decline to share them promptly, given the public prominence of this article, I think they should have released them. I of course doubt they made up the numbers, but the failure to release somewhat limits the ability of others to screen for methodological problems.
The problem with the "ratio of officially counted deaths" is that the Lancet study claimed to have actual death certificates for well over 90% of the deaths reported in the study.
Which would also suggest (if their methodology was anywhere near correct) that the Iraqi government should have records of almost ALL of those 600,000 to 1,000,000 deaths. Which they don't, by any stretch...
Good discussions about the statistics can be found at Deltoid, where it is clearly established that David Kane doesn't know anything about statistics. (see especially the comments of Robert, a well known statistician)
And when figures were postulated a year or so ago, leftists in Salon, ThinkProgress, The Age, The Independent, etc. etc (just google "army enemy body counts"), they raised Cain again.
Sounds like the logical fallacy of Appeal to Incredulity.
It shouldn't take the magic of statistics to notice something like this.
Well, it's a good discussion if you enjoy watching experts, or people claiming to be experts, berate each other. Also, 'well known statisticians' generally have last names.
Yes, body counts of civilians would be so embarrassing.
http://mpa.sciences-po.fr/sub500/sub50106.html
In other words, I would assume he has some idea of what he's talking about when he discusses how to estimate the death toll of the Iraq war.
Global warming is a "science". Evolution is "science". In other words they are both garbage and the mad rantings of the liberal "enlightenment" mind.
Science is dead, and this "study" is just another nail in its coffin.
http://scienceblogs.com/deltoid/2007/08/
robert_chung_on_david_kane.php
As Robert Chung points out in the thread, this doesn't mean the conclusions of the Lancet study are certainly correct. But many of its critics seem to have violated basic standards of good faith in academic argument.
If he so vehemently defends a study which has such deeply flawed statistics, should I really care what he has to say about yet another controversial statistical study? OK, I'm sure that's some kind of logical fallacy that someone will point out, but frankly I don't want to waste my time with him and I'm surprised anybody else does.
I can cite other examples where he's let his political leanings get in the way of facts but that one is so devastating I'll leave it there.
They certainly have some bearing.
For example, if the authors strongly supported the Iraq war I would tend to be skeptical of claims that they overcounted deaths whereas since they did, in fact, oppose it I would be skeptical of any claim that they undercounted.
But I think they need to release everything. And it concerns me a little that they won't.
C'mon now. Without having seen the results, there is ample evidence of bias. The financiers, authors and the magazine editors are all outspoken and often virulent anti-American and anti-war advocates. Both "studies" were timed specifically for release before the 2004 and 2006 US elections. The peer review of the first "study" was sacrificed to ensure that it would be released before the 2004 election.
Then there is this: Mr. Roberts tries to go to Washington. Roberts, who opposed removing Saddam from power, is the most politically outspoken of the authors. He initiated the first Lancet study and repeatedly used its conclusions to criticize Bush. "I consider myself an advocate," Roberts told an interviewer in early 2007. "When you start working documenting events in war, the public health response -- the most important public health response -- is ending the war."
This entire episode is an all too familiar academic disgrace.
As a trial attorney defending against personal injury and construction defect cases, I have discredited dozens of so called scientific or medical experts who prostitute science for ideology or money. I have an extremely cynical view of the science on nearly any politically controversial topic. From alar to global warning, the list of garbage science produced for ideology or money is long and ignominious.
The Lancet studies are one of the most blatant examples of politicized science.
In response to strong criticism regarding his methodology, e.g. 369 LANCET 101 (2007), Roberts promised to provide the data and descriptions of the methodology to other researchers, see, e.g., 446 NATURE 6 (2007). However, Roberts is only providing the information to "certain" researchers, and has refused to release it to well-qualified critics, 316 SCIENCE 355 (2007). One of the most damning critiques of the Roberts study is that it is hard to critique in the absence of data and methodology. For those of you defending it, on what basis do you have for saying that it "looks good"? In science, the burden is on those seeking to defend a conjecture, not those seeking to refute it.
The results are their basis...
For the record, I don't claim the study looks flawless. The number of data clusters, as Munro points out, might be too low. The problematic conditions in Iraq, which the study blames for massive numbers of deaths, unfortunately also make reliable data collection about mortality rates very difficult, and leave the door open to sampling bias or fraud. The official mortality figures, though, are subject to the same sorts of concerns. Furthermore, the basic conclusion of the Lancet study, namely that the real number of deaths in Iraq since the U. S. invasion began are dramatically understated by the official figures, has been corroborated by other studies. Thus, while the jury may still be out on the Lancet estimate, I don't think it's time to convict of fatal bias just yet.
Finally, we shouldn't forget that even a conservative estimate of the number of people to die as a direct result of the U. S. invasion is still around 100,000 people. Awfully large number of people to kill over some bad stovepiped intelligence, wouldn't you say?
In other words, the 650,000 figure was the upper end of a range spanning six orders of magnitude?
If so, I'd argue that all the study can conclude is that they don't really have any clue what the real figure might be with any reasonable level of confidence.
If you use the more common definition of order of magnitude, a ratio of ten, it spans about two orders of magnitude. Still an awful lot, considering that they're only 95% confident that the real value is between those ranges.
Sure, like Life magazine's readership claims looked good.
They're claiming that each sampled item was a stand-in for 2K events.
Too fine, even if the sampling was carefully conducted, which it wasn't.
Given that I was able to hunt down that information in about 30 seconds online, I have to wonder about Nicholas' motivations in criticizing the Lancet study as much as I do Kane's. Of course, Nicholas earlier referred to Steve McIntyre, whom a brief trip to Wikipedia revealed as a long-time mining company employee turned professional global warming skeptic with just a bachelor's in pure math and an Oxford masters in PPE, as an "expert statistician" should probably have already made wonder the same thing.
For the record, I'm a conservative Republican who initially supported the Iraq War. But unlike far too many fellow members of my party, I've never believed in burying my head under the sand when it becomes clear I've made a mistake, whether on foreign policy or the environment.
The reason that methodology and/or data isn't disclosed is irrelevant; that it isn't disclosed effectively excludes whatever is claimed from the process of science. Consider the following:
INVENTOR: I've done it! A perpetual motion machine! Look at it go! This meter says the energy in equals the energy out.
SCIENTIST: Common sense tells me this is fishy. Let me see how you are measuring energy, and let's look at your raw data.
INVENTOR: Sorry, can't do it. The meter is a trade secret, and I promised the guy who made it that I wouldn't tell anybody how it works or exactly what type of energy it's measuring.
Doesn't sound much like science to me. As for your particular rebuttal of my criticism, forgive me if I defer to four unanswered pages of critiques in one of the world's premier scientific journals, as well as a refusal to provide the relevant data to a London School of Economics professor under a confidentiality agreement, instead of a blog by "a computer scientist at the University of New South Wales." Again, I assert that the burden is on those making the conjecture, not those seeking to refute it.
On the other hand, it's common for personal information to remain unavailable in epidemiological studies. In this case, much of the data requested by the LSE researcher was destroyed soon after collection to ensure anonymity, given the consequences a violation of that could have in Iraq. The criticisms in Science were not unanswered; Burnham and Roberts sent a letter to the editors in response. The discussion in Deltoid was not carried out primarily not by Tim Lambert, but by the demographer Robert Chung. The Lancet paper was the subject for a plenary session at the 2007 meeting of the American Statistical Association, and received cautious approval from statisticians J. Asher and F. Scheur. None of this looks to me like evidence of a paper that's a scientific disgrace.
Of course, the only way to ultimately determine whether a study is reliable is to compare it with carried out by other researchers, e.g. the QRB study. By that measure, the Lancet study may underestimate the number of deaths.
I disagree.
The Soros-linked group funding was not mentioned in the original study. Soros et al have public policy agendas which they promote.
Undisclosed support by a drug company would cast a real shadow on a drug study. Why shouldn't undisclosed support by a policy group cast doubt on a study that reinforces the group's position?
Economic interest is not the only source of bias.
I disagree.
The Soros-linked group funding was not mentioned in the original study. Soros et al have public policy agendas which they promote.
Undisclosed support by a drug company would cast a real shadow on a drug study. Why shouldn't undisclosed support by a policy group cast doubt on a study that reinforces the group's position?
Economic interest is not the only source of bias.
You know, for all the awfulness of Saddam's regime (which I don't minimize), he didn't really kill/torture/maim all that many people. It seems that feeding his son-in-law into a wood chipper and executing entire towns every once in a while really obviates the need for any more violence. It may be, in fact, this awfulness (again, not to minimize it) that actually prevented the sort of free-for-all we have to day.
statfan wrote:
Please keep in mind that it's not people killed. It's excess deaths. That's things like kids who couldn't get medical treatment because the hospital had been blown up, or the doctor had fled, or the medicine wasn't available.
National Journal article:
If I recall correctly, Lafta's reports were the source of the widely reported (and by some, widely accepted) claim that the pre-war sanctions on Iraq were causing the deaths of "hundreds of thousands" of children.
I really don't see how one can reconcile Oren's assertion of relatively low pre-invasion mortality with the supposed methodology of the Lancet studies or with the supposed studies of pre-war mortality due to the sanctions.
For that matter, putting Lafta in charge of gathering and reporting all of the data for the study strikes me as being rather like putting Baghdad Bob in charge of CNN.
On the second link, I found:
David Kane asks:
To which 'Robert' replies:
I read this as a cute answer that avoids the question and leaves Robert's identity unknown. I'd also point out that Robert's debating style is remarkably adolescent for an established professional.
And you believe Wikipedia? Seriously?
For a start, the term "global warming skeptic" is not even close to the mark. He's also not professional - he is retired. If you have read his work, you will know that he is no slouch at statistical analysis either.
I wonder if we read the same study. Was there more than one? I clearly remember the wider confidence interval.
That, all by itself, should make one skeptical.
I recall that NASA, after being vigorously shaken by one or two amateur experts, admitted to having made some errors in their math about global warming.
And their biggest mouth, who spent about a quarter of his hundreds of interviews claiming he was being silenced, got a huge amount of money from Soros.
Seems to me that if Bush, Rice, Cheney, and whomever else you don't like is supposedly on the Saudi payroll, or following their own path to undeserved riches, then it follows that money must indicate dishonesty. So...? The Lancet study got money, Hansen got money. If it means something in one place, proving it doesn't in another place ought to be required.
Everything is politicized. Extraordinary convenience ought to be a huge red flag.
Did the Lancet article say that the bulk of the casualties were due to US air strikes? If so, then comparing tonnages dropped in Iraq in WW II and Viet Nam is a legitimate question.
Excess deaths includes, as has been said, those due to reduced access to medical care, not simply violent deaths. But those people would be the most fragile, who ought to have been winnowed out by the evil sanctions. What was left of the population would have been like the Spartans.
Something's amiss here. And the disproportion of those dying being men of military age--the least fragile--indicates combat,not old age, heart attacks, and untended athlete's foot. I do not joke about the latter. In nasty, dirty environments, such problems can lead to infections and sepsis.
If you dig through many of the other posts that Lambert has (see first link) on the Lancet surveys, you will find Iraq Body Count represented by a joshd. It's pretty clear who he is also.
BTW if you want a more recent survey, try this one, which puts the death toll from the war over 1.1 million (this is NOT excess deaths such as for the Lancet surveys)
No wonder the gravediggers are so bummed. The war's practically over and the million plus they had as a bonus on top of the ordinary business isn't there any longer.
This is a snarky way of saying there ought to be a million plus graves someplace in addition to the usual. Someplace.
You'll recall the national panic about the epidemic of kidnapped children a decade or so back. Seems there were thousands and thousands and thousands. Some folks, like me, asked where were either the remains, or the teenagers escaping from durance vile with their stories. Should have been stacks of one or the other.
Turns out to be about a hundred stranger abductions a year. The activists had gotten a good portion of society panicked by pointing to custody issues as "kidnapping".
The lesson is that activists will make up whatever numbers they need.
And if you agree with the activists' position, the fact that their latest "STUDY" is a slam-dunk confirmation of your/their position ought to make you very suspicious.
But it won't.
2) a lack of transparency in the data, which has raised suspicions of fraud
3) political preferences held by the authors and the funders, which include George Soros's Open Society InstituteIt shouldn't so much as serve to cast doubts? In this era, in this time? Especially so in the additional light of both of the other two factors?
Prior to the era ideologically and culturally infiltrated and kicked off by the New Left such reasoning might hold, might be sound. But currently and since the soixante-huitards, since '68, such reasoning is perforce abstracted from reality, is dubious in the extreme.
Cite to publication of the "QRB" study in a major journal, versus a media outlet?
Just a suggestion, but you might save catastrophic climate change for a different thread - v. here. It's not going to serve you well in terms of reflecting lack of bias or in support of more probative interests still.
As to the other, it invokes a sizeable set of issues, both long term and short term, some difficult to resolve and others nearly, if not actually, impossible to resolve. E.g., I cannot imagine why the military would want to get involved in its own study. If the results would prove to be "good," massive criticisms and rhetorical browbeatings would ensue from every quarter imaginable. If the results would prove to be "bad," what exactly would the contrasting measure be? A comparison with Saddam & Sons' Iraq, both past and in some "alternative universe" future, an unknowable that itself would need to bring such unmeasurable factors as near and longer term stability, both regional and global, into account? What, for example, are the "alternative universe" factors to be assessed if we had not entered the fray in WWI? Or WWII? Or Korea? Or the Truman Doctrine that guided so much of our praxis during the Cold War?
No thank you, I'll take a pass.
If that's the case, and if the data can't be released, then all we have left on which to evaluate the study is our evaluation of the credibility of the authors. That's an intereting standard. On what other issues do we use that standard. Is it sufficient to support accepting the conclusions of their study?
I am sure that if the authors had tried hard enough they could have released their data in such a form that it was sufficiently anonymous to protect whatever confidentiality there might have been. Instead, they appear to have destroyed it. I find this extremely suspicious. They are, in essence, saying "trust us" because we are "scientists". But given the apparent extreme policial biases of at least some of the researchers, why should we trust them? And how can you even check for basic math errors, more-less something substantial, if no one reputable can get at the original data?
No one defending the study here so far has really come up with any reasons why we should accept a study that cannot be verified. Sorry. Until I see more, I will consider it a political hack job pretending to be science.
Quite a few commentators on this thread, however, seem ready to dismiss the Lancet study simply because its authors have a political views and its results do not agree with the casualty estimates compiled by using media and official Iraqi reports. Those sources of information, though, are subject to at least as many sources of bias as the Lancet study. Journalists in Iraq don't have the freedom of movement that would allow them to keep tabs on all the violence in the country, while the Iraqi government is clearly underfunctioning and subject to political bias of its own as well. Thus, the short answer to the question of how many Iraqis have been killed since the U. S. invasion is that no one knows. But until we see more solid scientific studies on the question, that answer won't change.
Both subtly and more directly you are substituting your own spin and platitudes ("the military in a democracy ...") for something more substantial. I might respond with "citizens of the demos should spin less" (albeit with more nuance and sophistication), but you're likely to note that's unfair, or some such response. That's but one of the reasons why I said, no thank you, I'll take a pass. I.e. there are fruitful discussions and then there are less fruitful discussions.
Imo and as Yogi Berra might put it: this is not one of 'em.
The upper range is nearly a million.
WHOI Jacket made the point with historical comparisons, but here's another.
The upper bound of Lancet is about the same as the usually accepted death toll in the Spanish Civil War. Spain in the late '30s had about the same population as Iraq does today, but since the Kurdish region has had much less death, the comparison means we would have to accept that violence in the violent part of Iraq has been considerably more deadly than it was in Spain.
Seems unlikely.
While publication in a major journal is a significant step in ensuring reliability of a scientific conclusion, it is not per se indicative of reliability itself. As far as screening, publication in a journal such as The Lancet is indicative of only two things: (i) the study has at least some basis in science, and (ii) covers an interesting topic. In fact, publication is only worthwhile to the degree it enables other knowledgable researchers to critique the work. THAT is the basis of my criticism. The absence of details regarding data and methodology prevents determination of reliability; that is the issue in The Lancet study. The link to the ORB poll is no different. As for the quality of a legal argument, a published "study" that can't be verified would stand a good chance of being tossed via the general prohibition on ipse dixit. See Kumho Tire Co. v. Carmichael. Even in court, the focus is on review of methodology, not publication.
As Jayhawk has pointed out verification in these matters is by repetition, something no one who claims that Lancet I and II and ORB are wrong has wanted to do.
The thing I have been looking for is a report on the total number of burials at Najaf, which should be a pretty good metric.
The central issue that routinely concerns Climate Audit is the undeniable fact that the proponents of human-caused global warming [AGW] have consistently refused to allow their taxpayer-paid data, or their methodology, to be disclosed to the folks who pay the freight. This is, of course, dishonest. At the same time, James Hansen, Al Gore, Michael Mann, and the Real Climate propaganda site have all received major funding from George Soros, through MovOn.org.
But none of them - not one of these scientists - will provide unrestricted access to their data and methodology. What reason -- other than fear of a negative peer-review, or of the discovery of outright dishonesty -- would any scientist deliberately hide data that was collected and paid for with taxpayer dollars?
The accusations aimed at McIntyre [of being a *gasp!* retired mining company employee] are simply a psychological projection by those who refuse to allow their own side's data and methodology to be examined for basic truthfulness; we are expected to unquestioningly accept their purported science. For example:Someone with "just" a degree in pure Mathematics and an advanced degree in Statistics isn't your ordinary blogger. McIntyre has consistently [and very politely] requested the corroborating data and their statistical methodology from Real Climate, Mann, Hansen and others -- all of whom refuse to cooperate. What does that tell you about the ethics/honesty of those pushing their falsified AGW conjecture?
There is a lot of similarity between the pushers of the "one million+ Iraqis dead because of America" and the "give us lots more money or the climate is gonna kill us all" propagandists.
There's nothing new under the sun. This guy understood what's going on in in both of the instances cited above:
The whole aim of practical politics is to keep the populace alarmed (and hence clamorous to be led to safety) by menacing it with an endless series of hobgoblins, all of them imaginary.
~~ H. L. Mencken
They usually do both. First they examine and try to understand what the other guys did. They look for errors, recalculate, turn it upside down and inside out. They determine if it's worth persuing. Then they move on to replicating with brand new experiments and data.
"Thus, the short answer to the question of how many Iraqis have been killed since the U. S. invasion is that no one knows. But until we see more solid scientific studies on the question, that answer won't change."
I agree. And the Lancet study doesn't help.
Y2K and the NASA temp debacle.
Both were right out there for ANY competent statistician to find, but all the other competent statisticians didn't find 'em and McIntyre did.
Good enough for me.
People who diss McIntyre's competence are only revealing their own.
Here's the link.
To me this suggests that any political bias was either unconscious or of the "acting out" variety -- not so much aimed at any practical result, but rather just an expression of generalized moral outrage. There's a lot of that going around.
As for why the military doesn't count bodies, I thought the reason was to avoid making killing into the goal. You tend to get more of whatever thing you count, so keeping a body count might lead to operations that kill the most people even when doing so doesn't make sense.
This is partially true, but as the National Journal article states, the UN did its own more comprehensive survey in 2004, which produced dramatically different results than the 2004 Lancet I report.
Here is the relevant passage:
By contrast, in a 2004 survey, the United Nations Development Program used many more questioners to visit 2,200 clusters of 10 houses each. This gave the U.N. investigators greater geographical variety and 10 times as many interviews, and produced a figure of about 24,000 excess deaths -- one-quarter the number in the first Lancet study. The Lancet II sample is so small that each violent death recorded translated to 2,000 dead Iraqis overall. The question arises whether the chosen clusters were enough to be truly representative of the entire Iraqi population and therefore a valid data set for extrapolating to nationwide totals.
Given the dramatic difference in results between Lancet I and the UN report, the Lancet II results should be setting off red flags all over the place.
Also, when "scientists" decide to become "activists", then everyone should be suspicious of their results.
The answer is not supposed, IMO, to be logical. It is to be an emotional, hysterical ratcheting up of BDS, of an illogical, damn-the-consequences insistence on getting out. This would support the goals of the putatively biased studier people. Dump on Bush and get the US out of the ME.
As the NYT admitted some time back, even if, as they expected, our immediate withdrawal resulted in genocide, we should get out. You can't support that kind of view with logical questions of the sort you pose. It has to be an inchoate, emotional repulsion on the part of a sufficient number--not that high--of the electorate.
That's the first thing that I noticed.
Their methodology had been used before. The fact that it finally had something to compare against in Iraq and ended up wrong reveals the methodology itself is probably useless... a fact that probably invalidates a number of previous studies in the same vein.
Dogwood, this statement is not true, as a quick googling of the relevant information would indicate. The UN study does provide some indication that the actual number of violent deaths in Iraq since the invasion began is toward the low end of the Lancet's estimates, but it's an overstatement to say the results are dramatically different.
The UN estimated the number of violent deaths in Iraq in the first 13 months after the invasion as somewhere between 18,000 and 29,000. The 2006 Lancet study estimated the excess deaths during the first eighteen months of the invasion as between 69,000 and 155,000, with at least half of those deaths ocurring during the last six months of that period. Thus, for example, estimating around 30,000 dead during the first year of the invasion and around 40,000 dead during the first half of the second year is consistent with both studies. Furthermore, this demographic pattern would be consistent with media reports about the upsurge of violence in Iraq at the beginning of 04. Once again, though, it's hard to tell how accurate the Lancet's mortality estimates are after April 04 because we've had almost no comparable studies.
What on earth is anybody supposed to do with that?
There is a non-trivial (love that term) possibility that the excess deaths were outside the published max and min. And no certainty as to where, within the max and min the reality is.
It's useless.