Daniel Sarewitz and Samuel Thernstrom, of Arizona State University’s Consortium for Science, Policy & Outcomes and the American Enterprise Institute respectively, co-authored an op-ed in today’s Los Angeles Times on how the debate over leaked e-mails from the University of East Anglia’s Climate Research Unit, and climate change more broadly, is colored by the mistaken idea that science can resolve contentious policy questions that implicate fundamental values. As the note, “If ‘pure’ science dictates our actions, then there is no need to acknowledge the role that political interests and social values play in deciding how society should address climate change.” It also leads to excessively politicized conflict over scientific findings and prevents honest debate over the underlying policy questions.
problems such as climate change are much more scientifically complex than determining the charge on an electron or even the structure of DNA. The research deals not with building blocks of nature but with dynamic systems that are inherently uncertain, unpredictable and complex. Such science is often not subject to replicable experiments or verification; rather, knowledge and insight emerge from the weight of theory, data and evidence, usually freighted with considerable uncertainty, disagreement and internal contradiction.
Thus, we write neither to attack nor to defend the East Anglia scientists, but to make clear that the ideal of pure science as a source of truth that can cut through politics is false. The authority of pure science is a two-edged sword, and it cuts deeply in both directions in the climate debate: For those who favor action, the myth of scientific purity confers unique legitimacy upon the evidence they bring to political debates. And for those who oppose action, the myth provides a powerful foundation for counterattack whenever deviations from the unattainable ideal come to light. . . .
The real scandal illustrated by the e-mails is not that scientists tried to undermine peer review, fudge and conceal data, and torpedo competitors, but that scientists and advocates on both sides of the climate debate continue to claim political authority derived from a false ideal of pure science. This charade is a disservice to both science and democracy. To science, because the reality cannot live up to the myth; to democracy, because the difficult political choices created by the genuine but also uncertain threat of climate change are concealed by the scientific debate.
Bruce Hayden says:
Part of the problem here is that by admitting that the threat is uncertain, he negates the requirement that something be done right now. Instead we can presumably wait until the science is better, and, there may even be a real consensus, instead of the manufactured one that we have had until just recently.
December 16, 2009, 9:46 pmsureyoubet says:
“If ‘pure’ science dictates our actions, then there is no need to acknowledge the role that political interests and social values play in deciding how society should address climate change.”
Yeah. In the old days, we would just throw a virgin in the volcano. It seemed to work sometimes, so why worry about the scientific validity of the theory.
December 16, 2009, 10:01 pmVOR says:
The editorial seems presumptive of the “threat”. Hardly unbiased. I have yet to see any serious debate on the benefits of increased carbon dioxide and potentially warmer temperatures(e.g. increased crop yield, possibly longer growing seasons). Maybe history and the medieval warm period in particular should be studied more closely.
December 16, 2009, 10:08 pmA scientist says:
Another key passage:
This is a welcome acknowledgment. I wonder how many here share the sentiment.
December 16, 2009, 10:08 pmHarry Eagar says:
Not me. There’s 0 evidence that any unusual warming has occurred. And, if scientist had bothered to read all of the Economist screed he linked yesterday, he’d know why.
If these guys cannot figure out that there’s a right and a wrong side by now, they should stop wasting everybody’s time by yapping about it.
But, of course, in the sentence quoted by scientist, they turn the rest of their own statement on its head.
There are lot of people not worth listening to on the subject of climate, and these are two of them.
December 16, 2009, 10:27 pmMalvolio says:
Not me. East Anglia e-mails expose a small conspiracy (one that contaminates, at a rough estimate, 20% of the evidence), but the key insight is that that fraud was not uncovered by the scientific community it was perpetrated against.
Compare it to the Bellesiles scandal. Apparently, Michael Bellesiles concocted, or at least exaggerated, a great deal of evidence to prove a fairly obscure point about the history of gun use in the US. Other scholars in the field, other historians, librarians, commentators, and so on called him on it, instantly, incessantly, and ultimately effectively.
Compare it to the Killian scandal. 60 Minutes got hold of some obviously forged documents purporting to prove that George Bush had not fulfilled his National Guard obligations. Again, the actual point was fairly minor, but again, typography and computer experts jumped on the print-outs, retired military officers pored over the wording of the memos, and in this manner, the truth was ultimately revealed.
Look what we have here. A truly momentous issue, one concerned the fate of the entire planet, and a big fraction of the most central people were busy trying to “undermine peer review, fudge and conceal data, and torpedo competitors”. How was this scandal uncovered? Meteorologists? The much-vaunted peer review process? The UN? The FOIA?
No, apparently, it was just some Russian hacker with some time on his hands, a retired statistician, and a few other random people.
Is there any reason to believe that the other “warm monger” (a faintly unfair label I find too clever to resist) aren’t bent in the same way as the CRU? They are the same kind of people, with the same sorts of education and certainly the same incentive structure. Maybe we just need more Russian hackers and more retired statisticians and we’d find the same sort of corruption.
December 16, 2009, 10:49 pmklee12 says:
As someone who has some scientific training, I was shocked by what happened in Climategate. Dr. Derek Lowe expressed my feelings better than I can
http://pipeline.corante.com/archives/2009/12/01/climategate_and_scientific_conduct.php
I used to be a believer of AGW, more or less. The revelation about the that the code was buggy,
http://chiefio.wordpress.com/2009/11/21/hadley-hack-and-cru-crud/
caused me to reexamine my position.You just can’t justify basing policy that might involve a trillion dollars on buggy code. Why not astrology?
My feelings about climate change follow those of Prof Pielke
http://pielkeclimatesci.wordpress.com/2009/12/08/roger-a-pielkes-position-on-climate-change/
“The real scandal illustrated by the e-mails is not that scientists tried to undermine peer review, fudge and conceal data, and torpedo competitors, but that scientists and advocates on both sides of the climate debate continue to claim political authority derived from a false ideal of pure science.”
But CRU did not produce science as I know it. I don’t see the skeptics claiming political authority. Pielke seemed much more cautious in expressing his views than some of the advocates of AGW.
klee12
December 16, 2009, 10:50 pmBruce Hayden says:
Uh oh. Climategate goes SERIAL: now the Russians confirm that UK climate scientists manipulated data to exaggerate global warming:
Should be interesting.
December 16, 2009, 11:07 pmBruce Hayden says:
I should say that I really don’t know what to think of the Russian revelation. Others here, such as EV, grew up under their thumb, and may understand them much better than I. But they sure seem to enjoy the role of the spoiler. I am not sure whether they were getting pushed in Copenhagen to cut CO2 emissions or pay graft to the 3rd World, and if they were, this could be seen as counter punching.
I think though that for a long time, some have been wondering if there were a connection between the warming we have supposedly seen through the last 20 years of the 20th Century and the fall of the Soviet Union, and the consequent loss of a lot of cold climate recording stations. CRU, NASA, and NOAA supposedly compensated for that, but then again, maybe not….
December 16, 2009, 11:22 pmzuch says:
BTW, any note of the AP review of the e-mails? The AAAS review? Worth a post?
Cheers,
December 16, 2009, 11:26 pmBruce Hayden says:
Also, maybe of some interest, Hide the Decline … and More.
Guess what? NREL only gets 95% of its money from the U.S. Government, through the National Science Foundation, and apparently is therefore immune from FOIA.
I swear that I saw the NOAA emblem on their sign in the dozens and dozens of times I drove by their headquarters in Boulder. Oh well.
December 16, 2009, 11:28 pmzuch says:
… well, maybe, if you like to decide factual questions on the basis of no actual evidence. But wouldn’t that be a tad hypocritical?
Cheers,
December 16, 2009, 11:33 pmzuch says:
Where’s the “bug”? Maybe it was the kerning in the printout fonts, eh?
Cheers,
December 16, 2009, 11:37 pmSuperSkeptic says:
Even if this is insightful, I do not see any solution in the near future, as even those who are anti-AGW would surely seek to silence anybody who was today, say, anti-electricity (an electricity denier) in the very same way that the pro-AGW people seek to silence the anti-AGW.
Or, instead of someone who is anti-electricity, say, someone who is pro-virgins-into-volcanoes, i.e., something that, damnit, just everybody knows is dead-wrong.
December 16, 2009, 11:42 pmA scientist says:
AAAS is not conducting a review, but issued a statement affirming their prior position based on preponderance of evidence:
http://www.aaas.org/news/releases/2009/1204climate_statement.shtml
The AP review is linked here:
http://hosted.ap.org/dynamic/stories/C/CLIMATE_E_MAILS?SITE=JRC&SECTION=HOME&TEMPLATE=DEFAULT
December 16, 2009, 11:48 pmDavid Schwartz says:
Is repeatability merely a feature? Because in my view, it’s an absolute requirement.
December 17, 2009, 1:17 amA scientist says:
Which part of the “screed” are you referring to?
December 17, 2009, 1:36 amNick says:
Daniel Sarewitz and Samuel Thernstrom can’t really mean this.
That I should be grateful to Hugo Chavez for dropping the folderol about measurements and models and just getting down to the crux, that the “silent and terrible ghost in the room,” capitalism, must be and can be destroyed.
Our revolution seeks to help all people, Hugo Chavez says to cheers and applause. And socialism, the other ghost that is probably wandering around this room, that’s the way to save the planet, capitalism is the road to hell! Sustained applause. Let’s fight against capitalism and make it obey us! Standing ovation from the floor in Copenhagen.
Nothing about the authority of science in that.
No charade at all. Just raw political rage, stripped of myth and ideal. Not subject to verification, as our editorialists write (one from the AEI!), or of falsification, as they ought to have said, just to make it quite clear that they reject the open society, and that they are its enemies.
To beat back the carbon peril, to slay this fearsome beast, they’d have us pretend it’s AOK to undermine peer review, fudge and conceal data, and torpedo competitors. They think democracy’s at risk when we all don’t just keep schtum.
I don’t know what they think “democracy” means. But “science” ought to mean something other than lying, I think. An honest, open attempt to learn things?
December 17, 2009, 1:56 amA scientist says:
Repeatability typically refers to producing the same test results under the same experimental conditions.
If by repeatability you mean the ability to reproduce the final results from the same raw data with the same analysis methodology, I am quite certain you would be able to. The “readme” file is quite detailed; and the code appears to be deterministic. I would appreciate an example of where the code is *wrong*, or *deliberately wrong*. And no, the commented out portions don’t count. Nor do the descriptions of work-in-progress frustration.
December 17, 2009, 2:01 amklee12 says:
Did you look at the reference?
If you square a number and get a negative number (because of overflow) don’t you think that’s a bug? If not a bug, what is it?
klee12
December 17, 2009, 2:17 amAllan Walstad says:
Utter nonsense. Whatever the merits or final outcome of the AGW thesis, this particular scandal has precisely to do with scientists on one side of the issue trying “to undermine peer review, fudge and conceal data, and torpedo competitors…”
December 17, 2009, 2:25 amBruce Hayden says:
Sure, if you run the same programs with the same input data, you will get the same output. But that really doesn’t buy us anything.
The better question I think is whether you could get comparable output data if you were to use comparable input data. Or, indeed, if someone were to start from scratch.
The problem, as I see it, is that there is a lot of massaging of data going on, and there is some reason to believe that it isn’t being done in a consistent manner. Some, if not much, of it seems to be ad hoc. That may not be accurate. But it is the impression that a lot of us have gotten.
Imagine a physics research project where some of the data is nudged up, some down. Some data points are thrown out, and others averaged together. Then, see if you can get a paper published on the research.
December 17, 2009, 2:39 amClimateGate and the Cost of Blurring Science and Politics | Liberal Whoppers says:
[...] is the original post: ClimateGate and the Cost of Blurring Science and Politics Share this [...]
December 17, 2009, 3:03 amis-ought distinction says:
There’s even a word for this sorta thing. “Scientism.” What we ought to do does not necessarily follow from what is the case.
December 17, 2009, 5:58 amis-ought distinction says:
And I think we’ve established that zuch is pretty much dishonest and incoherent on the subject of climategate. A wonder that he is still posting.
December 17, 2009, 6:08 amA scientist says:
Have you ever heard of “debugging?”. This is what programmers do when they code. They, for example, check the program flow for errors by printing out intermediate values in a calculation after each step. If they find an error, they correct it. Really good programmers document what they did, either in the code comments, or in a “readme” file.
Here’s an example. Say, some stations had only temperature gauge and no rainfall gauge. The data is collected into a table, where unavailable entries are marked as -9999 or some other nonsensical value (just like 1/1/00 was used as nonsensical entry for the date in the infamous Y2K problem). The analysis program is supposed to flag and simply ignore those. Let’s say a new person writes a code without knowing this, and, say, does data fitting by minimizing sums of squares of deviations from a model. She expects deviations of a few (whatever units), but instead squares a large (for this compiler) number and hits an overflow. She then identifies what caused the problem, documents the particular debugging test and the solution, and moves on.
Do you suggest that after the error was found the code was allowed to stay uncorrected? If you do, please document. If you cannot, for lack of skill, time, or motivation, may I have an apology please.
December 17, 2009, 6:38 amA scientist says:
The latter, reproducibility, is ruled out in the sense that we probably won’t get a new set of historical weather station data. The former, a re-analysis of the original data using some agreed-upon consistent procedure (homogenization, etc.) is absolutely warranted, as is the sensitivity of the results to particular assumptions. But this is exactly what a number of research groups do.
Thankfully, in physics “reproducibility” is available as an absolute check of the results’ validity. For some subtle things, statistical data manipulation, and adjustment of data for *known* systematic errors (nudging data points up or down, or throwing them out) is routine and legitimate. Some such adjustments may well appear “ad hoc” to an outsider, as they do here.
Still, the prevailing premise here is that the data analyst did something unprofessional. I am yet to see a firm indication of that.
December 17, 2009, 7:36 amzuch says:
Thanks for the clarification.
Cheers,
December 17, 2009, 7:58 amzuch says:
Oh, really? Guess what, my good sir: There’s still some giants to be slain, and fair Dulcinea awaits you. To the tilt, Don Quixote!
Cheers,
December 17, 2009, 8:05 amzuch says:
Yes. I followed this link:
http://chiefio.wordpress.com/2009/11/21/hadley-hack-and-cru-crud/
to this very short, uninformative, and dismissive tripe:
There is this link:
which redirects to this:
http://guide.opendns.com/?url=www.di2.nu%2F200911%2F23a.htm
(i.e., a broken link).
Yes, if you take the square root of a negative number, you’ll see interesting things happen. Including a FP exception which will clue you in to some problems. And if you add 1^2 to (short) +32767, strangely enough, you’ll get a negative number. But is this a “bug” that caused errors in the output data? I don’t know and you don’t know. Could it have been coded better and more robustly? I think so. But that isn’t the gravamen of the foaming hystericals of the anti-AGW “Climategate” parade. It morphs from this into “fraud!” and into “we’ve shown AGW is false! Ta-daa!!!”
Cheers,
December 17, 2009, 8:18 amMark Harris says:
“the genuine but also uncertain threat of climate change”
I guess I don’t understand how something can be both “genuine” and “uncertain.” Could someone explain this to me?
December 17, 2009, 8:47 amRich says:
Climategate is precisely why the government should NOT fund the sciences. Scientists are like drug addicts. They become dependent on government handouts and it eventually taints/corrupts their work—instead of searching for the truth, they search for (and find) results they believe will please their “drug dealers”, the politicians.
December 17, 2009, 8:51 amrarango says:
I don’t know squat about the underlying science of global warming. I do, however, have some understanding that based on the geological record, our planet, some billions of years old, has gone through considerable climate change long before our puny species walked upon it. There is nothing in the current science that convinces me one way or the other, other than, perhaps, science, like most other human endeavors is influenced by money and politics, as well as a search for truth. The whole debate, IMO, has decended into farce.
December 17, 2009, 8:55 amDavid Starr says:
The emails are damning enough but the true scandal is only evident to those of us who know programming. The “analysis” programs can be seen applying an arbitrary tilt to the temperature data. In short the “hockey stick” is the result of deliberate programming, it is not in the data. The programmers applied a “very arbitrary correction” to create the blade of the hockey stick.
December 17, 2009, 8:57 amThis isn’t debate among peers, this is plain old fraud.
zuch says:
Non-trivial error bars that don’t include H[0].
Cheers,
December 17, 2009, 8:58 amzuch says:
Recombinant DNA is a myth! They just want our money!! Gravity is a fraud!!!
Cheers,
December 17, 2009, 9:01 amrhhardin says:
Pulling equations out of your ass is not science.
Computer models grow in the direction of funding, an evolutionary fact.
December 17, 2009, 9:03 amzuch says:
A good start….
Do you also know that humans didn’t live on it back then? That paleontology documents mass extinctions? Sure, even with climate change, it may be habitable … for cockroaches.
Cheers,
December 17, 2009, 9:05 amtom swift says:
Oh, rubbish. There is indeed such a thing as science, and it has nothing to do with politics. Many millions of us worldwide know perfectly well what science is. We work in various aspects of it every day. It is a methodology for understanding the nature and the working of the natural world, which of course politics is not. There’s little room for BS in science; once the BS reaches a notable level, it ain’t science any more.
Sarewitz and Thernstrom seem to be stuck barking up the wrong metaphysical tree because they are saddled with an unrealistic a priori assumption; namely, the assertion that the garbage which has been coming out of East Anglia is in some sense “science.” The rest of us can see that it is not. It’s mere politics with the word “science” scrawled across it in crayon. Real science does indeed carry an authority – or, rather, a meaningful correspondence with reality – which politics can never approach.
December 17, 2009, 9:11 amrarango says:
Zuch–yes, I am quite aware that our species is fairly new–And yes, I am aware that there have been mass extinctions–cockroaches and horseshoe crabs seem to have done fairly well. I think there have also been considerable climate changes since our species walked on it–like for example several ice ages, and we seem to have come through it just fine–so what, sir, is exactly your point other than mindless snark?
December 17, 2009, 9:12 amgrichens says:
“Have you ever heard of ‘debugging?’.”
Have you ever heard of “Beta Testing?”
December 17, 2009, 9:14 amA scientist says:
I presume your libertarian approach applies equally to:
–Military
–Law enforcement
–Fire
–Basic schooling
–Roads and highways
–Air traffic control
On a related note, it is just mind boggling how many of posters professing libertarian views are or had been supported by the taxpayer. This would be all current and ex-military; nearly all aerospace; “hard sciences” folk supported through a government grant; significant support of university faculty, including law school faculty, supported by government grant overhead, etc.
December 17, 2009, 9:35 amhttp://volokh.com/2009/12/16/who-are-you-non-lawyers/#comments
A scientist says:
This is just silly. First, beta testing is meaningless if your software has one or two users (yourself and, maybe, the PI). Second, to the extent that you are referring to code validation, the *stolen* *internal* code and notes may well be the record of “beta testing”.
December 17, 2009, 9:42 amMark Buehner says:
Forget millions of years ago- our planet was this warm and warmer a thousand years ago… which is precisely what Mann and Jones worked so hard to paper over.
December 17, 2009, 9:45 amrosignol says:
The real scandal illustrated by the e-mails is not that scientists tried to undermine peer review, fudge and conceal data, and torpedo competitors, but that scientists and advocates on both sides of the climate debate continue to claim political authority derived from a false ideal of pure science.
Ah.
So that scientists tried to undermine peer review, fudge and conceal data, and torpedo competitors, is not scandalous? Instead, the true scandal is that both sides tried to claim political authority? So both sides are equally guilty of misconduct, and we should ignore the undermining of peer review, fudging and concealing data, and torpedoing competitors, because it really isn’t that big of a deal?
Sorry, no.
December 17, 2009, 9:49 amMark Buehner says:
Are you referring to the tax payer funded code almost certainly leaked by a whistleblower? Just want to be clear.
We could clear all this up if CRU would just release their code and metadata wholesale. All anybody asks is to be able to reproduce their results from the raw station data. The fact that no-one can apparently do that is far more damning than anything else.
December 17, 2009, 9:50 amCannon says:
We see evidence of ice ages, but scientists don’t agree on what caused them. How, then, can any scientist claim that they’ve sufficiently isolated and factored out THOSE forces when looking at today’s climate?
In other words, where is the scientist that has shown that whatever caused ice ages isn’t what’s causing today’s climate change? (Yes, yes – you can’t prove the negative; that that fact only further proves my point. One can never exclude unknown factors that may better correlate with the observed phenomena.)
To another point, government shouldn’t fund science because true science remains tentative forever.
(But governments SHOULD fund engineering, which has nothing to do with science.)
December 17, 2009, 9:54 amgrichens says:
“the *stolen* *internal* code and notes may well be the record of ‘beta testing’.”
Are you saying that the notes represent the record of an independent assessment already performed?
December 17, 2009, 9:57 ammelk says:
Sarewitz and Themstrom attempt to make an evenhanded approach to Climategate, which is at least better than the “couple of 10 year old emails” response from the likes of Al Gore. But it seems to me that the hacked emails expose two issues that are of huge importance to the Doomsday argument which is, essentially, that global warming is mostly anthropogenic, it is ever-increasing at an alarming rate and it is unprecedented in recent times. By eliminating the Medieval Warm Period and by glossing over what seems to be a decided stalling of increasing temperatures since about 1998, these emails reveal an attempt to conceal the most crucial points.
December 17, 2009, 10:09 amThis is of huge significance and arm-waving about a consensus of many scientists about all the other issues will not make this go away.
Duracomm says:
A scientist said,
Output from the CRU code is being used to drive policy decisions that involve trillions of dollars. Not testing software in that situation is lunacy.
That is the problem.
It may be a record of beta testing but we don’t know because in the past CRU would not release their code or data. Making it impossible to independently replicate their work.
December 17, 2009, 10:11 amA scientist says:
Please do not put words in my mouth.
December 17, 2009, 10:18 amkcom says:
I think the thrust of this thread is well illustrated in this Bloomberg article: Scientific Goals May Be Missed in Copenhagen Accord
Check out the first sentence:
“World leaders taking control of stalled climate talks today in Copenhagen may find the measures acceptable to 193 nations fall short of what scientists demand to slow global warming.”
See that? The measures…scientists demand to stop global warming.
Since when are scientists in the position of making demands on society? That’s the realm of politics. As noted above, scientific discoveries don’t have any sort of one-to-one correspondence with political and social policies. One doesn’t flow inevitably from the other. I don’t remember ever electing one of these scientists to make political and social decisions for me. And once scientists get into the habit of making social demands based on their research then their position as neutral arbiters of scientific truth becomes fatally compromised. As human beings, they can’t do both. NASA’s James Hansen is one of those that could rightly be put in this category. He should be removed from his position if he insists on compromising his integrity in the way he has been doing over the last few years.
Of course the argument could also be made that this was an inartful sentence written by a journalist (heh!) that isn’t a fair characterization of the situation. But I’d still find it troubling that a journalist would even think of scientists in these terms.
Then there’s this: “For 20 years, scientists working for the United Nations have provided guidance for global climate talks.” What does that mean exactly? High school counselors provide one kind of guidance and mafia enforcers provide another. Which kind are we getting?
December 17, 2009, 10:21 amNostromo says:
Read Lord Monckton’s “Caught Green Handed” Then think. Then comment.
December 17, 2009, 10:22 amJeff says:
scientist …
the Harry Read Me file clearly shows that the CRU was not able to replicate their own work at that point in time …
given the fact that:
1) no raw data has been released or indexed … (and may be deleted)
2) no programming code has been released with release notes explaining why the code adjusted data …
3) no station info has been released …
Nobody, not CRU, not you or me, NOBODY can replicate, test or validate their work …
December 17, 2009, 10:23 amGreg F says:
Some uncorrected errors from the harry_read_me file.
The significance of this scandal on the science is that it affects the foundation. The climate models rely on the surface record and reconstructions of past climate. If they are FUBAR so are the models and the projections they have generated.
December 17, 2009, 10:23 amA scientist says:
Look at the links at
http://www.realclimate.org/index.php/archives/2009/12/are-the-cru-data-suspect-an-objective-assessment/comment-page-2/
for sources of weatherstation data, including CRU data. Run your own analysis.
December 17, 2009, 10:26 amgrichens says:
“Please do not put words in my mouth.”
If you were not referring to an independent assessment, then what did you mean when you wrote “the *stolen* *internal* code and notes may well be the record of ‘beta testing’.”
December 17, 2009, 10:27 amFen says:
Enron Broker: “Such science is often not subject to replicable experiments or verification; rather, knowledge and insight emerge from the weight of theory, data and evidence, usually freighted with considerable uncertainty, disagreement and internal contradiction.”
Uh no. Thats not science. Thats religion.
Enron Broker: “For those who favor action, the myth of scientific purity confers unique legitimacy upon the evidence they bring to political debates. And for those who oppose action, the myth provides a powerful foundation for counterattack whenever deviations from the unattainable ideal come to light”
No. Skeptics never bought in the the “myth of scientific purity”. The AGW frauds did. Nice strawman though.
Enron Broker: “The real scandal illustrated by the e-mails is not that scientists tried to undermine peer review, fudge and conceal data, and torpedo competitors, but that scientists and advocates on both sides – ”
Bullocks. The real scandal is that Climatology is rife with corruption and fraud. “Both sides do it” is the response of an intellectual coward. Or whore.
Enron Broker: “because the difficult political choices created by the genuine but also uncertain threat of climate change are concealed by the scientific debate.”
Cue the Priests to translate the Latin to us “ignorant” rubes. Go pound sand.
And no, I don’t want to buy any of your carbon credits. Maybe you can eat them.
December 17, 2009, 10:35 amFen says:
“We do not believe the East Anglia e-mails expose a conspiracy that invalidates the larger body of evidence demonstrating anthropogenic warming”
Do you have the faintest idea how much research and how many papers rely on the fraudulent CRU data?
December 17, 2009, 10:43 amMark Buehner says:
Thats a complete BS answer. It is equivalent to pointing to a pile of astronomy data from the last 30 years and saying ‘the proof for relativity is in there, go find it and you can replicate my experiment’. Obviously that is unacceptable.
CRU needs to release the code which explains how (and why) the station data was adjusted. There is nothing in the GHCN data that can indicate the adjustments CRU made to homogenize the data, so the HADCRUT data is essentially useless unless you are willing to believe that whatever methods they used to choose and adjust their data isn’t relevant.
To go back to the Einstein analogy, it would be just as if Einstein returned some ‘adjusted’ data from his observations of Mercury and claimed to have discovered relativity, and when asked to show his work to point back to the raw data and say ‘trust me that whatever I did to the data was proper’. That aint science.
December 17, 2009, 10:44 amzuch says:
Brilliant satire … — … — … oh. Nevermind.
Cheers,
December 17, 2009, 10:53 amA scientist says:
NOBODY?
http://www.realclimate.org/index.php/archives/2009/12/are-the-cru-data-suspect-an-objective-assessment/comment-page-2/
December 17, 2009, 11:08 amD. Ch. says:
Professional scientists working at universities, government labs, and private companies get paid for the research they do. This means next year’s, next five years’, next ten years’ paychecks depend on making a case for more funding at the same or greater levels. I have worked in this environment for thirty years (with a Ph.D. in physics) and know that this sort of “Big Science” is almost always run by individuals who started their careers with a science or engineering degree of some sort, entered a Big-Science institution, found that it was more fun to supervise than do actual science or engineering, and went on to become bureaucratically inclined managers who tend to think and act like bureaucrats everywhere else in society. These are the sorts of individuals, by the way, who show up in the media labeled as important scientists, but mostly doing public relations for their research programs. Always remember — you can tell from their polished delivery — that these people are not so much scientists and engineers themselves but are instead supervisors of scientists and engineers, used to making polished and important presentations in front large audiences, who maybe decades ago got a degree in science or engineering before going into management. In all large institutions, power tends to flow to those closest to, and most responsible for, bringing in the money. If a scientist or engineer in Big Science has some concern about the truth of an assumption in a current project, or about how well a large computer model is performing, or about the accuracy of some important test or experiment, and goes to a manager, saying in effect these things really ought to be checked out, the scientist or engineer will almost always be turned down unless the manager feels threatened with bad publicity or an outside investigation. After all, re-checking something no-one knows for sure is wrong can only cause trouble in the short term, and what manager likes that? When I was young, a supervisor tried to set me straight by saying that their research was always “success oriented” — that is, oriented toward coming up with a funding story that was likely to lead to more money for the team. Over time this sort of environment tends to turn professional scientists into four-flushers (In poker, a four-flusher cheats by claiming a five-card flush of cards, all of the same suit, fanning out the cards at the end of the hand so that the one card not of the correct suit is hidden by the four that are.) Society can live with this sort of scientific four-flushing as long as, at some point, an actual product or new technology gets built based on the research program. Then, if the managers and their teams were basically correct and the hidden cards are basically unimportant, the project is a success. Managers who don’t worry much about hidden cards tend be more successful in the short and medium term — after all, no one knows for sure that the bad cards are present, and it’s easier to give a sincere presentation of your case when you think everything’s OK. Usually, however, at least one of the hidden cards eventually turns out to be important, as it did when the Hubble telescope was first launched and turned out to have astigmatism. Not to worry, though, once the authorities have bought in, the project has to be pushed through to something resembling success. In the case of the Hubble, you may remember, there was special equipment designed to fix the problem, and a special space shuttle mission made to install it in orbit. Even after a big mistake like this Big Science still wins because you need to consult and thus pay the same experts — the same people who participated in the original mistake — to design a correction to the original error. The Hubble story, of course, turned out rather well; so far no other hidden cards have turned up to cause trouble. Not so in the Challenger disaster, however — remember how Feynman demonstrated at a congressional hearing the importance of the hidden card about how the sealant turned stiff at cold temperatures, and what bad news this could be for the space shuttle. Originally managers decided to hide the sealant card, because a fix would have added a few more tens of millions of cost to the project, and the card stayed safely hidden, until one relatively cold launch day in Florida, on world-wide TV, we see that “Oops, this hidden card turns out to matter a lot” Of course, again, most of the same people responsible for the original mistake — they’re the only space-shuttle experts around– had to be involved in the fix, and paid for this involvement. (I’m sure the congressional investigations and lawsuits weren’t any fun to be involved in, though.) The present AGW debate is just another adventure in Big Science. The climategate emails and data fudging do not shock me because similar things happen all the time in this sort of institutionalized and government-sponsored science when facts start to undercut the funding story; really, the only way to avoid it is to return to the not-so-distant past (say, the nineteenth and first half of the twentieth century) and stop paying scientists and engineers lots of money to do research. If you must, give them a one-time prize for an important discovery (like the Nobel) but make it clear that they cannot expect a follow up money as a matter of course. If you do not want to try this sort of return to the past, you might as well go all the way and recognize that professional scientists and engineers are like lawyers — they get paid to be advocates for research programs. Then we could follow up on a recent Chamber of Commerce suggestion and set up a “science court” with “science judges” to rule on the facts of the matter every time a Big Science research program becomes controversial. Like legal judges, science judges would get paid the same no matter how they rule; and, like legal judges, science judges should recuse themselves from ruling on research programs they have been connected to in the past. Science judges can recommend that a different team of researchers, one with no previous connection to the questionable research program, run more tests and experiments to check results. We can hope that this sort of process, with both sides of the controversy equally funded and under equally expert challenge, would tend to force all those “fifth cards” out into the open so that they can be seen for what they are.
December 17, 2009, 11:08 amzuch says:
See that? Bloomberg claims that “scientists demand to slow global warming”. Let’s be accurate, please. And let’s deal with the “raw data”.
Cheers,
December 17, 2009, 11:09 amzuch says:
Do you have any evidence of “fraud”? Why should we trust your assertions completely sans evidence, when you insist that we should completely ignore work based of allegedly deficient but voluminous evidence?
Cheers,
December 17, 2009, 11:14 amA scientist says:
The referenced to science papers describing the procedures to homogenize are described here:
“Filtering inhomogeneities out of meteorological data is a complicated procedure. Coherent surface air temperature (SAT) datasets like those produced by CRU also require a procedure for combining different (but relatively nearby) record fragments. However, the methods used to undertake these unavoidable tasks are not secret: they have been described in an extensive literature over many decades (e.g. Conrad, 1944; Jones and Moberg, 2003; Peterson et al., 1998, and references therein). Discontinuities may nevertheless persist in data products, but when they are found they are published (e.g. Thompson et al., 2008).”
http://www.realclimate.org/index.php/archives/2009/12/are-the-cru-data-suspect-an-objective-assessment/comment-page-2/
According to the same, raw, unadjusted data reproduces the CRU trend albeit with a larger error. The implication is that the homogenization procedure does not introduce bias.
You are welcome to run your own analysis.
December 17, 2009, 11:15 amrobotech master says:
“Not A scientist says:
Repeatability typically refers to producing the same test results under the same experimental conditions.
If by repeatability you mean the ability to reproduce the final results from the same raw data with the same analysis methodology, I am quite certain you would be able to. The “readme” file is quite detailed; and the code appears to be deterministic. I would appreciate an example of where the code is *wrong*, or *deliberately wrong*. And no, the commented out portions don’t count. Nor do the descriptions of work-in-progress frustration.”
Repeatability in a fantasy world created for the sole goal of proving something is easy… however their is another check on this fantasy world which always creates the results the “creators” want… its called the real world. The real world and the fantasy world created by global warming nutter HAVE NEVER MATCHED. The models have always been wrong and many models have likely had to hard code historic temps in because if they started 200 age by the time they get to the current year they predict the earth is roughly even with Venus.
That simply put means the models ARE WRONG. I know you don’t understand much about computer models however one can create a model that says 100+40=1… its very easy to do… The code that was released along with the readme file clearly show that raw data was in fact not even really used… they could have used any data including the last 30 years of lottery numbers and the model would produce the same result every time BECAUSE THE MODEL WAS CREATED TO PRODUCE THAT RESULT.
December 17, 2009, 11:17 amStu in SDGO says:
I greatly enjoyed the back-and-forth in the comments here, but how people like “A Scientist” can ignore the intellectual dishonesty of Jones and Mann et al is beyond comprehension. There are many excellent analyses in the blogosphere that examine Climategate in detail:
For an analysis of selected leaked emails from CRU go to:
http://bishophill.squarespace.com/blog/2009/11/20/climate-cuttings-33.html
For an examination of potential temperature tampering of GIStemp go to:
http://chiefio.wordpress.com/2009/11/09/gistemp-a-human-view/
For more analysis of temperature records used to fabricate the AGW hoax go to:
http://wattsupwiththat.com/2009/12/08/the-smoking-gun-at-darwin-zero/
For an analysis of the code and code notes of various climate models contained in the leaked material from CRU go to:
December 17, 2009, 11:28 amhttp://www.americanthinker.com/2009/11/crus_source_code_climategate_r.html
klee12 says:
As a matter of fact I don’t think that CRU detected the error; it was Harry of HARRY_READ_ME.txt. I linked to a reference to a document from the CRU files because it was to the point. Here is the full text
http://www.anenglishmanscastle.com/HARRY_READ_ME.txt
It’s a long file. The comment in question is note 17 (about 10% down from the beginning. It is obvious that Harry inserted the debug statements into the code and discovered the error.
17. Inserted debug statements into anomdtb.f90, discovered that
a sum-of-squared variable is becoming very, very negative! Key
output from the debug statements:
OpEn= 16.00, OpTotSq= 4142182.00, OpTot= 7126.
DataA val = 93, OpTotSq= 8649.00
DataA val = 172, OpTotSq= 38233.00
DataA val = 950, OpTotSq= 940733.00
DataA val = 797, OpTotSq= 1575942.00
DataA val = 293, OpTotSq= 1661791.00
DataA val = 83, OpTotSq= 1668680.00
DataA val = 860, OpTotSq= 2408280.00
DataA val = 222, OpTotSq= 2457564.00
DataA val = 452, OpTotSq= 2661868.00
DataA val = 561, OpTotSq= 2976589.00
DataA val = 49920, OpTotSq=-1799984256.00
DataA val = 547, OpTotSq=-1799684992.00
DataA val = 672, OpTotSq=-1799233408.00
DataA val = 710, OpTotSq=-1798729344.00
DataA val = 211, OpTotSq=-1798684800.00
DataA val = 403, OpTotSq=-1798522368.00
Let me give some context for those who have not had programming experience on large projects.
In a large software projects keeping track of all the programs and all the data files must be a very important task. A program may make use of many different files and a data file may be used by many different programs. Programs may change over the years and data files may grow. Programs way be written by many people and changed over the years. Documentation of what the file contains is very important.
The first line of the HARRY_READ_ME.txt file indicates the work went on between 2006-2009. From the text file, the task of Harry was to supposed to clean up the system. His task was not to look at any particular program and test it. He spent about 3 years on it and did not record he was successful in bringing order out of chaos. As a matter of fact if the repository of programs and data are in as poor shape as Harry indicates I doubt that CRU can reproduce the results, especially if some data sets have been tossed out.
Others have looked at some of the code, and the code indicates fudging, i.e. trying torture the program and data to reach a desired result.
http://climateaudit.org/2009/11/22/these-will-be-artificially-adjusted/
I think there are more damaging instances of problems with the code, but I don’t have time now to track them down.
My impression is that the coding was done by amateurs … scientists (not professional programers) who were interested in getting their individual program to work and give a results. Over 20 years or more the programs a amended, the data enlarged and your programs may no longer run. I am sure CRU cannot run the code and get their results. If Harry had completed his project they probably could reproduce their results.
klee12
December 17, 2009, 11:31 amPink Pig says:
I suspect that part of the problem is that words often have multiple meanings, and that one can easily obfuscate an argument by switching meanings in midstream. The true debate is not about science, but about the scientific method, which was devised in the 19th century to eliminate some of the noise uttered in the name of science. Science before that time included astrology and alchemy, which have since faded out. The scientific method is based on total skepticism — no theory is ever established by science according to the scientific method. The true question is not whether the CRU researchers are scientists, but whether their science is consistent with the scientific method. I simply don’t understand the tu quoque argument advanced here — to the best of my knowledge, no skeptic has ever proposed to shut anyone up or to drown out opposing opinions. Science can be, and often has been, political, but the scientific method is not (see the trial of Galileo for details).
December 17, 2009, 11:34 amMalvolio says:
It’s considerably worse than BS. If you were to spend three years of your life and run your own analysis, and found, gee, no AGW. What then?
Well, we know what then. Insiders would keep your paper from passing peer review, you’d be publicly vilified as a “denier”, you would be slandered in the press, yadda-yadda-yadda.
What really grates on me is that if I published a paper tomorrow that claimed mice had 41 pairs of chromosomes instead of 40 — as unimportant a factoid as I can think of — if I failed to include enough information to replicate my experiment, I’d be laughed at by the entire mouse-chromosome-counting scientific community; assuming I did include such information, a half-dozen other mouse-chromosome-counting scientists would try to replicate the study and succeed or fail.
But here, a group of scientists release calculations based on secret data, claiming the world is about to end, and nobody nobody in the supposedly scientific meteorological community makes peep one about replication.
December 17, 2009, 11:40 amMark Buehner says:
That link has been taken down at CRU (amongst other things of late). Secondly, they are describing how the gridding was done, which is a separate issue. Finally- the example Realclimate provides is humerous (we picked 60 random stations… oh yeah, how? from a hat? Explain your methodology) and oddly both the graphs are in temperature anomaly when the raw data should be… well raw temperatures. So the data HAS been processed (they took an average over some unstated time period). Got to hand it to Realclimate, at least they are consistent.
Regardless- that is a description of intended methodology. Its not a list of actual methodology. The methods aren’t secret- but the devil is in the details. Wide brush explanation of vaguely how you intend to homogenize the data isn’t nearly good enough! You have to explain WHY you choose each station and why you made each adjustment! Obviously! If it was done via an algorithm, release the computer code and we can all see if what they claim was done was done and how well. However, without question, there was a great deal of manual shoehorning involved in building the dataset, and those notes are critical (if they exist).
In other words, telling me how you intend to do something is entirely different from explaining specifically what you did in a case by case basis.
I don’t know what you mean by that, can you explain?
Gee thanks! What does that mean? I don’t know how CRU picked their stations and I don’t know what was done to the data (aside from broad the brush junk they took off their website). Hence, I (and nobody else on the planet) cant possibly replicate their work. I can try my own judgments to follow their ‘rules’, but that won’t produce a match.
Is it or is it not important to you that a scientists work can be replicated?
December 17, 2009, 11:43 amfishbane says:
Johnathan,
how the debate over leaked e-mails from the University of East Anglia’s Climate Research Unit
Whatever one’s view on the merits of what precisely the emails say about climate change, I believe it important to state the facts for what they are: this is not some whistle-blower leak, this was a criminal hacking event. And apparently not in isolation: at least one other computer intrusion has been committed, and the timing is at least suggestive of a goal of influencing the events in Copenhagen.
While it is certainly everyone’s right to incorporate the email’s contents into their view of the overall debate, ignoring the underlying crime and what one might reasonably infer about the motives of the perpetrators is to willfully ignore the obvious.
December 17, 2009, 12:11 pmrobotech master says:
To fishbane
Their is no proof of hacking and their is no proof of a crime as of yet… what do you base these guesses on?
December 17, 2009, 12:14 pmWilliMc says:
The employment of science to advance false claims which advance a particular political have been around for a long time. Consider the following, all of which were advanced on the basis of correlation:
DDT & Bird’s Eggs
Nuclear Plants & Radiation Threat
Acid Rain & Trees
Chlorofluorocarbons & Ozone
Global Warming & CO2
All of the above supposed dangers have been falsified, with one exception. Nuclear Plant disasters can cause radiation, but it is not any way as bad as they predicted.
In addition to Climategate, Second Hand Smoke and Cancer data was manipulated with bad statistics in order to show a causual connection.
Now, we find manipulated data from Austrailia, New Zeland, Russia, to add to a no warming trend in the U.S. since the 1930s, when so many of the hottest days ever, were recorded. I seem to recall it was 38 State records.
December 17, 2009, 12:22 pmDotar Sojat says:
During the Medieval Warming Period, and prior to the Little Ice Age, when the Vikings grew crops and ran herds in Geeenland, wasn’t Florida above water and the earth habitable? It was warmer then than it is now. Was that bad? Despite the “hidden decline” of the past ten years or so, the earth seems to be on a longer term warming trend, but people differ on whether personkind is the prime cause. I tend to think not. The rub seems to be that so many have a need for AGW to be true as a means to acomplishing other poitical ends, i.e. regulation of economies and human activity, that are not achievable without AGW as a justification.
December 17, 2009, 12:38 pmVegasGuy says:
Software Engineering is a discipline utilizing formal processes, tools and techniques intended to consistently yield requirements-compliant software. In the absence of a priori formal requirements, software *cannot* be engineered.
Programming is an activity that yields code. Scientific research that involves programming will yield what software engineers refer to as spaghetti code, or kluges, or just plain junk. It can be “tweaked” to yield output, but it will never be well-behaved and amenable to substantial modification without likely introducing unpredictable consequences. If the amount of code is significant, eventually its questionable functionality will become dependent upon one or a few users whose intimate knowledge keeps it running, much like any cranky one-off device. Turning a kluge into a well-engineered product is difficult, if not impossible, and usually requires starting over. Analysis of the code may prove to be equally difficult, especially when unique, modified data sets are involved.
Re beta testing: This is an undesirable form of software system testing which is conducted AFTER development is complete, internal testing is complete, and the product needs to be exercised in the kind of operational environment, or environments, where it will eventually be expected to perform (Microsoft is fond of beta testing, and in fact embeds software in its systems to provide feedback on its products well beyond the commercial release.). This is clearly not applicable to the current discussion.
December 17, 2009, 12:41 pmHarryEagar says:
A scientist asks: ‘Which part of the “screed” are you referring to?’
The addendum in which the climatologist from the Australian Bureau of Meteorology reveals that the uncertainty in the Darwin temperature adjustments is at least as large as the signal being sought.
December 17, 2009, 12:46 pmwillis says:
“The real scandal illustrated by the e-mails is not that scientists tried to undermine peer review, fudge and conceal data, and torpedo competitors, but that scientists and advocates on both sides of the climate debate continue to claim political authority derived from a false ideal of pure science.”
And AGW advocates continue to coverup their embarassing loss with false claims like this. I have yet to hear a skeptic scientist claim political authority derived from anything. AGW is a power play pure and simple to reshape the world to conform to their agenda. Skeptics have tried to subject the process to real science and only managed to get their careers destroyed by the AGW “scientists.”
December 17, 2009, 12:56 pmHarryEagar says:
A scientist also sez: ‘Really good programmers document what they did’
I suggest you read Stephen Schneider’s ‘Science as a Contact Sport,’ in which he justifies not turning over code in FOIA requests because:
1) it is so idiosyncratic that no one else could use it; and 2) it contains many ‘undocumented subroutines.’
He is talking about the models that suggest catastrophic warming.
The whole thing is a con.
December 17, 2009, 12:58 pmzuch says:
Cites for these ‘facts’? Thanks in advance.
Cheers,
December 17, 2009, 1:17 pmfishbane says:
Richard,
Now, it’s all about the theft. Forget the content.
Don’t put words in my mouth. I never said any such thing. I suggest you read my original comment again, as you clearly failed to comprehend it the first time.
Robo Techmaster:
I don’t know what you consider proof, so I don’t know how to answer your assertion. But Phil Jones of the University of East Anglia asserted that it was a hacking event, the emails in question were placed on a Russian server, and the existence of the emails was posted on the Climate Audit site by an unknown person. There is a summary of this here. It is widely reported as a hack event, and I haven’t seen anyone dispute this with any real information. If you have such, please post it.
December 17, 2009, 1:18 pmMike says:
“The real scandal illustrated by the e-mails is not that scientists tried to undermine peer review, fudge and conceal data, and torpedo competitors…” Actually, yes it is the real scandal.
December 17, 2009, 1:20 pmThe real scientists are the ones who are asking to see the data and the models so they can subject them to independent scientific inquiry. This is how REAL SCIENCE works.
This op-ed in the LA Times is just another way of trying to distract attenion from the real scandal.
Andrew Myers says:
Remember that 99.9+% of scientists have no connection to politics whatever. Let’s not overreact based on what seems to be bad behavior by a few climate scientists. As a scientist I find their evident behavior unacceptable. But they’ve probably ruined their careers, so the incentives for good research practices are already there.
The scandal, if there is one, is in extrapolating from poorly validated scientific research to far-reaching political remedies. If public policy is going to be based on science, then yes, we should insist that the science be extremely open and convincingly validated. But for ordinary scientific endeavors, insisting that scientists have to meet this standard (e.g., are subject to FOIA) would be incredibly destructive; it would end science (and higher education) as we know it.
December 17, 2009, 1:24 pmzuch says:
Cites for the above claim (about the falsifications, not the one admitted danger)? Thanks in advance. The brown pelicans would thank you too.
Cheers,
December 17, 2009, 1:25 pmGreg F says:
Try Here
December 17, 2009, 1:32 pmTMLutas says:
The importance of the scandal is that it is provoking independent reviews of the national data. In Russia another shoe’s dropped in that there’s an accusation of siberian cherry picking with a quarter of stations used, virtually all stations that have been moved have been used (moved stations need adjustments that have been the subject of years of controversy), and an accusation that only the warmer stations are being used. Russia is very influential in most global climate models used to justify AGW claims.
New Zealand and Australia are both undergoing “second looks” that don’t look very good for the AGW consensus. The UK is undergoing a 160 year review of its data. In the US, the ongoing surfacestations review is likely to have increased impact when it finally finishes because of the CRU controversy. The huge number of bad stations in this “premier” USHCN network is a travesty.
So, yes, the case isn’t proved by the CRU emails. The code and comments are more damning. The entire affair is damning enough that we’re primed for follow on scandals as the data is rechecked. If people truly have been inventing stations and putting in incorrect data (as Harry Read Me.txt alleges) it’s going to come out now. And if it’s not, at least we’re going to have a proper look.
December 17, 2009, 1:34 pmFub says:
I haven’t read HARRY_READ_ME.txt, and I haven’t read the code in question. But, I do know enough about these sorts of bugs in analysis of large data sets to raise my eyebrows.
Foremost: they don’t necessarily indicate bad data.
They may be caused by bonehead bugs, or “stoopid programmer tricks”. But they can also be caused by naive assumptions about machine arithmetic. If the latter, they can be corrected by many means.
Long story short: Floating point computations (including double precision) are finite precision. They can bite you in the posterior, especially when your data set is large, and even more especially when you make naive assumptions about the arithmetic. The real problem is that if your result seems plausible, and even worse if your code appears algebraically correct, you may not even realize that you’ve been bitten.
At least the result Harry spotted wasn’t plausible.
See Prof. William Kahan’s poignant (and quite humorous) paper, How Futile are Mindless Assessments of Roundoff in Floating-Point Computation? (PDF). Prof. Kahan is a godfather of highly accurate numerical computation.
In the first graf he observes:
But these climate computations do matter. I agree with Dr. Derek Lowe, also quoted by Klee12 above: “You just can’t justify basing policy that might involve a trillion dollars on buggy code. Why not astrology?”
Are there subtle but correctable arithmetic bugs in the CRU’s massage and analysis codes? Nobody knows. But we do know that one programmer spotted one brazen ugly bug at one point in code development and data preparation.
Given the enormity of the political and economic decisions that may be based on the results, a thorough and gimlet-eyed review of the numerical methods used to reach the results should not be out of the question.
December 17, 2009, 1:37 pmfishbane says:
GregF: thanks for the link, I hadn’t seen that.
I don’t find it terribly convincing; that’s primarily a discussion of the author’s understanding of how mail servers are generally configured, and a batch of assumptions based on the structure of a zip archive.
One assumption is that the mail server was saving a copy of mails, and that that machine was unlikely to also be a repository for the random, non-email cruft. Apparently the author is unfamiliar with NFS, and the general use patterns of mail servers by technically inclined people. For a variety of reasons, folks who spend a lot of time at the command line tend to end up doing so on mail servers; they’re frequently the primary gateway to the net, so they’re (1) a common hop when shuffling files around, (2) likely to be more insecure than other machines, because lots of people have access to them, and (3) likely to be targeted in any attack, because of one and two. Now, this is supposition on my part, too – I don’t know. But I’m explaining why I don’t place a great deal of weight on that person’s analysis. I’m a long-time sysadmin too, and having seen my share of messy networks, find his reasoning unconvincing.
Others are speculating that the FSB is behind it.
December 17, 2009, 2:01 pmBruce Hayden says:
The problem with assuming that the CRU stuff got out through the site being hacked because that is what has been most reported depends on several assumptions. One is that those doing the reporting are unbiased. Another is that they independently got their information. But we know that isn’t true. It isn’t that there is a conspiracy, but rather, that most of those reporters reporting the same information are doing so because they got their information from the same place.
The problem with the outside hacker theory, for me, is that what was leaked was a small part of the whole. In particular, a small part of the emails. Likely, some of the more damming of them. But small, none the less.
I prefer the theory that all that material had been assembled, for some reason, such as responding to a FOI request, and then wasn’t released officially. In response, someone, almost assuredly from the inside, released the information previously assembled.
In order to make those selections of emails, either someone had inside access to the email archives, or spent days and days online hacked into their systems. Or, what was released was a minuscule fraction of what the (alleged) hacker downloaded.
Also, you have the problem that many of these emails were likely archived, given their age (and, no, not all of them are that old, some are fairly new, contrary to assertions by AlGore). So, this is more indicia of an inside job, and likely after a FOI request. Similarly, getting access to both email AND programs, etc. is problematic in a hack. The (alleged) hacker would likely have had to compromise multiple firewalls and security systems. Doable, yes, esp. given the apparent programming expertise of those involved (i.e. they may have been as incompetent in security as they were in programming). But highly unlikely, esp. today.
December 17, 2009, 2:58 pmJerry M says:
A Scientist,
Tell us why these CRU ‘Scientists’ only used 40% of the Russian temperature measurements, you know the 40% which showed some warming, and left out the other 60%, which showed cooling?
This has been shown to have been done for Austrailia, New Zealand and Nashville, TN. What gives?
December 17, 2009, 3:02 pmklee12 says:
There is a translation of the Russian paper on the data that was sent to CRU. See
http://climateaudit.org/2009/12/16/iearussia-hadley-center-probably-tampered-with-russian-climate-data/
and click on the link [Update: here lea1] near bottom of 2nd or third paragraph.
This paper is damning! Read the introduction at least, but the whole paper should be understandable to college graduates even if you didn’t major in a science. Several points:
1. On Dec 1, the British Meteorological Office made available some part of database. The paper points out there 5000 stations are used in global temperature computations but data of 1500 were released.
2. Of the 5000 data sets from weather stations, “to calculate the global land termperature the Hadley Centre has used the data for only a quarter (121 out of 476 of the Russian stations … The territory of Russia constitutes about 12.5% of the Earth’s land area … 121 stations, which represents only 2.4% of the total number of stations used to calculate global temperature”
3. The data sets chosen seemed to be those that would predict rapid increase of temperature, and those that were omitted did not support increasing temperature.
klee12
December 17, 2009, 3:13 pmBruce Hayden says:
But not apparently independently. Rather, they appear to work together to fix things when one disagrees with another, with the CRU data apparently considered the Gold Standard.
It is not that adjusting data is, all by itself, unprofessional, but rather, we don’t know the why. For example, why did CRU eliminate most of the Darwin stations, then identically adjust two of the three remaining sites up over the last decade by about 2 degrees C, while leaving the third unbiased, when all three have almost identical raw temperature data, and then averaging them together. That sort of thing. The adjusting may be totally legitimate, but how do we know that?
December 17, 2009, 3:18 pmMalvolio says:
I don’t like arguing over definitions and I certainly don’t like attacking the beliefs of people whose conclusions agree with my own, but I should point out that if you define Software Engineering as VegasGuy does, you have defined it as almost utterly worthless.
There may be situations where the requirements can be accurately prescribed a priori and those requirement remain accurate for the time it takes to implement, test, and deploy the system, but in the 30 years I’ve been programming, I haven’t see such a situations.
Take a very small part of the current case. Whoever undertakes to model global temperature records has to collate the raw records of some thousands of stations and observations. What he will learn in the course of doing so is that every file is just a little bit wonky. The information is represented differently, points are missing or mis-recorded stuff like that. A useful skill would be writing code that can reliable accept whatever information is available; “Software Engineering” as defined above would be detecting and rejecting all imperfect data files (the vast majority), since that would be the only way to write the requirements.
December 17, 2009, 3:18 pmBruce Hayden says:
Actually, sometimes it does. But what about the situation where the bug isn’t found? It may have worked fine when run with original, possibly well defined, data, but errors occur when less well behaved data is utilized. Is the bug caught?
The code I have seen looks like it was written either by the scientists involved, or by graduate students working for them. Not by anyone schooled and trained in software engineering. Yes, code like that can be debugged, but Ian/Harry seems to have pulled out his hair trying to do so and rationalize it. By all indications, he had a thankless job. I think that if I were in his position, I would have bailed into something more sane within a month or two. And, yes, I have had jobs like that, but never for three years, or longer than maybe three months.
Part of the problem that they needed to have debugged was the side effects. These are far less apparent than the program not going where it is supposed to. Some commenters above have mentioned some of those. For one thing, the programmers didn’t always seem to be aware of what happens when different types of variables interact, or what happens with overflow. And, that doesn’t even take into account the problems with accumulated errors through use of (digital) floating point numbers.
December 17, 2009, 3:34 pmGreg F says:
Fishbane,
I don’t find your argument terribly convincing. It wasn’t about how “mail servers are generally configured”, it was about how the UEA’s mail servers were configured. He supported that with headers available from some of the emails. The assumption that mail servers were saving a copy is not a big assumption. In fact, it would be a rather silly to think they were not archiving all email for strictly legal reasons. Your hand waving past the details with claims like email servers are “frequently the primary gateway to the net”. Again, were talking about UEA’s system. The idea that any university would use the email server as a “primary gateway to the net”, is well, I don’t know what to say. Also, what does a email server have to do with “shuffling files around”? Someone who resorts to generalizations to dismiss a detailed specific report is not making a convincing argument.
December 17, 2009, 3:39 pmrobotech master says:
IPCC report assessment 4… simply open up to the model prediction/forecast graphics and all the models state either some or massive warming over the next… well forever. Think its 4 maybe or 5 models in the IPCC all of which are wrong. In fact the closest model to being right is the model that basically predicts no warming… of course this model is solely based on no more raise in CO2.
I consider proof being a legal body such as the police saying they intend to present/press charges against whoever did it based on a crime they committed…
Right now the only ppl claiming crime are CRU and the parrots in the media. I know a hair bit about criminal justice and right now I see nothing from the info reported that any crime took place(expect by CRU).
The closest I’ve seen to this being a crime is that the police are looking into the matter to see if a possible crime was committed…
As was stated earlier by others the likelihood of this being a leak is far greater then it being a “hacking”. We know this because of the info that was released(all appears to fall under an FOI request) and the fact that it was released after repeated FOI requests were denied based on BS.
Once again unless their is some proof I think its best to stick with the most likely answer vs arguing that some super hacker collected e-mails for the last 3 years and posted them on the web… because for the most part this wasn’t some smash and grab the e-mails(1,000s) had to be collected over a good period of time by someone… its far more likely that someone collected them for a FOI then someone spent months/years breaking into servers, sorting through them and picking out the “juicy” e-mails to release.
December 17, 2009, 3:42 pmBruce Hayden says:
Let me add that scientists pretending to be programmers (or their graduate students) likely don’t even know about those sorts of problems, and so don’t take them into consideration when doing the original programming, or, indeed, during debugging when they try to follow the code. This is the sort of stuff you are more likely to get in a CS program. It is often very subtle. The code looks right. It isn’t though, because something is happening under the surface that isn’t apparent, under the surface.
That reminds me of some of the problems like this that I have encountered over time. Anyone remember PL/1? It converted anything to anything, all under the surface, and occasionally would give you a warning that something questionable had happened. It was trivial to code examples that failed – for example, if you had a numeric string, you could force it to not match itself by converting it (automatically) to binary and back. So, it could go from left justified to right justified, and no longer match. All without any apparent problems.
I hated that language, since I have always been the paranoid type, and want all of my type conversions to be explicit.
December 17, 2009, 3:44 pmMark Buehner says:
1.Phil Jones being the embarrassed party, i’m not sure how much standing he has (or technical expertise for that matter).
2.The Russian Server is immaterial. What does it demonstrate? If it was a Haitian or Korean server would it matter? Would a whistleblower post it on his home server, is that what you are insinuating? I could post anything on a Russian server in a matter of minutes. So what?
3.Emails posted at climate audit. Again, where would you expect to find them?
You’re also leaving out the fact that this stuff was leaked to the BBC FIRST, who promptly sat on it. Occam’s razor suggests it was someone intimately familiar with that system and the players involved, and able to extricate the proper items without fishing through terabites of chaff. If they were simply looking to ruin careers or start a scandal, going to the BBC as a first resort seems unlikely as well (their response, or lack there of, was good proof as to why that is).
December 17, 2009, 4:07 pmSagar says:
Zuch:
having perused your posts, among all the snark and clever reposte it is not clear what your point is.
Are you telling us that AGW is a fact? will that AGW cause problems? will the problem be significant enough that it needs a “human solution”? are the current “solutions” the right ones? is the cost of the fix worth the benefit?
unless you can prove that the answer to all of the above questions is a definitive “yes”, you have no case for Cap and Tax or Kyoto or Copenhagen or whatever the globalwarming alarmists are trying to do.
The burden of proof is not on the skeptics; we are not asking the taxpayers to fork over billions or trillions of dollars. You have to prove your case for funds. Please do.
December 17, 2009, 4:29 pmHarryEagar says:
Mark, only part of the CRUtape file was given to the BBC, apparently.
And the first post of the entire CRUtape file was not to Climate Audit, was it?
I have been reading Hoggan’s ‘Climate Coverup,’ in which he crows on just about every third page about having received ‘leaked’ emails that make climate skeptics look bad.
I have been a denialist ever since Hansen declared he knew what the global surface temperature was in 1900, so I’m primed to think Hoggan is wrong. But fair’s fair.
If the pro-AGW guys liked the leaks before, they are in a poor position to complain about them now.
December 17, 2009, 5:17 pmBruce Hayden says:
I don’t think that it really matters a whole lot how the CRU stuff ended up getting disclosed, except for all of those who want to discount it because of their claim that the CRU system was hacked and the stuff was stolen. And that is just trying to change the public focus from the contents of the disclosure. Regardless of how it got out, the stuff speaks for itself (ok, not to res ipsa loquitur levels).
Indeed, I am to the point now that I just ignore people whose response to the CRU disclosures is that the data was stolen. So? And that makes the data false? How? It is hard to make this argument with a straight face, when a number of the people involved have (maybe inadvertently) confirmed their authenticity.
December 17, 2009, 5:59 pmMark Buehner says:
My point is- if you are a hard core hacker denialist, or a hired gun oil company shill, is your first stop with your hacked data the BBC? I hadn’t considered the Beeb a particular bastion of skepticism of global warming (and I say that with all the sarcasm I can muster). If militant green hackers broke into Mcintyres emails, would they take them to Fox news?
December 17, 2009, 6:03 pmDaily Right 12/17/09 « The Quantum Conservative says:
[...] *ClimateGate and the Cost of Blurring Science and Politics, by Jonathan H. Adler. [...]
December 17, 2009, 8:17 pmFen says:
Do you have any evidence of “fraud”? Why should we trust your assertions completely sans evidence, when you insist that we should completely ignore work based of allegedly deficient but voluminous evidence?
Are you actually claiming that there was no fraud at CRU?
Can I interest you in some Enron stock?
December 17, 2009, 9:28 pmElliot says:
I’d suggest it can buy us quite a bit.
That is a first and very elementary step anytime one is investigating data and coding. It provides a good indication the integrity of the input data has been maintained.
Suppose the folks at East Anglis can’t come up with programs and data sets that yield the exact same results they previously obtained? Exact here means absolutely the same.
If the same results are obtained, then one can begin looking at how the raw data was massaged to produce the input data set, identify raw data that was not included in the input data sets, and what the program then did with it.
Another question that seems to have emerged is exactly what raw data is. I’m getting the idea GHCN data is the source for much (most?) of the climate research. If this data was adjusted from the actual temperature readings taken from the thermometers, how was it done? What was adjusted, what was eliminated? If that data is biased to warming, then everything downstream would also be biased. Anyone have a better grasp of this?
December 17, 2009, 9:49 pmzuch says:
WTF is this self-congratulatory alleged “comprehensive network analysis”?!?!? Looking at friggin’ path names? Talk about hackery…..
Cheers,
December 17, 2009, 10:25 pmzuch says:
… if your conclusion is predetermined, and you want to take very weak circumstantial evidence mixed with wild conjecture as “proof”. But no one has shown that any of the alleged flaws or mistakes were in fact wrong as applied to the matter at hand, or affected the data in any significant way.
Cheers,
December 17, 2009, 10:28 pmzuch says:
This is easy. Someone let the machine use 32 bit integer arithmetic to add sums of squares. They overflowed the 32 bit integer, and got a -1.79B number at one point … when the last data element that was run through was an obvious outlier (at 100 times the magnitude, more or less, of any of the other data points). It overflowed, they got a FP exception trap, and they had to find out why. I’m sure someone did. While the code was not robust, the data seems far more suspect, and wouldn’t have caused a problem without that extreme outlier data.
Cheers,
December 17, 2009, 10:36 pmzuch says:
When you’re clueless, you ought to STFU.
Cheers,
December 17, 2009, 10:45 pmzuch says:
Harry doesn’t look too competent (which may explain some of his frustrations). He was attributing [as I explained in a previous thread] the errors in the “sum-of-squares” code to some floating-point errors or funkyness, when it’s pretty obviously an overflow in 32-bit integer arithmetic. Agreed, they should have forced FP calculations and a FP accumulator, but as I said above, this would have been relatively innocuous if there wasn’t one bogus data value. Perhaps the FP exception was a blessing in disguise, flagging some bad data.
Cheers,
December 17, 2009, 10:54 pmzuch says:
This is “comprehensive network analysis”?!?!?:
This person writing this is clueless. Really. If he thinks that a directory titled “Documents” shouldn’t contain a mix of file types, he’s a moron. It may be disorganised, but I suspect if someone examines his basement floor for Cheetoh crumbs….
That he worries that e-mail files should contain only the body of e-mails shows a simplistic and ignorant view of how e-mails are stored on different OSs (along with a bit of arrogance that the one method he knows is the only way to do things). As does his complaint about non-sequentiality of file names. IOW, a person speaking far above their level of competence.
Don’t bother citing this incompetent again. Thanks in advance.
Cheers,
December 17, 2009, 11:07 pmzuch says:
Once again, how about some argument, rather than bald assertion. Thanks in advance. Show that they’re “all .. wrong”.
Cheers,
December 17, 2009, 11:13 pmzuch says:
Sorry. I doubt I can help you much. I state things as forthrightly as I can. You can read them, and if you have problems with something I say, you can ask for clarification, and if I have the time, I’ll try to help you. Beyond that, it’s up to you.
Did I say that somewhere? Did you think I said that? I’m being as clear as I can, and I don’t think that I would have given any rational person reason to believe that I asserted that. Please try to respond to what I’ve said and try not to assume I’ve said anything I didn’t say. I think that I ought to be the person deciding what my message is, not others. Those that do such might be accused of “straw man” argumentation.
Cheers,
December 17, 2009, 11:22 pmzuch says:
The burden of proof for an assertion is on the proponent. If you want to argue there was “fraud”, provide the evidence. It’s not up to me to disprove this assertion, particularly when there’s no evidence provided to support the original assertion. Sorry, that’s the way things work.
Cheers,
December 17, 2009, 11:27 pmDuracomm says:
Zuch said,
Yet he was in charge of the very important task of figuring out what the code did and making sure the code was functional.
That brings up this question. If someone you describe as “doesn’t look too competent” ends up in the important position of making the code work how many other people who “doesn’t look too competent” ended up being responsible for other critically important work?
Bold emphasis mine.
Heckuva a lot of assumptions their Zuch, I’m not sure you casual attitude toward the problems with the code is justified.
If code of this quality would not be allowed in life critical systems (medical devices, flight control systems, etc.) It absolutely should not be used to generate data that is being used to justify decisions with trillions of dollars of economic impact.
December 17, 2009, 11:57 pmrobotech master says:
Umm they claim that we’d be 5 degrees warmer then we are now… and thats according to the heavily “adjusted” data that has tons of warming added. So once again they are wrong.
December 18, 2009, 1:50 amrobotech master says:
stupid edit
heres the link
http://www.ipcc.ch/pdf/assessment-report/ar4/syr/ar4_syr.pdf
goto page 46
3 models all claim by this time we’d be freaking venus.
The only model thats close is the model that assumes CO2 has zero impact on climate. Marked as “Year 2000 constant
December 18, 2009, 1:58 amconcentrations”
Nobody At All says:
Since this (oddly) hasn’t been mentioned in any of the comments, some of you may find this open-source GISTEMP coding project interesting:
http://clearclimatecode.org/
It invites the participation of skeptics, so have at it.
December 18, 2009, 2:02 amrobotech master says:
Unless their releasing the raw data creating code is meaningless since you have nothing to put in the code…
December 18, 2009, 2:10 amFen says:
Zuch: The burden of proof for an assertion is on the proponent. If you want to argue there was “fraud”, provide the evidence. It’s not up to me to disprove this assertion, particularly when there’s no evidence provided to support the original assertion.”
You obviously haven’t been following this. If you want to pretend that
1) deliberately adjusting raw data to match your theory
2) ommiting raw data that disagrees with your theory
3) perverting the peer review process to censor research that disproves your theory
4) destroying data to prevent review by indpendent experts
is not fraud, then there’s no point in wasting time on you. You’re either too lazy to do your own research, or simply intellectually dishonest. I guess you’re a Denialist…
Every paper that relied on CRU as a source of data is now invalidated. And thats the bulk of AGW research, if not all of it.
December 18, 2009, 5:43 amzuch says:
Clue for you: The guy “making code work” is not considered to be in an “important position”. I should know; I was one for a while, and even while paid more than the Ph.D. post-docs, got less recognition. But not all such people are incompetent; but this guy Harry seems to have missed the boat on the FPE. Which means that you ought to read the HARRY_READ_ME file with a grain of salt, rather than insisting that all his bitching is the truth, the whole truth, and nothing but the truth.
I’ve explained why I think the FPE might not be the big problem that the anti-AGW people are making it out to be. Amongst other things, it did trap, and came to the attention of the programmer. Past that, I don’t know what was done. But I’d assume something rather than nothing.
Such systems are realtime systems with exclusive control of critical processes. There’s no possibility of independent validation, error-checking, non-real-time correction, etc. [or, in many cases, just turning the machine off and ignoring it]. So no comparison whatsoever.
Cheers,
December 18, 2009, 6:44 amzuch says:
I think you’re hallucinating. Venus surface temperatures are over 400 °C. The models show warming by 2100 of 1.8 to 4 °C.
What’s more mystifying than your hyperbolic claim is your claim that the models are wrong now. Where does it show that? Just a reminder to any time-travelling SpaceCorps™ guys out there: The year is 2009, not 2090.
Cheers,
December 18, 2009, 7:04 amzuch says:
Where?!?!? It seems you have the gift of seeing things that others can’t see. You know, Haldol can help with that.
Cheers,
December 18, 2009, 7:07 amzuch says:
The problem is that #1 hasn’t been shown, only alleged, #2 may have a perfectly innocent (and good) explanation, #3 hasn’t been shown (nor has the ‘research’ [such as it is] been shown to “disprove [the] theory”], and #4 hasn’t been shown. Alleged … often at 110 dB … and over and over again … but not shown.
Huh? How has the CRU data been invalidated? Even if the allegations above are assumed true, that doesn’t make the data “invalid”. Questionable, perhaps, but not invalid. It would be like tossing the whole barrel for a few rotten apples. Yes, you could do it to be “on the safe side”, but you’d be ignoring a lot of good apples, and losing a lot of money.
Cheers,
December 18, 2009, 7:16 amNobody At All says:
GISTEMP data sources are available here, if you would like to contribute to the open-source coding project.
December 18, 2009, 7:38 amFen says:
Zuch: “The problem is that #1 hasn’t been shown, only alleged”
No, its been proven. The emails reveal the CRU Cabal conspiring to dampen warming spikes [Medieval Warm Period] that undermine their presentation of AGW trends to the IPCC. Fraud.
Zuch: “Huh? How has the CRU data been invalidated? Even if the allegations above are assumed true, that doesn’t make the data “invalid”.
All “evidence” of AGW can be now explained away as man-made adjustments to the raw data. Adjustments that rely entirely on the selective and biased judgement of the Climatology Cabal, as exemplified in the leaked emails.
Zuch, you’re not a very intelligent troll. You hold the IT guy to a higher standard than the scientists, without realizing that in doing so you undermine your own logic: See, its obvious to any reasonable layman that the CRU Cabal did not adhere to scientific practices.
December 18, 2009, 8:03 amThis makes them incompetent and, in your own words: “Don’t bother citing these incompetents again.”
Fen says:
http://wattsupwiththat.com/2009/12/08/the-smoking-gun-at-darwin-zero/
Pay particular attention to Figure 8.
December 18, 2009, 8:09 amNobody At All says:
A distribution of 6,533 adjustments made to GHCN raw data can be found here, if you are interested in data from more than a single station. Data points which failed quality control can be found here, under files named v2.m*.failed.qc.
December 18, 2009, 8:26 amNobody At All says:
Not to harp on an old subject, but by refusing to open comments upon his reproduction of the wattsup Darwin post, Lindgren effectively insulated it from criticism on Volokh.com. Links to such criticism (such as the Economist; further links are collected in the initial, “Skepticism’s Limits” post) are now buried in comment #140 to a post on the relationship between science and public policy, rather than in proper context. Not an especially efficient way to run a blog.
December 18, 2009, 9:09 amzuch says:
Nope. Only in the high-volume blatherings of RW anti-AGW partisans. FWIW, the MWP was included in the Mann article, reviewed by the NRC panel, and has been taken into account. Claims that it got “edited” out are simply false. That is to say, propaganda. Lies. Whatever you want to call it, however you cut it, dishonest.
All evidence of AGW “can now be explained away”? Fine. Go do it. Show your work. Publish. Until then, it’s just unsupported allegations, and we’re free to ignore them.
No. Reviews by a NRC panel previously upheld the conclusions. And a recent review by the AP with the help of scientists said that nothing particularly untoward was happening. As did the AAAS [disclosure: I'm a long-time member].
And where am I holding the IT guy to a higher standard? I just pointed out that he’s not the sharpest knife in the drawer, but that doesn’t make him incompetent. It may explain some of his carping, though. Your (that is, anti-AGW ideologues’) “explanation” for his carping is less convincing.
Cheers,
December 18, 2009, 9:31 amzuch says:
I thought I said that you should desist from citing “WattsUpWithThat” (and explained why). Low quality control standards over there.
Cheers,
December 18, 2009, 9:36 amDuracomm says:
Zuch said,
Maybe not in the academic world, but out here in the real world (where there are consequences for screwing up data) it is considered to be critically important job.
We are using data generated by the code to make decisions that have massive real world impact.
You continue to argue the guy who was in charge of maintaining the data generating code and making sure it worked correctly is not competent.
You are confirming that the data from CRU can’t be trusted.
December 18, 2009, 10:11 amMark Buehner says:
Check the comments on that link- its a bad attempt at slight of hand. He shows that the adjustments average out to about zero (what you might expect). But he doesn’t take the time scale into account- ie, if you add 1 degree for every year after 1950 and subtract 1 for every year before, you will get a manufactured warming trend with a net adjustment of zero (since they cancel out.
In other words, reducing temps in the past is equivalent to raising them in the last twenty years. Either serves to produce a warming trend.
December 18, 2009, 10:27 amrobotech master says:
Yes its called hyperbole… and the models of course are wrong… which you simply skipped over arguing… but hey all the models predict warming and yet were cooling thus the models are clearly WRONG.
Those temps have been adjusted who know how many times and they change on a near yearly basis do to the “adjusting” programs they use… it is worthless info.
Those adjustments are bogus as bogus can get. They are based on already adjusted and selected data thats been “approved” by the cultists. The “raw data” has already been adjusted…second the data is cherry picked to no end. First until they list station by station adjusted and the reasons for these adjustments the argument presented in the link are worthless. In fact one can argue the fact that many of the stations used are near urban heat bubbles thus you should see a larger amount of data being adjusted down not at all. The graphic presented has no time scale on it and I assume they are using temps “created” that go back to 1850s or something along that line… which of course would mean that the vast majority would have no adjustment because they were created based off proxy…
December 18, 2009, 10:31 amrobotech master says:
To Mark Buehner
Good call I didn’t even look at the comments but yeah the argument has been debunked
with link to http://statpad.wordpress.com/2009/12/12/ghcn-and-adjustment-trends/
Which shows lots of fun stuff taking place under the hood.
December 18, 2009, 10:40 amNobody At All says:
This was addressed in Note 2 – he is discussing the the trend of adjustments, not the average of adjustments.
December 18, 2009, 10:54 amDotar Sojat says:
I almost find myself thinking that AGW ought to be incontrovertible because poor Zuch seems to need it so badly.
December 18, 2009, 11:12 amElliot says:
Then that’s a big problem. If he’s not important, it’s reasonable to ask if they considered properly working code to be important.
December 18, 2009, 11:33 amFen says:
Zuch: The problem is that #1 hasn’t been shown, only alleged
Fen: No, its been proven. The emails reveal the CRU Cabal conspiring to dampen warming spikes [Medieval Warm Period] that undermine their presentation of AGW trends to the IPCC. Fraud.
Zuch: Only in the high-volume blatherings of RW anti-AGW partisans
No. Not unless the CRU Cabal is part of your conspiracy theory. Wacko. Do your research.
Or not. Zuch, just continue with your denial. We already suspected that the AGW alarmists would swallow anything. Now we’re learning they have no shame.
And no, I don’t want to buy your carbon credits.
December 18, 2009, 11:45 amFen says:
This reminds me so much of the Clinton Sexual Discrimination Scandal. The day after Clinton waved his finger at us and said “I did not have sexual relations with that woman…”
Remember how many people insisted he was innocent?
And afterwards, how quickly they all parroted “its just about sex!”
But this is better. The AGW alarmists and CRU in particular have been sliming skeptics for years, even affecting the careers of real scientists who voiced concerns about the theory. So this is delicious. I hope they face legal action, or at least a good tar and feathering.
Because if there IS a warming problem, this just set our solution window back at a decade.
December 18, 2009, 11:58 amFen says:
From: David Axelrod
To: Information Warfare Trolls
Subj: Immediate Action – HealthCare
Drop AGW/CRU.
New talking points for HealthCare in progress.
Require immeditate lobby of media and information brokers.
Overtime approved! Except you Zuch, contract terminated.
Yes We Can!
December 18, 2009, 12:15 pmRichard Aubrey says:
I dunno.
December 18, 2009, 12:22 pmI’d pay zuch double. He has a major talent to insist on the most obvious falsehoods, and, this is the important part, keep others engaged in the nonsensical argument when they could be doing something useful.
Mark Buehner says:
Which still ignored temporal components. The argument is that the intent is to ‘condense’ the temperature record so that it seems more stable and static (until recently) than it really is. By making the 30′s and 40′s cooler, and the late 1800s warmer, you end up with a far flatter graph that ticks up in the 90s(like a hockey stick). Visually, that is certainly more startling than a sine wave, and the graph he produces is exactly what you would expect to see if somebody were flattening the curve.
There may not be any nefarious intent to this, it could well be an underlying assumption that temperatures over the decades should be fairly linear, and homogenization tends to do that, but that isn’t necessarily reality. This link shows you how much temperature is adjusted over time.
December 18, 2009, 12:28 pmSagar says:
@ Zuch,
if you are not “saying that AGW is a fact”, i have no further argument with you, about whether the climate “scientists” are frauds or not.
my point was the burden of proof was on the climate “scientists” to show that we need to spend all the money on fighting “climate change”, and they have not done that.
December 18, 2009, 12:45 pmNobody At All says:
His analysis *accounts* for both the possibility of compensating trends (I would point you to his comment in the thread started December 13th, 2009 at 11:51, for example, to get a feel for what he is doing); and time (see, e.g. the thread started by Carrot Eater at December 13th, 2009 at 09:43).
Neither I, you, nor anyone else on this thread is going to benefit from rehashing that post’s comment threads here, so I’ll leave it at that.
December 18, 2009, 12:53 pmGreg F says:
His analysis is incomplete and misleading. As Mark Buehner points out it ignores the time element. It also ignores the fact that all the stations are not present over the time period. IOW, his data set is not stable over time.
December 18, 2009, 5:20 pmHarry Eagar says:
Nobody at all comments: ‘Links to such criticism (such as the Economist)’
What say you, then, about the BOM climatologist’s statement to the Economist that the 2 Darwin adjustments (0.6 C and 1.0 C) are ‘within the margin of error’?
The Economist editorialist does not understand the meaning of that, and I sure would like to know if that statement applies only to Darwin or to his opinion of the precision of 2 different sets of adjustment procedures generally.
It might be kinda important, since if it is a general situation, that takes all of one continent and part of another out of the so-called global network, as far as using the temperature series for anything.
Note to zuch: I ain’t quoting watts. I’m quoting the guys attacking Eschenbach’s report at wattsupwith that. To my mind, the defense has done more damage to the credibility of the numbers than Eschenbach could have.
December 18, 2009, 11:45 pmNobody At All says:
I’m not an expert, but it sounds like he’s describing what NCDC might call a class 2 surface station. Not a “general situation… that takes all of one continent and part of another out of the so-called global network.”
December 19, 2009, 12:41 amGreg F says:
The premis he bases his conclusion on is wrong. From the cited blog.
This would be true if, and only if, all the stations had measurements for the entire time period. This condition is not true of the surface station record.
A simple proof is assume there is only 2 stations, stations A and station B. To keep it simple assume both stations have an unadjusted trend of 0. For station A we adjust with a trend of +1 degree/decade and station B -1 degree/decade. Assume that Station A has 100 years of data and Station B has 50. In spite of the fact that the average adjustment of the trend is 0 combining the two stations would result in a overall positive trend.
December 19, 2009, 11:49 amNobody At All says:
No, this is not how his analysis works. Please, see comment threads referenced above. In short, a trend of adjustments is calculated for each data source, calculated for the time that the data source is available. In the example you give, Station A is not combined with Station B to give an “overall trend.” Two independent trends are reported: the trend of Station A, and the trend of Station B.
December 19, 2009, 1:04 pmNobody At All says:
I’ll also reference & excerpt commenter Tom Hinton’s thread (December 16th, 2009 at 04:58), if that’s helpful:
The explanatory image is here.
As Carrot Eater points out (in an earlier comment, I think), for this quasi-normal distribution of trend of adjustments to hide a combination of trends which, in total, create an overall warming trend (for example, by hiding inter-interval u-shapes that would result in a net slope of 0), that same combination of adjustments would need to result in an (adjusted) temperature curve that is u-shaped. (For example, adjusted temperatures in 1850 would need to be the same as the adjusted temperature in 2009.)
This isn’t what the adjusted temperatures report: we don’t hear about the “u-stick,” but the “hockey-stick.”
December 19, 2009, 2:26 pmGreg F says:
Which is what I did with station A and Station B. The trends were +1/decade for station A and -1/decade for station B. That makes the average “trend of adjustments” zero.
I reported 2 independant trends. What you don’t seem to understand is his failure to report the overall trend is the flaw in his argument. I reported the overall trend to show that flaw. The temperature data is, in the end, combined together just like I did with station A and station B.
Carrot Eater’s assertions are also flawed. Adjustments could clearly create shapes other then a U shape. It all depends on where each stations data is in time, the trend adjustment, and the length of each record. Don’t believe me? Calculate the adjusted values for each year for Station A and Station B based on the trend. Put the adjusted station B values at beginning of the record of the adjusted Station A values. Average the two values from each station for the 100 year record. If you plot them you will find no trend for the first 50 years (the 2 trends cancel) followed by an upward trend for the last 50. Hardly qualifies as a U shape now does it?
December 19, 2009, 4:54 pmzuch says:
I said “WTF cares?” The RW is a bunch of hypocritical ["Lashes" Livingston, "Knobby" Newt, "Homewrecker" Hyde, "Climber" Chenowth, "Boobies" Barr, all that just at that time, and it goes on and on and on: Vitter, Foley, Ensign, Sanford] panty-sniffers in high dudgeon that anyone has any sex. It ain’t their business. If they think it’s naughty, they should stop doing it.
Cheers,
December 19, 2009, 5:07 pmzuch says:
Yeah, and I assume that g = 9.8 m/sec^2 and that ice melts at 0°C (at least to first order of approximation). You know, some of the argon data has been contaminated and shown to be completely unbelievable. I guess we go back to H[0]; the earth’s 6000 years old….
Cheers,
December 19, 2009, 5:11 pmzuch says:
Oh, yes, obviously false. Because you simply said so. Hard to argue with that … because that’s not a legitimate argument.
Cheers,
December 19, 2009, 5:14 pmrobotech master says:
To Nobody At All
The problem with that argument is that “adjustments” are only 1 of about 6 ways they adjust the data… this graphic only covered 1 type of adjustment. So once again the graph is meaningless.
December 19, 2009, 5:24 pmNobody At All says:
Greg F:
Testing whether data adjustment is responsible for global warming, we agree that where each datapoint represents the slope of station adjustment over time, then:
1. If each datapoint extend for the entire reference interval (e.g. 150 years), this will be represented as a histograph centered to the right of zero. We do not see this.
2. If there is an inter-interval u-shaped data adjustment that creates a 0 slope, yet increases present-day temperature relative to past temperatures, such an aggregation of slopes would require that the adjusted data look have a u-shape (rather than, e.g. a hockey-stick shape). We do not see this.
If the objection is that:
3. There are two sets stations (for ease of use, we’ll call them “old” stations and “new” stations); the old stations do not extend into the present, and the new stations do not extend into the past. The slope of the old stations is negative (to the left of 0 on the histograph), and the slope of the new stations is positive (to the right of 0 on the histograph). The histograph is centered on zero; nonetheless, the adjusted temperatures demonstrating warming (the hockey stick) is caused by data adjustment.
One way to determine whether the adjusted warming trend can only be explained by two types of station data (old and new – the first of which has a negative slope and the latter of which has a positive slope) is to determine whether the adjusted trend (the hockey stick) is well-correlated with data from stations which are both old *and* new (i.e. have data available for the entire period).
As you probably noticed, the post links to precisely this type of experiment (warning: RealClimate post), comparing the adjusted CRUTEM3 data to raw data from World Monthly Surface Station Climatology.
I will excerpt the description, but the entire post is worth reading (and has already been linked to in these comments):
If the follow-up objection is that these 100-year records have been adjusted down in earlier years, and adjusted up in subsequent years, we now have good reason to believe that this is unlikely: see #1. However, the skeptical can confirm this by running the above GHCN analysis for the Sets A and B from World Monthly Surface Station Climatology.
December 19, 2009, 6:41 pmLaura(southernxyl) says:
I see the linked article about the AP review. See the first name listed on the byline?
Then look here:
So, yeah, a very objective investigation was obviously done.
December 19, 2009, 10:28 pmGreg F says:
I will not let you move the goal posts. I used a simple example to show that GG’s assertion is wrong. For reference his assertion is:
So instead of either admitting I was right or refuting my argument you instead move the goal post to some tripe at Real Climate. Let me know when Real Climate posts the sites they selected at “random”. Good luck getting them to do that.
December 20, 2009, 4:18 pmNobody At All says:
Your argument consisted of an untested hypothesis. I showed you a way to test it. Let’s not get personal.
December 20, 2009, 4:56 pmNobody at All says:
The goal post, by the way:
“[T]hat adjustment do interfere with readings and if they are biased toward one direction they may actually create a warming that doesn’t actually exist: either by accident or as result of fraud.”
(Top of the post, second paragraph)
December 20, 2009, 5:54 pmGreg F says:
Nobody at All,
I used a simple example to show that GG’s analysis, which you were presenting as proof, is wrong mathematically. Thanks for playing.
December 20, 2009, 6:45 pmNobody At All says:
Greg F:
Your untested hypothesis assumed a dataset that you nowhere illustrated, much less referenced. I showed you a way that you could test the truth of your hypothesis.
Actions speak louder than words.
December 20, 2009, 7:03 pmGreg F says:
Nobody at All,
It is not a hypothesis, it is a mathematical fact. The only claim I made was:
I provided a simple mathematical proof the above was true and GG’s assertion was false.
The data set is obviously the one GG referenced (GHCN dataset) since that is what he used for his argument. The following show the station count over time for the GHCN station record from 3 different people:
http://chiefio.wordpress.com/2009/11/03/ghcn-the-global-analysis/>1
December 20, 2009, 8:29 pmhttp://savecapitalism.wordpress.com/2009/12/02/ghcn-database-adjustments/>2
http://statpad.wordpress.com/2009/12/12/ghcn-and-adjustment-trends/>3
Greg F says:
Messed up the links.
December 20, 2009, 8:37 pmExample 1
Example 2
Example 3
Nobody At All says:
Greg F:
Thank you for the links.
Starting with the 3rd:
* “Annual Averages For Individual Stations” (approx. 1850-present): the annual average adjustment made to each station appears to cluster around zero. Moreover, the football/spade-shape is not tilted upwards, but is even. (Just eyeballing it.)
* “Mean Annual GHCN Adjustment” shows the *average* of each *average* adjustment, for each year.
– Because this is an average, not a median, outlying downward adjustments are able to skew the result in a way that does not reflect the adjusted trend. For example, it appears from this figure that the years 1900-1925 are subject to extremely large downward adjustments. If this is true, then in the chart entitled “GHCN Average Adjustments (per year for all series)” in your second link ought to show a large divergence between the red (adjusted) and green/blue (unadjusted) lines. But (to my eyeball), it doesn’t – there is no large divergence from 1900-1925. Nor, for that matter, does the trend of red vs. blue/green divergences (to my eyeball) the “average of averages” in the third link: according to this, the average adjustment became steadily and linearly less and less between approx. 1950 and 1990. But the gap between red and blue/green doesn’t become smaller and smaller.
– In short, I am not sure what information the “average of averages” is supposed to impart. Looking at the slope of adjustments seems methodologically clear.
A little more on the second link: While this does indeed appear somewhat U-shaped, hypothesis #2 (non-warming trends show warming because of a first adjustment down, then a second adjustment up, resulting in a slope of 0) would require that the raw temperatures stay somewhat flat while the adjusted temperatures do a U-shape. Here, eyeballing it, it looks like the u-trend of the adjusted temperatures tracks a u-trend of the raw temperatures. So this isn’t the dataset that hypothesis #2 is looking for.
The first link:
Honestly, I didn’t read the whole thing. Yes, a spatial analysis would definitely solve the debate over whether adjustments to individual locations are skewing the trend. Too bad he didn’t do that.
I am not up enough on the literature to know the extent to which this spatial analysis has been completed; “Carrot Eater” cites one paper – Peterson and Easterling, “The effect of artificial discontinuities on recent trends in minimum and maximum temperatures”, Atmospheric Research 37 (1995) 19-26. It is behind a paywall when I google it. He summarizes it as conducting a spatial analysis for the entire northern hemisphere, finding small effects on maximum temperatures, no effect for minimum temperatures.
Without this spatial analysis, however, we can still eliminate hypotheses. #3 is easy, and the data is readily available.
December 20, 2009, 11:06 pmRobuk says:
And who is to say 500, 800 or even 1200 PPM of CO2 is not the ideal level.
January 9, 2010, 12:01 pmAnthony Cadden says:
thanks for this Info. I often read here…
March 3, 2010, 3:53 pm