Here’s what Crook writes:
I admire expertise, and scientific expertise especially; like any intelligent citizen I am willing to defer to it. But that puts a great obligation on science. The people whose instinct is to respect and admire science should be the ones most disturbed by these revelations. The scientists have let them down …. That is outrageous.Megan McArdle adopts a world-weary tone similar to The Economist’s: this is how science is done in the real world. If I were a scientist, I would resent that. She has criticised the emails and the IPCC response to them, then says she still believes the consensus view on climate change. Well, that was my position at the end of last week, and I suppose it still is. But how do I defend it? There is far more of a problem here for the consensus view than Megan and ordinarily reliable commentators like The Economist acknowledge. I am not a climate scientist. In the end I have to trust the experts. That is what we are asked to do. “Trust us, we’re scientists”.
Remember that this is not an academic exercise. We contemplate outlays of trillions of dollars to fix this supposed problem. Can I read these emails and feel that the scientists involved deserve to be trusted? No, I cannot. These people are willing to subvert the very methods — notably, peer review — that underwrite the integrity of their discipline….
fsfsfsfsfsfsfsfs says:
But the climate scientists were not experts either.
That is, a large portion of “climate science”, possibly the majority, entails the statistical manipulation and modeling of large, complex data sets over long periods of time using complex custom software.
This is a classic problem in software engineering. Yet none of these self-proclaimed “climate scientists” had any experience in software engineering. At least, I infer this from their code.
By trying to manipulate these incredibly disparate, heterogenous datasets using software they hacked together with glue and wishful thinking, they were simply going far, far, far outside their area of expertise. By representing that their data was pristine, that their completely untestable software worked, they were making representations that they had no expertise to make.
Interestingly, the IPCC itself is full of economists, political scientists, and the like – are there any computer scientists there?
Be that as it may, to have any kind of confidence in this mess, all of this junk will have to be rearchitected from first principles. You’ll need a full DBMS, a way to keep track of the provenance of every bit of data, clear and clearly documented code, rigorous modular code checks, a suite of unit tests and robustness checks. You’ll need teams of software engineers, database architects and statistical modelers to check all this.
Anyway, what a fascinating story. You’ve got all in one place:
(1) An interesting scientific question: is there significant, dangerous anthropogenic warming that would be ameliorated by decreasing carbon emissions?
(2) An interesting meta-scientific question: how did the CRU work pass peer review and get so widely cited?
(3) Important general questions on how to insulate science from policy, so that science is funded on its merit not on its accord with pre-defined policy goals;
(4) Fascinating case study in media reactions, in which most of the major media outlets are completely stonewalling any coverage at all of a major story; those that do cover it, do so minimally and dismissively, with coverage driven by blogs – it’s like Rathergate^100;
(5) Political ramifications yet to be determined, possibly negligible, possibly significant;
(6) Human interest galore; and
(7) Fascinating sociological case study: how large numbers of people can be persuaded to adhere strongly to beliefs they know little about.
I think this is the most entertaining Volokh thread since syllepsis!
November 30, 2009, 7:01 pmwb says:
EV,
I fully agree. I would be mailing the Provost of the University about an ethics issue if this were an instance in my journal.
November 30, 2009, 7:11 pmWoodland Critter says:
The climate researchers faced one of the most difficult challenges imaginable. Trying to model a non-linear system(s) with an unknown number of variables using data outside their control. These factors, in an of themselves, should frighten any knowledgeable person trying to make predictions with the model. In comparison, quantum physics is relatively straight forward. All of the above should have resulted in highly skeptical fellow scientist tearing the model apart and challenging every aspect.
Reproducibility should have been critical to accepting the predictions of the CRU. Instead, it does not appear that there was any skeptical challenge to the research and now, there is little likelihood of such testing.
As a result, there is no scientific method behind the CRU results and consequently no science. All work based on the CRU work needs to be removed from the literature until such time as it is reproduced. Note that the above analysis disregards other issues like the statistical competence of the climate researchers and allegations of bias.
In addition, the hypotheses of all papers that have been influenced by the CRU work need to be revisited based on bias introduced by the CRU non-science.
Climate scientists the world over should be reacting in horror at what was revealed in the emails, disavow the work to date and begin immediately to rebuild their data with scrupulous integrity.
November 30, 2009, 7:30 pmepignosis says:
It would be better to put your faith in the message of the disheveled guy on the corner with the sign that says “repent, the end is near”. (period outside the quotes?)
AGW proponents are, in effect, yelling “fire” on a crowded planet. That the resulting stampede will unfold more slowly in no way diminishes the potential for social and economic injury. Therefore, having a little more time at our disposal, would we not want to examine carefully the claims of those who opine that it is just a reflection from a red metallic candy wrapper?
Buckland says it clearly:
They thought they were doing God’s work and any small lapses were made up for by the many things that they were doing right.
Add a new entry to the Wikipedia article on “cognitive bias”. NASA has had some rather unpleasant experience resulting from very smart and talented people who have succumbed to cognitive bias (Challenger & Columbia accidents). One expression of that bias is when a group of scientist adhere to a belief so strongly that they are willing to ignore or rationalize all evidence to the contrary. Some refer to this as groupthink, though that does injustice to the scope of the danger in this particular case.
What permeates climate science regarding anthropogenic global warming (AGW) is properly called crusader arrogance. “I am intent on saving the planet, and I am one of the few people smart enough to know what is necessary, everyone who disagrees with me is not only wrong, but is a threat to humanity.” These true believers have forfeited intellectual honesty. Data has been manipulated, opponents have been ridiculed, castigated, impugned, and silenced. Contrary information has been suppressed. Talented scientists with integrity have lost their jobs.
Before trillions of dollars are sent to other countries in the hope that such a sacrifice will surely satiate the climate gods, a real debate needs to occur. The Gore mantra that “the debate is over, the science is settled” is ludicrous. (Who comprised the panel of impartial, learned arbiters for that alleged debate?)
To put this in perspective, water vapor is the major contributor to warming by atmospheric gases, 95%. Carbon dioxide, all sources, is 5%. CO2 from human activities is about 3.6% of the 5% for a total contribution of 0.18%.
They would have us believe that this tiny sliver has tipped the global balance irreversably based on models that cannot even account for the present cooling trend or the Medieval Warm Period.
November 30, 2009, 7:40 pmStruthius says:
Nothing to add — except that I agree completely with fs(etc)and Critter. Especially fs.
November 30, 2009, 7:42 pmdave h says:
As a scientist (postdoc in biophysics) I definitely disagree that I should ever be able to say “it’s messy, but trust me, I’m an expert”. There are some real-world details that I know from working in science that others probably don’t know. For instance, how much data you furnish to a journal, either in the paper itself, as supplementary material, or as a response to reviewer’s requests, or how important reproducibility is. However, it’s an important skill (and one many scientists, myself included, don’t possess a great deal of) to be able to distill our work in an understandable way for a lay audience. If at first I give a cursory overview, and someone asks a question about the details, I should be able to give a satisfactory explanation, which itself may prompt more questions which I should also be able to answer. At no point should I be forced to invoke my authority as an expert.
From that perspective, there are aspects of this saga that I agree are just how scientists work and talk (calling something a “trick” is not nefarious). However there are aspects that are very troublesome, and which would not be tolerated in my field (which is admittedly very quantitative – I’d guess out of necessity it’s okay to be messier as fields get less and less quantititave).
November 30, 2009, 7:43 pmrhhardin says:
The integrity of science comes from curiosity, not peer review or any other process.
Curiosity is notably absent in global warming.
November 30, 2009, 7:46 pmBruce Hayden says:
You make a good case that teams such as this should have software engineers as major contributors. But is it going to happen in the near future? Don’t hold your breath.
I would suggest that there are a lot of parts to this. One is that these scientists have no idea what they don’t know. They think that the FORTRAN class they took so long ago taught them to program. This is like someone who took one semester of freshman physics thinking that he is a climate expert. So, they either try to do the work themselves, never having written more than 100 line programs before, or they get someone around school to do the work. This may mean an undergraduate, a grad student in CS, or maybe even in their own department (who may have taken a Java class instead of a FORTRAN class). In any case, I would doubt that the people brought in for the programming are considered full members of the team, but rather closer to the secretaries in merit and status. And, they got the software that matches the importance they placed on it, and those who developed it – garbage.
This isn’t, of course, the only place that has this sort of status problem that scares off the best and brightest. Maybe 35 years ago, I worked at a hospital as a programmer, and was so far down in status that it was hard to come to work. MDs were gods, and could make or break you just for looking at them wrong. And everyone else there had status in direct correspondence with how close they were to the MDs and how many medical type degrees they had. I quickly jumped to an agency that respected software engineers (Census), and then to a computer company.
It doesn’t have to be this way. A classmate from college has spent his working career developing software at a university for medical devices, etc. He has been a valued member of the teams he has worked with, and has been listed as a contributor on any number of papers over the years. But I will suggest that his case is the exception.
That is the way it should be though – given the importance of software in these projects, the software engineers working on them should be treated as full team members, given status commensurate with their contributions, including being named as contributors to papers.
Wishful thinking, I know, which is one reason I became an attorney.
November 30, 2009, 7:47 pmtherut says:
Reminds me of all the times Senator Schumer said “settled law” in regards to the now non-existing collective 2nd amendment. You know it was “settled law” in his mind and others. Will he ever admit his arrogance. No and neither will the environmentalists cause they become extremists or fundamental enviromental believers.
November 30, 2009, 7:49 pmkdackson says:
Yeah, but that would require actual work.
November 30, 2009, 7:50 pmepignosis says:
Regarding the software issue, this may be a distraction. If the code can be reviewed, and accomplishes the task correctly – does what it was intended to do, then the scientist is satisfied.
Is the fact that the code is akwardly constructed or difficult to maintain, issues that concern people with more advanced programming skills, necessarily germane to the issue here?
November 30, 2009, 7:55 pmBruce Hayden says:
Let me suggest that this was fairly quantitative, and should have been treated accordingly. One of the possible criticisms is that in their desire to get results, they failed to implement the sorts of quantitative controls that they should have. And, as a result, we now see a lot of massaged, fudged, and forced data, without really enough documentation to determine whether or not that massaging, fudging, and forcing was justified and done accurately.
November 30, 2009, 7:57 pmh2u says:
Lack of clarity in the code was hardly the only problem…
November 30, 2009, 8:00 pmJones' Cell Mate says:
Yeah, but that would require actual work.
Worse than that, it would have required a couple of people to admit that there are large gaps in their knowledge-base.
It’s always strange how certain “intelligent persons” struggle mightily with that precise issue.
November 30, 2009, 8:02 pmh2u says:
Check out a pretty detailed analysis of the code here and here.
November 30, 2009, 8:05 pmD.R.M. says:
Climategate is a good thing for climate science.
First, it will result in people demanding the raw data and the code for all climatology claims. Second, it will draw the interest of people to the subject, and thus more complete and intensive review of the data and analysis methods, finding errors and improving quality, improving our actual understanding of the climate.
In the meantime, of course, it will seriously undercut political efforts related to carbon control, since it raises uncertainty in the short term. But that’s fine, right? Climatologists are scientists seeking the truth, not activists looking to impose policy, right?
November 30, 2009, 8:05 pmDavid Schwartz says:
epignosis: We already know that is not possible. The code processed data that no longer exists, so the process cannot be replicated. And the code relied on certain things being correct in the environment that it is no longer possible to determine were correct.
To give a trivial analogy, suppose we had discovered that the code would give incorrect results if it was ever run on a Thursday between 7PM and 9PM. If the information about when the code was run is lost, and we cannot run it again, we know that we cannot trust the results — period.
There are four major classes of such bugs known definitively to be in the code.
1) Use of static data or static places: This means the code sometimes stores things in a particular place, but always the same place. If two instances of the code are ever running at the same time, they will step on each other, producing incorrect output. This is the least serious, as someone could simply assure us that they never ran two instances at once. We’d have to trust them on this, but under normal circumstances, that wouldn’t be a disaster.
2) Failure to test for error conditions. In many places, the code fails to test for obviously insane error conditions but just goes on processing. That means that if the input is somehow fundamentally broken, the output may be subtly broken. Again, we don’t have the input to retest.
3) Reliance on the user to select the correct input sets. The code relies on the user to tell it what data to process and doesn’t make sure the data is correct, it must trust the user. CRU had data sets with identical names but fundamentally different data. There’s no way now to be sure uncorrected data wasn’t used where corrected data was appropriate or that data wasn’t corrected twice.
4) Reliance on the user to select the correct run-time options. During the run, many of the programs relied on the user to select the correct options as the run progressed. The options were not embedded in the results. A single mis-key could cause the output to be invalid.
Unfortunately, these types of defects combine in a multiplicative way. A mis-key during a run could result in subtly bad input that could cause an error condition that’s not detected resulting in radically bad output that’s not detected because of lack of input validation …
These sound like one-in-a-million bugs, but we’re talking about machines to do billions of operations. So it’s like a one-in-a-million guy — in China.
November 30, 2009, 8:06 pmBruce Hayden says:
To the extent that we are expected to trust the output of the programs, yes. Right now, we have little reason to believe that the code does what it is supposed to, and a fair amount of reason to believe that it doesn’t.
Poorly written, awkwardly constructed, and difficult to maintain code almost always means that it doesn’t do what it is supposed to. And, even if it does, it is hard to verify that it does perform properly.
And keep in mind that it isn’t just that the code was awkwardly written, but rather, that parts of it were extremely poorly written. So badly written that the only reason that parts of it appear to have run at all is that error conditions were ignored. I could keep on, but there are others better qualified than I who have opined at length about the coding deficiencies.
November 30, 2009, 8:09 pmfsfsfsfsfsfsfsfs says:
The answer to your question is emphatically: YES.
There are two distinct reasons for this.
The first reason is argument from authority (or experience). The specific problem of managing very complex, continually changing software with very complex data from heterogeneous sources is one of software engineering. So you should ask a software engineer to architect the problem, because software engineers have many years of experience in dealing with exactly this kind of problem (and answering exactly this kind of question, and understanding exactly what will happen and why when you try to grow a solution organically.). Every software engineer I know sees or has seen software like this, has heard the “if it works it’s fine” excuse, usually after the software doesn’t work. And by the way, losing the data is “not working” for scientific software – data management is a key part of software engineering.
The second reason the answer is “YES” lies in your words “what [the software] is intended to do.” Except for trivial programs, we simply do not know if software actually does what it is intended to, unless we have certain software engineering protocols of the kind I named. In this specific instance of climate change, we do not know what it is intended to because (a) the software is long-lived, so the first author may not remember what he thought it did; (b) the software is worked on by many different people, not all of whom know what it is intended to do; (c) the software must constantly change to adapt to new data formats, and we are not sure this does not break something else; and (d) there is no way actually to verify the software does what it is intended to do.
Basically, the combination of (a) through (d) means you have to rearchitect it. And the fact that many climate scientists do not understand this, is precisely why we should not accept their claims of expertise – they think they’re expert software engineers, but they’re not. And when there are problems (losing data, unreproducibility) they blame aleatory incidents rather than poor software architecture, even though such problems are utterly predictable (even if their detailed manifestation is aleatory) from the software architecture process used to develop the software.
November 30, 2009, 8:10 pmkdackson says:
Epignosis is correct.
For those of you who care, all branches of science and engineering rely on the grad students responsible for the research to do their own programming. They typically do not have the resources to construct “class A, easy to decipher” code. So they do the best they can. Of course they need to do manual calcs to verify that the basic flow and logic are correct.
Yeah, too bad we can’t all get our degrees in Comp Sci. But when I wrote code, I tried to make it as clear as I could because when the inevitable problem cropped up, I was the one who had to debug it.
And as for the “dissing” of FORTRAN, sure it’s not the best language out there, but it does the job. And there are a whole pile of library codes written for common tasks that would be too onerous for us to reproduce (why reinvent the wheel?).
The fact is that the code is a scrambled mess, and that’s just too damn bad. But these people look at programming (in general) and the code (in particular) as a tool, NOT as a final product. Almost no one cares about the underlying code (I had to once to verify that a typo in my thesis and in a subsequent publication 5 years after the fact did not make it into the code – thankfully I still had it and could check it out and verified that yes, it was a typo in the thesis) as long as the result passes the sniff test.
But the result did not even have a “sniff test” to pass, because everyone was holding their nose because they wanted to believe.
November 30, 2009, 8:11 pmStephen M (ethesis) says:
Have you viewed the TED presentation?
http://www.newcoolthang.com/index.php/2009/11/global-warming-tedtalks-2/1610/
November 30, 2009, 8:27 pmRHSwan says:
For numerical programming I don’t think there are any better programming languages than the modern versions of FORTRAN. Most people are familiar with the 1977 version. There have been at least two if not three updates since then with many modern programming features.
November 30, 2009, 8:29 pmzuch says:
Prof. Volohk quotes Crook:
… wringing her hands in woe and weeping crocodile tears for the horrible injury performed to the reputation of science … that were never there for some reason for the decades (if not centuries; can you say “Galileo”?) long assault by conservatives on science.
More Crook:
… while other contemplate breathing through very long straws.
Cheers,
November 30, 2009, 8:36 pmkdackson says:
What about this one:
http://www.youtube.com/watch?v=UNRW5qMRnIU&feature=player_embedded
November 30, 2009, 8:36 pmh2u says:
Finally something the Stimulus Package might actually be good for…
November 30, 2009, 8:47 pmguy in the veal calf office says:
Here is a skeptical examination of the Global Climate Models (GCMs) that underly all of the IPCC projections & scenarios.
Here Ross McKitrick writes about his experience within the IPCC process.
November 30, 2009, 8:48 pmBruce Hayden says:
I agree here. While FORTRAN was not my first language, it was my first production language. Much of the 1980 Census was written in FORTRAN, and much of it was done well. It didn’t hurt that the critical editing code went through multiple code reviews and used an ALGOL-68 like preprocessor to impose decent structure on it. Much of the rest used the newly released FORTRAN-77 and its more block structured capabilities. Yes, it was still FORTRAN, but we did what we could to mimic the capabilities in more structured languages (for example, we forced explicit declaration of variables through IMPLICIT DOUBLE COMPLEX A-Z).
But, on the other hand, like many languages, FORTRAN could be, and often has been, abused. Anyone else have the privilege of trying to debug code with assigned goto’s? (Yes, I know, COBOL was worse for this, and almost anything else). When I was at Census, I remember one FORTRAN program that we got to debug. The GS-12 programmer had taken on-the-job retirement, and went out on vacation the week it was due to go into production, after having spent 6 months writing it. We spent a couple days trying to understand it, then two of us started from scratch, and got it operational with a day or two to spare. The original had 2000 line DO loops, jumped in and out of them, didn’t declare any variables, reused them without any rhyme or reason, and, yes, had multiple assigned goto’s. OH, and no comments.
November 30, 2009, 8:48 pmkdackson says:
Just goes to prove that GS-12′s were and are overpaid.
Back in the day when I was a TA in Engineering School, one of my first assignments was in the Frosh FORTRAN class. I pretty much failed anyone who did not at least attempt to use structured code (and subroutines when they learned what those were). If the comment lines did not equal about 1/3 of the total code, then it was a pretty good bet the grade was a “D”.
November 30, 2009, 8:56 pmbyomtov says:
OH, and no comments.
Comments are for children. :-)
November 30, 2009, 9:02 pmzuch says:
I don’t seem much there in the excerpted commentary by Harry other than griping about not knowing how to do (or find) things. That doesn’t mean the original programmer didn’t know; [s]he probably did, but in pretty much usual academic fashion, didn’t bother to write it up very well if at all.
FWIW, I’d say that even supposedly competent industrial programmmers who are following proper coding guidelines often suffer from a deficit far too common: The idea that if it’s clear to me when I write it, it ought to be clear to you when you read it. Rigorous comment policy simply moves the dilemma from the code to the comments, and while competent comments may clear up some issues, introduces another element of possible error in that the comments may not accurately reflect what the code does (for instance, I’ve never seen anyone put into the comments the substance of the bugs they have in their code).
Cheers,
November 30, 2009, 9:05 pmh2u says:
This link is more detailed. My favorite part: finding out whether stations may influence each other not by means of basic trig functions but rather through creating a virtual graphic and drawing circles on it and seeing if they overlap. Awesome.
November 30, 2009, 9:08 pmzuch says:
Jones’ Cell Mate:
This is an exceedingly strange comment for a blog with comment threads suffuse with people that insist that no one knows anything but who insist on pontificating at great length about the definite conclusions they’re come to about it. I do hope nobody takes offence.
Cheers,
November 30, 2009, 9:15 pmzuch says:
Does it work? Do you think that the trig is perhaps more accurate? I’m interested in where you have determined and shown that such influence follows the precision and definitivity of trig functions….
Agreed, there may have been more efficient ways of doing this [unless the mapping was done by coprocessors that actually offloaded the main CPUs], but that’s not the relevant question here.
Cheers,
November 30, 2009, 9:21 pmh2u says:
From the link…
And from the actual code comments:
So, no, the code doesn’t work. And using trigonometry isn’t only more accurate, it’s verifiable math as opposed to the programmatic-voodoo that was used instead.
November 30, 2009, 9:26 pmfda says:
Check out the following from NIPCC and SEPP which refute a substantial number of the IPCC claims and which are written by climatologists
http://www.sepp.org/publications/NIPCC_final.pdf
November 30, 2009, 9:27 pmhttp://www.NIPCCreport.org.
Bruce Hayden says:
One of the problems was that it almost always worked, and they needed it to always work.
November 30, 2009, 9:28 pmJagermeister says:
I’d like to highlight the issue fs raises of heterogeneous datasets. Way back in college, one of my term papers was a quantitative determination of the rotation of the galaxy by the analysis of red-shift data collected from published sources. This is settled science, requiring only a little math, and the physics is well understood. There is nothing magical about it. But because all that data came from so many different sources at so many different times it was impossible to tease a statistically meaningful result from the data.
The project was well within my knowledge and capabilities (math/CS major, engineering minor), but the results were underwhelming. Getting anything approaching a meaningful value required “tweaking” various datasets in various ways to attempt to correct for various observational and measurement errors, which required tracking which values came from which datasets, how they were collected, if they had any prior pre-processing, and so forth.
The point is that large volume analysis of real-world data is difficult at the best of times (when one has complete control over the collection process), and becomes almost impossible as the data becomes removed from the analyzer in both time and methodology (data collected years before, by different methods, in varying units, subject to arcane and obscure “corrections”).
This isn’t an argument to throw up one’s hands and give up the attempt, but is an illustration of the necessity to preserve original data, with complete annotation of all of the details of collection, and the methodology of analysis. What may be opaque to today’s researcher may be made transparent by a later consumer who may have a better key for extracting a valid “signal” from the observational “noise”, or the proper counter-method for an “artifact” introduced by the measurement process or equipment.
November 30, 2009, 9:28 pmzuch says:
No. That’s more a matter of algorithmic design. Such type statistical DP projects, even if complicated numerical processing problems, pale in terms of complexity to real-time applications, H/I applications with incorrect and sometimes malicious input, and non-synchronous type systems complete with possibly randomly missing or errored data, etc.
Cheers,
November 30, 2009, 9:28 pmzuch says:
You know this how?
Cheers,
November 30, 2009, 9:29 pmkdackson says:
Why cloud the debate with facts when our deliberate fabrications get us the results that will make people listen to us?
– Head, Department of MSU**
MSU [=] Making Shit Up
November 30, 2009, 9:31 pmChrisIowa says:
I have to say that although I don’t understand most of what he’s saying, whoever wrote “Harry_Read_Me” did an admirable job of keeping a log. I know a few programmers (and am married to one). I’m sure that to find such a detailed log of such a complex project is a rare event.
November 30, 2009, 9:42 pmJagermeister says:
I respectfully disagree. I’ve worked in real-time software for over 30 years, in everything from communications to weaponry to medical devices. The problem statements and solutions were under my control, or at least, amenable to my analysis. Having to analyze someone else’s data, for about which little information is known concerning acquisition, circumstances, retention, assumptions, etc., is akin to hell for those of us used to determinism. I feel intense compassion for “Harry” wading through all the CRU data, attempting to reverse engineer the data in order to get something meaningful out of it. I’m sure he would much rather have had something more definite to work with that a massive set of what almost amounts to pseudo-random numbers.
November 30, 2009, 9:42 pmzuch says:
Minor quibble here: If the program gives garbage output for “obviously insane” or “fundamentally broken” inputs, this may not be the fault of the program. If I’d spent my time doing input validity tests on all the possible inputs to subroutines I wrote, I wouldn’t get at real work done. At some time, you have to assume some good nature in others, and get on with the tasks at hand. After all, they’re using your code, you aren’t using their data….. Sounds cruel, but if you’ve worked in the field… If they pay me for taking care of their excessive stupidity, then again….
Not to mention, what do you care if the output is wrong for “fundamentally broken” input?
Cheers,
November 30, 2009, 9:44 pmkdackson says:
Sure, why care when there is an economy worth trillions just waiting to be destroyed on the say-so of a program that gives wrong results?
That is the fundamental issue here.
The output cannot be trusted, but we are basing decisions that will change to course of this country’s economy on that untrustworthy output.
Brilliant!!
November 30, 2009, 9:49 pmzuch says:
Wow. Just wow. You wouldn’t believe all those useless C programs that — each and every one of them — take all their input from a file descriptor called “stdin” and send all their output to file descriptors called “stdout” and “stderr”. Thanks for the heads-up, David. From now on, we should throw away all results ever derived from these poorly written and error-prone programs.
Cheers,
November 30, 2009, 9:50 pmzuch says:
Is this a serious argument?!?!? Just curious if I should be laughing, or writing yet another correction/clarification/smackdown….
Cheers,
November 30, 2009, 9:59 pmElliot says:
I’d say Harry has everything he needs for a best seller.
November 30, 2009, 10:00 pmzuch says:
Argument from incredulity/ignorantiam? Or is it “argument from adversity” (perhaps just a new logical fallacy basically summed up with “I’d rather not believe it”?
Cheers,
November 30, 2009, 10:07 pmzuch says:
Cheers,
November 30, 2009, 10:09 pmGordon Langston says:
Prof. Ward Farnsworth had a Chapter in his recent book, The Legal Analyst that dealt with cascades which relates closely to this incident.
“It creates incentives for people to deliberately suppress what they believe or know; Timur Kuran refers to this as “preferable falsification” and suggests that it is the stuff totalitarian societies are made of and on which they depend. This is the most important point to grasp about reputational cascades: their power to distort discourse and enforce a fake consensus.”
November 30, 2009, 10:09 pmzuch says:
Or people used double complex variables to count loops….
Cheers,
November 30, 2009, 10:17 pmOperationCounterstrike says:
Only one problem: McARdle is right–this IS how science works.
If you don’t like it, go become a scientist and try to do things differently.
November 30, 2009, 10:22 pmDavid Schwartz says:
I’m not sure if you’re being serious or not, but I’ll give you the benefit of the doubt. Your comparison shows that you utterly failed to understand what I’m saying.
A program specifically designed to operate on arbitrary data and perform a “representation-neutral” function (such as counting the lines in a file or searching for lines that contain a particular string) has no reason to validate its input. It is in no way comparable to a program that relies on its input to have a particular format. Programs that rely on specific formats do in fact check their input for validity. (And, for example, even ‘grep’ will alert you if you send it binary data that it makes little sense to search through for text strings.)
Also, your argument proves the opposite of what you think it proves. Remember, in this case, we don’t have the original data. If I said “I ran that analysis, piped it into a line counter, and got 40 lines containing the string of interest. Sorry, I lost the file I piped into the line counter and have no way to reproduce it”, you would be *right* to be skeptical. Likely I have a working line counter and likely I have a working search program, but how do you know that I piped the right file to it or correctly generated the output file that I piped into it? (Remember the manual intervention issues I mentioned?)
On the other hand, if I used a special line counter that validates that it is in fact operating on the right data, you can have a higher level of confidence that I piped the right file into the line counter.
And specialized counters do in fact have such checks. A specialized program that checks how many database records contain “foo” will generally check that it is in fact operating on a database of that type with the correct fields. If not, that lack will have to be compensated for elsewhere to avoid the cascading error scenarios I mentioned .
I am an expert on this commenting inside my area of expertise, and I think you will find other such experts will agree with the specific problem areas I mentioned. They really do make it impossible to “retroactively” validate the output without the input. Such a validation can never happen.
November 30, 2009, 10:25 pmzuch says:
Haven’t show me the code doesn’t work. I know you don’t like it (and neither do I), but that’s not the issue.
Once more a warning: Comments should be treated as what they are: Someone’s idea (or more charitably, hints) as to what’s happening. Treating them as definitive to what’s happening is a recipe for disaster.
Cheers,
November 30, 2009, 10:28 pmA Conservative Teacher says:
People are finally catching on that this whole ‘global warming’ thing was merely a way to jam fascism down the throats of the American people. The Democrats and Obama make us feel guilty for being successful and prosperous and then take away our liberties, freedom, and property. Global Warming is a Green Ghost that steals freedom from us in front of our very eyes.
November 30, 2009, 10:31 pmzuch says:
I had better ones. Think I have them saved around somewhere. And they didn’t consist mainly of “I can’t figure this shi*te out!”
Cheers,
November 30, 2009, 10:33 pmRyan Waxx says:
Nice system, Zuch.
A: The comments don’t count because they don’t necessarily reflect the actual code.
B: The code’s sloppiness, bad methods, lack of safeguards don’t prove the program doesn’t work.
C: So, you can only look to the results in Zuch’s world. And if the code spits out what it’s been massaged and altered to spit out? There’s proof it works!
Very, very convenient… for someone.
November 30, 2009, 10:34 pmzuch says:
You may have found such difficult but …. I think I stand by my previous statement.
Cheers,
November 30, 2009, 10:36 pmzuch says:
Fer shure. When Palin makes the charts…. ;-)
Cheers,
November 30, 2009, 10:37 pmjuris imprudent says:
The original had 2000 line DO loops, jumped in and out of them, didn’t declare any variables, reused them without any rhyme or reason, and, yes, had multiple assigned goto’s. OH, and no comments.
Reminds me of an infamous programmer who’s code I once had the misfortune of attempting to maintain. Someone else had already labeled him the Master of Baroque Code. I had my own choice names for him.
November 30, 2009, 10:38 pmzuch says:
Assuming your conclusion?
Cheers,
November 30, 2009, 10:39 pmzuch says:
On a more general note:
If the commentary here is any indication of AGW “scepticism” [read: "denial"], I’d say that the CRU people have a right to feel just a tad irritated that they’re being Swiftboated.
Cheers,
November 30, 2009, 10:41 pmJagermeister says:
Zuch: You’re trolling now. Have fun…..
November 30, 2009, 10:43 pmzuch says:
You have to do step one here: Show that the input is “fundamentally broken”. If you can show this, the quality of code is unimportant. If not, then this particular quibble about the code is immaterial. That was my point.
Cheers,
November 30, 2009, 10:45 pmzuch says:
If you were dishonest or incompetent. But using such a hypothetical to “prove” you were incompetent or dishonest would be rather circular then, eh?
Cheers,
November 30, 2009, 10:54 pmzuch says:
So you’re suggesting that the analysis programs (in the interest of “repeatability”) hard code all the options and the input streams into the code to be available for all posterity?!?!? And you’re a programmer? Remind m e not to ever hire you….
Cheers,
November 30, 2009, 10:58 pmClayton E. Cramer says:
I don’t know what sort of programming you do, but if you don’t verify the validity of your inputs, I pray that your code isn’t responsible for:
1. health equipment
2. public safety
3. money
Yes, there are definitely people who write code like you apparently do–but they aren’t serious programmers. They’re hackers.
I had some contract work recently that involved writing unit tests in C#. I found that lots of the code responsible for importing database records from an external source did no validity checking on the inputs. Yes, the vast majority of the time, the records being imported were probably okay. But assuming that an input dollar field is always numbers, or that a part number conforms to the specified criteria (alphanumeric, for example) is just laziness. And one of these days, failure to test for invalid inputs will mean that you will silently process invalid input records, producing garbage output.
Of course, you probably have a job, Zuch, producing low quality, unreliable software. I don’t at the moment. Maybe yours is the right method–as long as no one cares whether the code is actually producing valid results.
November 30, 2009, 10:59 pmClayton E. Cramer says:
That’s not what he said. It is a perfectly sensible action to verify that you have at least a plausible input file for the specified type. If you find that the input records don’t match the expected type, you throw an exception. At least, real software engineers do. If you are writing code for a video game, or to make decisions about the world economy, maybe it isn’t so important.
November 30, 2009, 11:01 pmMalvolio says:
The issue is, the code is too badly written to rely upon. There is really no other issue. Show you the code doesn’t work? First, show me the planet is getting warmer.
Wow, does that sound familiar.
November 30, 2009, 11:01 pmzuch says:
Maybe it ought to run checksums on the input data to verify that it’s operating on the specific data it was written for and previously run on. That’s a rock solid ‘program’ … and doubles duty as a boat anchor.
Cheers,
November 30, 2009, 11:03 pmClayton E. Cramer says:
You should be embarrassed to say this. Yes, you do sometimes find comments that don’t reflect the actual current state of the code. But this is not at all confidence inspiring. It suggests that someone is just hacking code as fast as they can, with no awareness that the results might have to be correct.
November 30, 2009, 11:04 pmsilvermine says:
FORTRAN is often still used in science. It is good for problems where you need a large number of significant figures. (Basically, really really big numbers or numbers with a lot of stuff after the decimal). Other languages are (or were, but are better now) prone to rounding just to keep the memory requirements smaller, or to keep the program running faster. Well, scientists would rather have all of the significant bits, even if it takes twice as long. ;) So FORTRAN it is.
However, in the cases that are being criticized, the programmer used FORTRAN to manipulate text, which is this day and age is silly. Lots of things manipulate text far better.
Anyway, the program is horrible, the data is horrible, and the scientists are horrible. I do *not* believe scientists act this way. They may wish they did, but most disciplines keep them mostly honest… This is just a case where the group of scientists was small enough, and willing to collude enough, that they could basically control all journal activities for an entire discipline of research.
(Note: Former biochemist here, who worked on a computer model. ;) I would never expect people to just believe what I said. I would find a way to explain it. If they cannot explain it, it’s because they don’t really understand it. Feynmann could have explained anything to anyone. But then, he really knew what he discovered. He *knew* it. And he knew that the climate researchers did not know what they were doing and were fooling themselves. There’s some great videos of him talking about it on Youtube. ;) And yeah, I programmed my own FORTRAN. ;D It did not look anything like the crap the climate researchers produced.)
November 30, 2009, 11:04 pmClayton E. Cramer says:
It would be simpler and more robust to verify that the input data conforms to certain standards of validity. For example, verifying that temperatures are in a valid input range, or that the latitude is between 0 and 90, and the longitude is -180 to +180. Or, for an example of where the Zuch school of programming takes you, verifying that the Congressional district in which jobs are created or saved is in the specified legal range of district numbers for that state. (Greater than 0; less than StateCongressionalDistricts[stateNbr].) But that’s only for boring sorts like myself, who think that programming is supposed to be an exact science, not a form of creativity.
November 30, 2009, 11:09 pmMalvolio says:
They are being swiftboated – like Kerry, they, and the world, are being presented with powerful evidence of their misdeeds. It is irritating, but hey, don’t do the crime if you can’t do the time.
November 30, 2009, 11:09 pmClayton E. Cramer says:
It’s a bad sign when you find yourself reading code and saying, “Why am I maintaining an entry in the Obfuscated C Contest?”
November 30, 2009, 11:12 pmClayton E. Cramer says:
Real software engineers actually do write code, in places where you are messing with big chunks of money (although far less than revising the world economy), where you checksum, or at least check digit, important input data. For example, credit card numbers often include a form of checksum that catches reversed numbers.
Zuch, maybe when you write software to maintain the books for a “medical marijuana” club, they don’t much care if the books balance at the end of the month, but when writing real software for real companies, it does matter.
November 30, 2009, 11:15 pmzuch says:
I didn’t say they don’t count. I said they weren’t the primary authority as to what’s happening (an unremarkable fact, when you think of it). As I said, comments are an unreliable (although usually trustworthy for honest and competent programmers) account of what’s being done.
What’s surprising here is that the vast majority of commentators seem to quote the comments exclusively as authority on what is being done, all the while that they’re complaining bitterly about the absence of raw data [read: "programs"] and the potential inaccuracy of derivative data [read "comments"], See any issue here? I mean, is it to much to ask for intellectual consistency?
They don’t. That’s a simple fact.
Assumes ‘facts’ not in evidence (one of which is a claim I’ve said something I haven’t). Not much of a legitimate argument.
Cheers,
November 30, 2009, 11:16 pmzuch says:
I used to joke I was a walking lawsuit waiting to happen. Working on projects to inspect jet engine turbine blades for defects, record ultrasound fracture data for rails for railway systems, various diagnostic medical instrumentation, and emergency systems for telecommunications amongst others.
I’m quite confident of my abilities … and the record bears it out. If you seriously think that it’s even possibly to do comprehensive “sanity” testing of inputs, you are simply mistaken. At best, sometimes, you can just say you’ll do what you said you’d do, and you’re not responsible for the consequences (something that is a bit of a standard disclaimer on essentially all software sold).
We do try and trap some obvious eedjitcy, but people are rather inventive. I often suggested that the QA people ought to hire monkeys, or put blindfolds on people an let them lean on the machines….
But get friggin’ real here … even Microsoft, with their known lax QA, has to get their products out the door not too many quarters behind the competition and announced release dates.
Cheers,
4Q2. I suspect that if I looked at all of your code, I’d find numerous instances where you have not covered all input conditions either. Which was my point.
See that bolding? And, as I said, if someone chooses to send you garbage sometime, there’s no real harm in sending them garbage back. Not everything is your job as a subroutine … and if you don’t understand that, you may not be a hacker, but you certainly are a poor excuse for a programmer (and likely not a very productive one).
Hmmm. I could say something. But I won’t. As it is, things are slow … but I’m busy trying to get our boat ready for the crossing to Panama. Different problem set … you haven’t seen true “spaghetti” (and the meaning of the phrase that “all electronics is in the end analog”) until you’ve seen yacht wiring.
Cheers,
November 30, 2009, 11:41 pmzuch says:
It may be, but is this a problem [or more specifically, was it for CRU data]? If the output is total garbage, wouldn’t that be a clue as well?
Cheers,
November 30, 2009, 11:44 pmzuch says:
Depends on whether the issue is that the code is wrong, or that you don’t know that it’s wrong. What I keep hearing is the former. The latter is curable … but it takes effort on your part.
Cheers,
November 30, 2009, 11:49 pmzuch says:
Errors (or assumptions) in logic in the code tend to make their way into the comments as well. This might be alleviated somewhat by forcing someone else to comment each bit of code. But what pay grade would you put such a person at?
Cheers,
November 30, 2009, 11:53 pmzuch says:
FWIW, not only do I tend to look first at code before comments, but in many cases I have found it necessary to go sifting through the assembly/ICE output as well to find out what was really happening.
One of my favourite anecdotes was when a friend of mine and I had been working on some code together. I was to call his subroutine, and let that do some processing. It wasn’t doing its job, and we started looking into it. I found that, with optimisation flags on, the compiler simply omitted the JSR to his code. I kept kidding him, “Wow, now thats’ some smart optimisation! It knows your code is useless!!!”
Cheers,
November 30, 2009, 11:59 pmDavid Schwartz says:
Zuch: I don’t see any substance to your responses. You’ve showed the same level of (likely deliberate) misunderstanding of what I’ve said. I don’t know if you’re trying to convince the uneducated or unable to understand the points I’m making. I’ve tried to give you a presumption of good faith, but that’s getting harder and harder to maintain.
The problem is not that any one of these things, by itself, is fatal. Often, if you have one weakness, you probably know where it is and you watch out for it. Competence can cover a lot. The problem is the conjunction of problems that interplay with each other.
The fact that the code will output ludicrous results when it runs into trouble (such as negative values for sums of squares) without noticing is compounded by the fact that the next program (which *knows* that these inputs are supposed to be sums of squares) doesn’t sanity check them means that you *need* the intermediate data (or the original data) to validate the final data.
That’s just one example of one thing that they found.
The type of retrospective validation that is being proposed here is simply impossible.
I could go through your misunderstandings point by point, but I don’t think it’s necessary. Mostly what you did is looked at a complaint I made, proposed the worst possible solution to the complaint, and then argued (surprise!) that the solution was worse than the problem. Sure, if those were the solutions they would have used, it’s probably better they didn’t solve them.
Maybe I’ll do a few:
With respect to your “checksums” proposal, that’s not what I said. Validating that it’s the specific data you meant from hardcoded values is obviously silly. However, that doesn’t mean that the failure to validate input is not a problem, it just shows that your proposed solution is silly. Duh. The right solution is not necessarily obvious. It may be to validate that the input is within sensible ranges or has the correct format. It may be to cross-check metadata (does the X file and the Y file we’re combining cover the same date range?). It may be to copy metadata from the inputs into the output so someone looking at the output can tell what inputs were used. It may be that this one problem is too hard to solve, and we have to accept it as a known weakness, accepting that it worsens our results and makes some types of validation harder.
With respect to hardcoding the user input, that might sometimes be the right thing to do. If you almost always use the same set of options, defaulting to those with a special flag to change them might be the right answer. Taking all the options from a template and printing which template was used and asking for operator approval might be the right solution. Copying the options selected as metadata into the output, and keeping that metadata through further processing might be the right solution. Or it might be that this is too hard to solve properly, and we may have to accept it as a known weakness. Again, we have to accept the specific consequences of doing that. (And, with luck, if we don’t have landmines everywhere, we can watch out for them. But without the original data, it may be hard to prove we didn’t step on one.)
But it should be immediately obvious to anyone with knowledge in the field that it will not be possible to validate the cooked data without the raw data given these specific weaknesses of the cooking process. This honestly is not a debatable point. Really, I’m not making things up here.
And with respect to the “total garbage” output, yes, that will be obvious if, AND ONLY IF, the next stage checks its input. If the total garbage is only a minor input to the next stage and the intermediate data is never checked (and not kept), the result will be that the final data is silently slightly off.
November 30, 2009, 11:59 pmJeff Hall says:
Clayton, would you marry me?
December 1, 2009, 12:05 amzuch says:
Note the bolded.
Did you have the opportunity to purchase compilers back then, perchance? I’m all in favour of good tools, but sometimes your [academic!] budget means you work with the computer you have, not the computer you might want or wish to have….
Cheers,
December 1, 2009, 12:06 amBruce Hayden says:
Part of the reason for this was that the FORTRAN compiler would kick out a lot of the uses of double complex when used with other data types.
But, interestingly, FORTRAN 90 has deleted the use of floating point numbers as loop indices.
December 1, 2009, 12:11 amDavid Schwartz says:
I agree with your comment about FORTRAN. Sometimes it’s best to use the tool you know best or that you have most handy (or the one you just used) even if it’s not the best tool for the job. There’s no reason you need the “best tool” to count how many lines are in a file or extract fields from a line of text. An analogy might be a construction worker who needs to hammer in a nail. If the hammer in his hand will bash in that nail, he’ll use it. And it won’t matter whether or not it’s the “optimum hammer”.
December 1, 2009, 12:14 amzuch says:
I disagree on both. The checksum is easy. And as to robustness, your approach doesn’t detect the use of the wrong input data files that have ‘reasonable’ but inappropriate data values. The checksum at least ensures “reproducibility” (and use of the original raw [albeit perhaps 'unreasonable'] data). But then you’d need checksums on the program to verify that another wasn’t inadvertently run. And from there, it’s turtles all the way down….
Cheers,
December 1, 2009, 12:14 amzuch says:
They using complex variables? Byte size accumulator variables? What’s the scoop on this supposed error?
Cheers,
December 1, 2009, 12:18 amzuch says:
Do I have to repeat myself once again?!?!? I’m all for getting to the bottom of any problems, but somehow I have this bias for looking at original data (say, the code used, the data used, and the problems encountered), rather than relying on ‘derived’ output (your summary conclusions) and the honesty of the proponent. If you get my drift. Translated into English: CItes, please. You will convince me much more easily of any problems that way. I’d say that all supposed ‘specifics’ I’ve seen so far of the alleged code inadequacies have suffered from the deficiencies I’ve complained about above (including citing caustic comments that essentially translate as “WTF?” as authoritative).
Cheers,
December 1, 2009, 12:25 amtheobromophile says:
The accuracy of the software aside…
People are beyond furious about this because global warming, then the more amorphous “climate change”, was sold to us as scientifically proven; those who challenged the science (either because their degrees and work experience made them sceptical, or because of other issues) were mocked as “global warming deniers” and put on par with those who deny the existence of the Holocaust or who believe that 9/11 was orchestrated by the Bush Administration.
Scientists, engineers, and generally intellectually curious people who questioned the determination that AGW exists were also regarded as crazy deniers, leaving no room for them to argue that science is almost never settled, that large systems (especially the entire Earth) have variables that are difficult to control for, or that the science, even if accurate, should not lead to almost unrelated policy results. I mean, all Hell would break loose any time someone would point out that we should consider other pollutants that may be put into our environment in an attempt to reduce carbon dioxide emissions.
Given that AGW proponents made it an all-or-nothing deal in which you either acknowledge the “settled science” or are a crazy denier, it is hardly surprising that people reacted to ClimateGate by taking the only other alternative to AGW belief presented to them: wholesale denial. Had reasonable discourse been a part of this discussion for the last few years, ClimateGate would not matter so much.
December 1, 2009, 12:26 amzuch says:
I will grant you this point.
Nonetheless, while I continue to adhere to high programming standards as circumstances warrant, I’d say that picking on academic programmmers for style flaws is picking low-lying fruit, and not cause for the present brouhaha which is yelling at the top of its lungs: “We need to toss it all out and start from scratch … and also mandate the reading of every word of each and every bill on the floor of Congress before anything else gets done, while we’re at it….”
Cheers,
December 1, 2009, 12:34 amBruce Hayden says:
All this reminds me of some more of my experiences as a programmer for the 1980 Census. First, we had a file titled “Garbage”, and it contained just that. Actually, we had several of them, each a bit different. And using them as input files would be part of our testing. It worked surprisingly well.
Secondly, as to checksums. Back then, due to the cost of media, most of production was run with tapes. Literally thousands of tapes. The same programs would be used with hundreds of different tapes within a short period of time. The I/O system we used generated and checked checksums, and our production branch meticulously cross checked every production run to make sure that the input files had the correct checksums and tape numbers. It was a lot of work, but it did seem to work.
My claim to fame at Census was writing the I/O system that read the raw optical data from the FOSDIC machines (Census built early OCR machines). It was great – I got to code in assembler, which was much more fun than FORTRAN, and I probably wrote the most obtuse code of my lifetime. It was obtuse because it was run on an early pipelined machine (Univac 1110), and I shuffled instructions and register usage to optimize the instruction overlap. No one else could apparently maintain it. But it did, fortunately (or hopefully) run correctly, translating billions of pencil marks in a matter of weeks.
December 1, 2009, 12:36 amzuch says:
“… amelioration will raise out taxes and gas prices <*waugh! waugh! waugh!*>.”
At least that’s what I’m hearing over and over from a good number of people on this blog (although not necessarily you).
Cheers,
December 1, 2009, 12:37 amsitzpinkler says:
What if we pull public funding until they shape up?
December 1, 2009, 12:45 amBruce Hayden says:
I would be very surprised if they didn’t have a C compiler lying around. My experience, at least from the early 1970s through early 1990s when I finally gave up being a student (ok, I did have three graduate degrees by then), was that most universities had such as a matter of course, esp. if UNIX were being used. Of course, you can abuse C as much, if not more, than you can FORTRAN. While that language isn’t the best with strings, it is, in my experience, far better than FORTRAN (And, I have done plenty of string processing in both to know).
December 1, 2009, 12:45 amzuch says:
I’m actually in the middle (as are, I think, Profs. Adler and Volokh). I think it quite likely but not proven. Time will tell … for sure. What I object to is the Swiftboating being done, and the poor quality of the AGW criticism (coupled in many cases to thinly veiled opposition to policy alternatives that might recommend themselves given certain outcomes) I find here. I think the criticism overblown, unfair, and certainly not a good thing at a time when public knowledge and appreciation of science seem to be nearing a nadir.
Cheers,
December 1, 2009, 12:46 amJohn Moore says:
Geez, I really, really hate to side with Zuch, but…
AFAIK, most research software (other than general purpose stuff) is done by grad students and other folks who are not professional programmers. The state of the code at CRU does not surprise me at all.
Okay… enough of agreeing with Zuch… now to th emeat of things…
The important point is that CRU is no longer just another research establishment. When a lot of this stuff was done, it may have been. But now, it is producing results which may result in dramatic social policy changes before the science can be validated/invalidated.
That means that the quality control on the overall process should be equivalent to that of a safety-of-life system. That includes:
…software
…scripts
…data file management
…metadata and documentation
…intermediate result files, and their metadata and documentation.
…unit and integration test suites
…version control
It is clear that they come nowhere close to this standard.
Put another way: the standard for ordinary science, which will be checked by someone else *if it matters*, is much lower than what CRU should be held to.
The state of their process is understandable, and in the current circumstances, utterly inexcusable. That they have been contributing to such important policy efforts while knowing that their code and data management was of such poor quality (which they had to know) approaches criminal negligence and fraud.
As to the scientific consequences, consider that they are the primary source of surface time series for climate analysis – as designated by the UN (tells you something about the UN and IPCC standards, btw). Their results have been used to calibrate, and to validate/invalidate other data sets and other measurement techniques. That means that their sloppy work has polluted the entire climate prediction field, from modeling (which uses paleoclimatic data for calibration and for development of parameterizations) to statistical inference based on the past to instrumentation. It’s all now called into question.
Also consider the social effects within the science community. Here is what Richard Feynman had to say about that effect:
Translation: Millikan was the authority. Millikan was wrong. Everyone else threw out data if it was too far form his, and it took a long time to converge on the real value.
Meanwhile, everyone’s off to Copenhagen. No time to converge on the real value here… too much economy wrecking to do…
December 1, 2009, 12:47 amJames N. Gibson says:
After going through all this I’m going to be rude. Clayton and Zuch, your missing the point.
I’m not a programer, just the poor smuck engineer who tends to be called by other engineers for help getting out of a problem in MS Office without wanting to go on hold with the IT help desk. Yes, I have programed in Fortran: had no choice since I had to complete two programs for graduation with my bachelor’s. But how good, or clean, or easy to follow my programing was was not important. What was important was that I was able to write a program to run a series of data sets over and over through a set of formulas worked out in the second world war to size turbocharger’s for certain altitudes and size of engine. In short it doesn’t matter how good the data is coming in, or how good the coding, if the formula that the data is run through is faulty. And no one has shown me that the climatologists have a generally accepted formula, or even group of formulas, to serve as the basis for the generation of any computer program.
December 1, 2009, 12:54 amBruce Hayden says:
I would consider that a feature, not a bug. At least usually. I would suggest that if a comment indicates a bug, then it is likely that there is one, while if a comment looks good, then you should probably see if the code does what the comment says it is trying to do.
But that, unfortunately, reminds me of another story. After Census, I was doing work for NOAA, and it was a COBOL shop. I wrote some assembler for them, and they insisted that I comment the code. Fine. But then, they wanted it flowcharted. Not so good. Until I found an automatic flowcharter for assembler code. Not surprisingly, the output was 20x the underlying assembler code in size, since every instruction got its own box. But we were all happy. They had their flowchart, and I didn’t have to do it by hand (and, yes, it was worthless).
Getting back to Zuch’s comment. I have suggested before that in this sort of work, the optimum is to include professional programmers as integral members of the research team, and reward them appropriately, both through salary and recognition. I don’t think that is the place to cut corners.
December 1, 2009, 1:01 amJohn Moore says:
I wanted to add one tidbit… the technical term for some of what happened to the CRU software is “cruft“.
December 1, 2009, 1:17 amBruce Hayden says:
I can see your problem, esp. with the later. But one place where I disagree is that the criticism is uncalled for. For one thing, these scientists have gone from doing just science to being political, and that, I think, opens their work up to critic by those affected by their advocacy, even, and maybe especially, when cloaked by the mystic of science.
Are any of us experts in their area of expertise? Hardly. Never will be. But a lot of us, I think, know an awful lot more about what went into all those fancy graphs we see so often predicting the end of the world if we don’t do what they tell us to do immediately, if not sooner. And, the more we see, the less we trust the messengers.
Does all this mean that man isn’t causing at least some global warming? No. But what it does call into question is the rush to beggar ourselves and our children.
You suggest elsewhere that we shouldn’t be worried about a trillion here and a trillion there just to be safe. But we aren’t just talking about those of us with decent jobs having to eat hamburger instead of steak, but rather, whether our kids get jobs, or whether we end up with a permanent unemployed underclass. We are talking potentially millions of jobs, just to be safe here (in the midst of the biggest recession of most of our lives).
December 1, 2009, 1:21 amtheobromophile says:
zuch: the snark about gas and such first, then the rest. I live in the Northeast. I am not hard-pressed to remember summer 2006, in which gasoline hit $4.50/gallon and every middle-class and lower-middle class person in New England began to panic. The question on all of their minds was how to afford home heating oil, which looked like it would be about twice as much as the previous year. When the price of oil came down, people rejoiced, not because filling up their cars no longer cut into their latte budgets, but because they could actually afford to turn the heat on in their houses that winter.
That carbon cap-and-trade would lead to the same result does not mean that people are whining; it means that they are actually wondering how to pay their bills. My two biggest problems with carbon cap-and-trade are that they will hurt the working poor the most (as the price of getting out of bed in the morning – i.e. food, electricity, gas, and heating oil – will radically increase), and manufacturing will be sent to developing nations that lack our environmental standards for production. I would rather have manufacturing in America, with our strict pollution controls, new and cleaner technology, and threat of lawsuits if anything goes wrong, than have it all move to a country wherein people regularly pollute with reckless abandon.
December 1, 2009, 1:44 amKazinski says:
Zuch,
I think Zuch has been working on CRU. Here are some actual comments from Harry_read_me.txt:
I work on the SCADA system on a municipal water system. The idea of sending garbage back, when you get garbage in is ridiculous, when you get garbage in, you send an error message back, or don’t you do error handling, Zuch?
December 1, 2009, 1:46 amDavid Schwartz says:
Now we don’t know how quickly this particular problem was caught or how quickly it was contained. It’s possible this *specific* error caused no damage whatsoever. (In fact, most likely it aborted the first time it tried to take the real square root of a negative number, which is the most logical thing it would have done.)
I’m not saying this specific error made their data worthless. I’m saying that this type of intermediate error, produced because of poor input validation and not caught because of poor error-checking (they caught this by accident, due to the square rooting) means you need the input to validate the output.
December 1, 2009, 2:07 amtheobromophile says:
That said, zuch, I think that you are mistaken in putting your criticism of the anti-science zeitgeist at the feet of the anti-AGW crowd.
Most of the criticism of the AGW agenda that I’ve heard has been about science, not about anything else. (On that note, please stop listening to what liberals tell you about how conservatives think; we’ll be more than happy to break it down for you ourselves, without the middleman.) The main issue that people (and I, for the record) have is that the climateologists seem to be making what I can only describe as the Larry Summers mistake: reducing a system with a nearly infinite number of variables to one single variable – and, to add to the absurdity, to do a static analysis on a changing system.
That alone does little but undermine, not promote, respect for science; even a layperson has an intuitive understanding of the sheer complexity of these issues. Furthermore, distrust of science seems to be highest when science gets too cozy with policy – especially policy that has a funny way of taking over people’s lives. Americans are well aware that science has been used to justify all sorts of interesting things, from blatant and horrific discrimination to some pretty interesting power grabs.
Challenging science will never undermine support for it; rather, science will then become like that team who isn’t afraid to play any other team that offers up a game but remains undefeated. Nor will treating science like anything else be helpful; it is, as I already said, an evolutionary process. Underestimate non-experts at your peril; they understand these things and, lacking good answers, will simply disbelieve you.
December 1, 2009, 2:10 amAlanDownunder says:
Easy. Weigh the relative integrity of the methods of the respective sides. It was thought be be 100-0. Now you can call it 99-0.
Sceptics still believe in insurance. Only denialists don’t. Even sceptics think that denialists are either loopy or bent.
December 1, 2009, 2:13 amMalvolio says:
Wow, that’s a whole lot of fail in just two sentences.
It isn’t as if someone at CRU made an error. It’s that the senior staff of CRU apparently conducted a decades-long campaign of deception and intimidation and no one on “their side” noticed or cared.
On the other side? There is no other side. There are just thousands or ten of thousands of people who have begin to believe that they’re being lied to and don’t like. They don’t claim to have checked each other’s work and they certainly don’t vouch for each other.
December 1, 2009, 3:40 amAlanDownunder says:
@Malvolio,
Epic fail is thinking that the CRU disclosures have stopped the glacial melt and the sea level rise.
Also in thinking that ideology and short-term bottom have nothing to do with the denialism.
December 1, 2009, 6:59 amzuch says:
And:
What say some show trials, eh? That’ll teach ‘em.
Cheers,
December 1, 2009, 8:21 amzuch says:
“… say their political opponents.”
Cheers,
December 1, 2009, 8:53 amClayton E. Cramer says:
And you are willing to see the world economy transformed (and probably in a bad way) based on code that is this badly written?
December 1, 2009, 9:11 amClayton E. Cramer says:
I didn’t miss the point. I was pointing out that bad coding techniques (such as failing to check the validity of inputs) are a serious problem. But it sounds like there are a number of other serious problems as well.
December 1, 2009, 9:14 amClayton E. Cramer says:
You are essentially saying that because we can’t get perfection, there’s really no need to even try.
Yes, there can be data values that are within valid range, but still completely wrong. But that doesn’t argue for failing to check that inputs are within the valid range!
Your “if they feed you garbage, they deserve garbage out” is the sign of someone who doesn’t take his job very seriously. People make mistakes about input files all the time. If you have hundreds of programs that are all quite similarly named, you may end up running the wrong program. Solution: don’t have hundreds of similarly named programs available to the user. Create a menu system that invokes the correct program for the intended purpose, and have the output of that program clearly identify what program produced it. You can still screw up, but at least your output dataset or graph will tell you which program produced this output.
December 1, 2009, 9:20 amzuch says:
Where did I make such a suggestion? And the costs (and benefits) of various policy options are more uncertain than the climate science.
Wouldn’t it be best to get all the science nailed down as well as we can before we say what policies are appalling?
Cheers,
December 1, 2009, 9:26 amwws says:
Kazinski says: “I think Zuch has been working on CRU.”
I think you’re on to something, seriously. The fact that Zuch has virtually hijacked this thread and made it all about *him* tells me he’s got some serious skin in this game. Look at the times of his many posts – he stayed camped out on this one thread almost continuously from 8 last night til 1 am, and then he started right back up at 8:21 this morning. Looks like he’s got some personal vested interests to protect here.
December 1, 2009, 9:34 amzuch says:
On, yes. The Kyoto accords were deadly and costly. Hey, waiddaminnit … weren’t the Republicans (nominally) in charge? What could possibly have caused such a spike (and subsequent even as mysterious decline)?….
Sorry for the continued snark. Sometimes I just can’t help myself….
Cheers,
December 1, 2009, 10:23 amLaura(southernxyl) says:
Zuch.
We never signed the Kyoto agreement. You know that, right? Do you think Theo is saying that Kyoto caused gas prices to go up?
Are you incapable of figuring out what she’s saying?
Or are you arguing in bad faith?
Here’s an exercise for you: Read Theo’s comment again, and then, without snark, explain what she’s saying. You don’t have to defend it, just explain it. If you can’t or won’t, you must realize what that says about you. “Can’t help the snark” indeed.
December 1, 2009, 10:38 amzuch says:
Wow. Learn something new every day. Like that sometimes I have to make my sarcasm more explicit or leave it out altogether, as the audience isn’t as discerning as I might find elsewhere.
Cheers!
December 1, 2009, 10:49 amzuch says:
Nah. Just some free time waiting for a bearing for the ‘Green Machine’ to come in from the U.S. Wind power, here we go!!!!
Cheers,
December 1, 2009, 10:53 amzuch says:
Depends. Discussed above. Sometimes it just ain’t your problem. The important thing is that you do what you say you’ll do. I’d note that many “stdlib” funtions return a “-1″ on “error”, rather than setting “errno” or throwing an exception. Is the “-1″ garbage? Well, in a way, it is. What it most emphatically is is the advertised behaviour of the function under such circumstances.
Cheers!
December 1, 2009, 11:03 amLaura(southernxyl) says:
Zuch, you can’t help the snark because that’s all you’ve got.
Cheers!
December 1, 2009, 11:09 amLaura(southernxyl) says:
As to sarcasm, if you’re going to use it, it might behoove you to recognize it when other people use it. Otherwise you demonstrate that you are in over your head.
December 1, 2009, 11:10 amzuch says:
Reals (even single precision) can far exceed 2B. I think the programmer is mistaken.
The value is in the neighbourhood of -2B, an interesting value, being somewhere around the smallest negative (and largest positive) integer in 32 bit architecture. There might have been some implicit integer conversion somewhere (all the input values are integer values to 2 decimal places), and there might have been integer overflow.
Yes, that would be a bug. Too bad it looks like our courageous critic Harry didn’t take the time to track it down….
Cheers,
December 1, 2009, 11:13 amzuch says:
That’s sarcasm. Right? ;-)
Cheers,
December 1, 2009, 11:15 amzuch says:
Depends on the nature of the criticism. Arguments from incredulity (or ignorance) hardly help (see, e.g. the ID approach).
And then we have the creationist demands for things that they know are impossible and unreasonable, such as the demand for the two new transitional fossils each time a transitional fossil is found. By their count, we know less and less as time goes on, nicht wahr? The demand for more and more data is never-ending (while, you’ll note, the amount of “data” on their side is slim and constant; the prime method ot proof of their theory is to attempt to wound an alternative one).
And then they say the whole edifice is crumbling and it’s all disproved, refuted, and total garbage when something like Piltdown shows up (but not Paluxy, interesting, no?). Anything about this sound familiar?
But this type of “criticism” is not honest. Nor is it science. It is based on the outcome (or lack thereof) desired.
Cheers,
December 1, 2009, 11:26 amDavid Schwartz says:
First, the two situations are not comparable. A single, advertised error code is not the same thing as outputting whatever garbage the nonsensical input or bug happens to produce.
But more importantly, this on its own would be perfectly fine, even if it wasn’t a single, advertised error code. We don’t demand (or typically even aim for) perfection — real-world disciplines are always about compromises. However, this one issue can be part of a “perfect storm” of problems. For example, if this function outputs -1 under cases where its input is invalid (which it should if that’s the advertised error code), the previous code may not bother to check if its output is valid. Why bother if the next stage traps invalid input anyway? (Or it could be part of a general policy of not validating output.)
Well, what if the next stage doesn’t check if its input is valid either? Now you have a perfect storm. A bug in one stage causes invalid output. The invalid output causes the defined error code to be output. The defined error code is taken as valid input by the stage that doesn’t check its input, and produces valid-appearing output because it was a minor input. (Say, it’s the magnitude of a correction that should be a small, positive number.)
Add that to a circumstance where we are looking for a very small signal buried under the noise of a very large signal and wherein we can only validate the final output for much larger signals.
The point is, what they were trying to do is an incredibly hard computer science exercise, even for top large data set experts. Small signals can easily be lost in large ones just through a simple mistake like not scaling before summing (for averaging) or summing in a data type of the wrong precision.
If their results cannot be repeated from the same (or similar) raw data, they are basically worthless for the purpose of finding a small signal in the large one. These are not style nits.
They literally engaged in bug fixing until the output looked right. The result is that the output looks right — no more, no less. The next bug fixed could have changed any results beyond that and almost certainly would have.
December 1, 2009, 11:26 amzuch says:
… which of course changes the truth value of the underlying science.
Cheers,
December 1, 2009, 11:30 amzuch says:
Are you willing to see the world transformed (and probably in a bad way) solely because some code is this badly written?
Why don’t we examine whether this is just fluff and bombast, or actually significant? When we’re done with that, maybe we can discuss policy implications. It works both way, you know….
Cheers,
December 1, 2009, 11:35 amafu says:
What a ridiculous thread. The HARRY_read_me.txt refers to work on the CRU TS 2.1 product which is a legacy project and not used in any current climate predictions and therefore have no bearing on the current climate change debate.
If you want to be taken seriously, please criticize current data and provide a cite to actual code, data sets and journal articles, not unverified and probably illegally obtained .txt files.
December 1, 2009, 11:42 amDavid Chesler says:
I read this during work sometimes to get away from just this sort of thing, math-rich software written by bright capable folks outside their area of expertise. Reading the Pajamas Media excerpt sent me screaming back to what I get paid to do.
December 1, 2009, 11:55 amFub says:
Thread winner!
I don’t know whether to bow ceremoniously in awe, or to send you a sympathy card (having also once in days of yore suffered the misfortune of an assignment to herd, rope and brand some data with a Univac).
In any case I thank both Clayton Cramer and Bruce Hayden for your incisive comments on appropriate DP and computation practices in these threads.
December 1, 2009, 11:58 amzuch says:
No. I’m saying that nitpicking code without demonstrating an actual material flaw, and then saying the whole thing (many parts of which aren’t even affected by the code in question) is garbage and needs to be tossed is a bit overblown (and perhaps motivated by other sentiments than an affection for good science). But that’s what I keep hearing over and over on these threads.
FWIW, that bug that Schwartz cited above included a value that seemed to be, while abberrant compared to the rest of them, within a superficially “valid” range. Range checking wouldn’t help, would it? In fact, the fact it coredumped may have been a[n accidental] blessing, no?
On the contrary, I do. I’ve explained this above. I think you should consider what I said apart from the implications for the present dispute on the level of CRU error checking, and see if it makes sense.
I’d note that on Solaris systems, various debug flags can be set to kernel memory allocation. This is sometimes (lacking an ICE or logic analyser that could help) the only way to run down abstruse system panics. They introduce “red zones” around allocations that will often indicate buffer overwrites, and mappings and unmappings of various buffers may trap exceptional accesses immediately, rather that wait for subsequent corrupted data panicing the system far from the proximate cause. But all this takes memory overhead and CPU resources. So you generally run without such until needed. What? No range checking?!?!? My, how horrible. But concerning the Solaris OS, to paraphrase Galileo, “yet, it does move” (and faster than Micro$ux).
Cheers,
December 1, 2009, 12:02 pmRichard Aubrey says:
afu
“illegally obtained files” Is that a concern? Really?
December 1, 2009, 12:03 pmI’m not the one who noted that when Daniel Ellsberg stole the Pentagon Papers, the discussion was all about the content.
When somebody stole an e-mail from Jay Rockefeller about politicizing the Iraq war, the discussion was all about the stealing.
Helps to hide the content, I suppose.
Let me also say that, wrt the complaint that the stolen, hacked, corrupted, BS stuff from CRU is obsolete and means nothing…. They are either liars or incompetent. Thus, further information from them must be treated carefully.
Do you disagree?
John Moore says:
zuch says:
You mean like the “scientist” in charge of the GISSTEMP records (the US surface temp series), James Hansen demanded?
December 1, 2009, 12:21 pmDan Weber says:
You guys are making me agree with Zuch. That’s pretty impressive.
For a scientific program, it doesn’t matter one bit if the program isn’t resilient against hostile inputs, or is sloppy, or is running inefficiently.
This code is embarrassing to look at, because I wrote stuff like it waaaay a long time ago. I would sometimes find an extremely slow or ugly way to run, but it would run.
I can believe that there are scientific problems with this code, but I’ve yet to hear what they are.
December 1, 2009, 12:23 pmJohn Moore says:
Notice also how the CRU emails, and Zuch’s end in “Cheers.”
:-)
December 1, 2009, 12:24 pmzuch says:
Fine. Show it. Don’t assume it. I keep saying this, but isn’t the criticism of the CRU that they [allegedly] assumed their conclusions rather than derive them?
Actually, that describes a very large portion of debugging. ;-)
OK. Show your work.
Cheers,
December 1, 2009, 12:33 pmDavid Schwartz says:
Richard Aubrey: I don’t think they were either liars or incompetent. I think they just made a lot of choices that seemed reasonable at the time and lost the big picture. It’s surprisingly easy to do in a large project with lots of people with different areas of speciality. The guys running the show are not computer science people, and they just expect that those guys can keep things going without a problem. They have no interest in understanding the crumbing of the foundation. (And you see a lot of that attitude in this thread, frankly.)
Ian Harris, the probable author of the HARRY_READ_ME file, appears to be quite competent. He’s simply in an impossible position, dealing with people who just want him to produce answers. Had he pointed out the flaws he obviously knew about with the proper level of alarm, likely somebody else would have been found who wasn’t so “difficult”. (And perhaps he did, and was told to just make it work, we don’t really know.)
My point, and I think it’s an important one, is that even with competent, honest people, a set of compromises made over a long period of time and under terrible time pressure can make a one-in-a-million “perfect storm” of problems inevitable and practically certain. We now have evidence that the CRU code was riddled with such compromises and plagued with the precisely these consequences.
Of course, it’s always possible they realized that and started over at some point and their newer work is entirely reliable. But they now have that burden of proof and have lost any presumption of reliability due to their competence and honesty. We now know that was not enough, at least in the past.
December 1, 2009, 12:36 pmDavid Schwartz says:
That is a professional judgment from years of experience. If you don’t agree, feel free to ignore the last part. It isn’t critical to my argument. I admit, the output might happen to be entirely right. However, it is definitely not reliable because of the combination of the inability to reproduce it and the compromises made in producing it.
Actually, that particular data set can’t possibly happen to be entirely right. Data corrupting bugs were identified and repaired and the data not re-run with the fixed code. However, at least for the bugs we specifically know about, the effect of those bugs was likely minor and may have been entirely rejected in later passes. Fixing the bugs likely would have changed the output from one valid set to a different “more valid” set because most of the bugs would have resulted in valid data being rejected.
And that actually brings up another problem. If we could re-run the CRU data set, there are some very interesting things we could do. They fall into the general category of investigating how robust their warming conclusion is. For example, we could delete a few stations, re-run the program, and see if the warming is substantially the same. We could try different corrections, at both the upper and lower ranges of what is scientifically justifiable, and see what results we get.
The conspiracy theorists will argue that this was precisely what CRU wanted to avoid. At least for some physics-based modeling, we know the warming that results is very brittle with respect to input changes. (Or at least was, I don’t know offhand how current that overservation is.)
December 1, 2009, 12:47 pmDan Weber says:
I’d like to see another algorithm to determine if any two of N points of the form(a,b) are within L distance of each other that runs in O(N) time.
And why in the world would you use “trig functions” to figure that out, anyway? Talk about overthinking the problem. Unless “multiply by itself” is a trig function now.
Seriously, there are problems in the CRU emails, like the attempts to delete information to hide from FOI requests, and we should pay attention to those. Not to people who don’t understand ninth-grade math.
December 1, 2009, 12:56 pmzuch says:
Details?
Other that that, I agree with your penultimate peragraph. Maybe if some of the CPU horses being expended on blog text processing were harnessed in a distributed SETI-signal-processing type arrangement, we could have tons of perturbation runs … and less heat. ;-)
Cheers,
December 1, 2009, 12:57 pmtheobromophile says:
Thanks, Laura.
As for the respect for science element: zuch, I began discussing climate science; that you had to drag creationism into this is beyond ridiculous. That’s the epitome of a straw man argument.
(Incidentally, our view of evolution has evolved since Darwin wrote “Origin of the Species.” We’re still studying the rate at which evolution occurs – interestingly, advances in that field go a long way towards disputing creationists. I cannot help but wonder what would happen to junk science and scientific scpetisicm if the response to people who question long-held beliefs were not to STFU, but to explain that science is a process and that we do not yet know something does not mean that it is unprovable.)
December 1, 2009, 1:25 pmRichard Aubrey says:
David.
December 1, 2009, 1:39 pmOkay, they’re not liars and their incompetence is a matter of the organizational structure.
Do we now take them at their word?
In addition, do we dismiss their e-mailed discussions of how to hide, delete, or destroy information, along with discrediting skeptics?
Tweets that mention The Volokh Conspiracy » Blog Archive » Clive Crook (at The Atlantic) on “ClimateGate” -- Topsy.com says:
[...] This post was mentioned on Twitter by Addie Bendory, Eugene Volokh. Eugene Volokh said: Clive Crook (at The Atlantic) on “ClimateGate”: Here’s what Crook writes: I admire expertise, and scientific ex.. http://bit.ly/51mTva [...]
December 1, 2009, 1:40 pmRichard Aubrey says:
Crap. What is it with this thing? Anyway.
If the CRU were honest and attempting to be competent, they’d be saying,
December 1, 2009, 1:43 pm“Ooooh shit! Boy, did we screw up. We see it now. Jeez, are we sorry. We’ll start over, take a 50% pay cut to make up for all we earned while we were screwing the pooch. We’ll do it right and we’ll make arrangements for anybody who wants to to follow us step by step, with generous opportunity for feedback and input.
Sorrysorrysorrysorry.”
I’ve been listening….
zuch says:
I don’t accuse you of such; I’m just noting the similarities in the general tenor (and impetus) of the ‘scientific objections’. If you disagree with my take, why don’t you explain why? If this were just a scientific question, well and fine, but then why do I see in each and every thread on the so-callled (“poisoning the well”?) ClimateGate many posts on why we can’t trust the science because it will cost all kinds of money and raise my taxes?
If this were a controversy about TS mutations in development in 2nd instar Drosophila melanogaster larvae, we wouldn’t have a swarm of “I’m not a scientist, I just play one on the InterTooobz” bloggers engaging in code ‘reviews’ of leaked academic code based on freakin’ irate comments…. Nor would we have energy companies funding ad hoc “think tanks” criticising the work (quite similar to the way they had their hired guns like Steve Milloy doing the same for the tobacco companies….)
The criticism is not sua sponte. It is political in nature.
We have the example of the NCSE and TalkOrigins. These are excellent resources, but they really haven’t done the job of driving a stake through the heart of the creationists/IDers, who continue to enlist sympathetic swarms of nabobs and even legislators (not disjoint sets, mind you) and try to wreck science education and public knowledge … and sometimes succeed (see, e.g., Texas and the school book fights).
I can understand the frustration of the actual scientists when faced with this kind of stuff. As I pointed out on another thread, public “debates” between creationsists and various biologists are at best “no decisions” and in many cases, the creationists “win”. This is something that ought to scare you as much as it does me. And we have a fair number of people (even a majority at one time) who thought that we’d found WoMD in Iraq, and a substantial portion of the U.S. population thinks that Obama’s a Muslim and/or not a citizen. Having the facts on your side doesn’t go far in the U.S. nowadays, and this ought to be troubling for anyone that appreciates rationality.
At the very least, will you admit that nothing we’ve seen so far justifies saying that all the climate data is “garbage”?
Cheers,
December 1, 2009, 1:47 pmDavid Schwartz says:
I’m not sure if you really want them, but yes, they exist and I can provide them. Are you just trying to tire me out or do you feel like it’s fun to make me substantiate every minor point I make? We have tons of evidence that errors were left in the code because there was reason to think the error wouldn’t be substantial.
Emphasis mine.
There are better examples, I just don’t have the time to locate them right now. Compromises were taken that were known to reduce the quality of the output. There is no reliable way to estimate the cumulative effect of all those compromises (other than repeating the process, which we cannot do).
But I don’t think this is a very serious problem. We can repeat the process with whatever raw data we can manage to scrape together. And then we can run the numbers all different ways. If we repeat the CRU results, we’ll know how robust they are. If not, we’ll know how robust they aren’t.
December 1, 2009, 2:10 pmzuch says:
Once again relying on gripin… — um, sorry, comments. Absent context. As I’ve pointed out previously, it sounds a lot like he was trying to reproduce or rerun something that others had done. Do we know for what purposes his runs were made, and whether and how that run made it into the data sets (and what use was made of them)? And his ignorance of what had been done before is just that; not an indication that the prior person had done things wrongly. If he did things wrongly (or ignored errors when he ran the procedures/programs), that may in fact be significant (but it is yet to be shown that it is; did he document his failings sufficiently for others to go in and try to complete the job?) and this may be attributable in part due to poor prior documentation, but it hardly makes him any angel here, or some authority on the alleged deficiencies. Maybe someone else needs to do it. On that I will agree. But in the meanwhile, H[0] is not assumed to be 0 and variance infinite.
Cheers,
December 1, 2009, 2:36 pmSG says:
You’ve emphasized the wrong word. The correct claim is “nothing we’ve seen so far justifies saying that all the climate data is “garbage”.
What we do know is 1) someone with full access was unable to duplicate the data, 2) some amount of raw data is claimed to be missing (and missing under suspicious circumstances), 3) the people at the heart of this have not always acted in good faith, 4) much of the notionally independent climate data has been validated/adjusted to conform with this data and 5) peer review in this field is not as independent or as rigorous as we may have thought.
So no, we don’t know that it’s all garbage. We do have valid reason to suspect that an indeterminate amount of it may be garbage. We have no reason to have faith in this data or any other data until it’s been independently verified.
And no matter how much you may assert to the contrary, it’s CRU’s burden to prove their case, not others to disprove it. Given what we now know, under no reasonable standard have they proved their case. It doesn’t mean they’re wrong, but your ever farther reaching justifications don’t even begin to make the case for them. That their code is bad is not determinative; that their code is not repeatable is. They simply don’t (at this point) have scientific data.
Perhaps Harry’s incompetent, but unless and until someone reconstructs the processing and the raw data (if necessary) such that their results can be independently examined, then we are absolutely justified in dismissing CRU’s results and any results that build off of CRU.
So don’t defend this – it’s indefensible. If you want to rebuild confidence in the AGW hypothesis, go start mining the literature for work that’s truly independent of CRU, or help whoever is trying to reconstruct CRUs work in a repeatable fashion. Climatology will be better off for the effort. Your turd polishing exercise is not helping anybody or anything.
December 1, 2009, 3:46 pmCareless says:
ITT (and the last one on the subject I noticed): Zuch defends scientists threatening to destroy archaeopteryx rather than let people from the Discovery Institute have a look at it. Their behavior is reprehensible! I can’t believe people are defending this.
December 1, 2009, 3:56 pmzuch says:
What we know is that he was frustrated. What we don’t know is his abilities … or that it was impossible.
Raw records. Whether the data is missing or not (or cannot be reverse-engineered) is unknown. And the “suspicious circumstances” is “garbage” (if you pardon the expression). The records were discarded decades ago, long before the tenure of Jones or the “hockey stick” analysis.
You mean they were caustic and dismissive of their tormento… — ummm,
“critics”? Imagine my surprise.
How independent and rigourous did you think peer review is? After all, peer review allowed Pons and Fleischmann’s article through, as well as another Ig Nobel laureates’ paper in Nature, a rather big embarrassment for them when they were show up by a magician?
That being said, keeping creationists off the peer-review boards of Cell Biology would hardly seem to be corrupting the peer-review process. Can you explain why this is much different? For point of reference, I’d note that most of the scientific work of critics of GW/AGW reports have been critical of the work of others, and much less original research or theories have been produced by them (that stood the test of time, anyway). To paraphrase an old aphorism, “those that cannot do, criticise.”
Cheers,
December 1, 2009, 4:41 pmzuch says:
Well, go for it!!!!. As I hinted in my last post, there is a lack of original research by the critics. Go do it!!! No one’s stopping you … and you shouldn’t need the records, that the CRU spent their time assembling, to do it either….
Cheers,
December 1, 2009, 4:44 pmNobody At All says:
A question of philosophy of science: Is there a burden of proof, and who bears it?
According to someone like, e.g. Quine, you would say that our individual scientific beliefs each contribute to a metaphorical “web” of beliefs. No individual belief is held sacrosanct, but is held tentatively. When we replace an individual belief of the web, this can be thought of as “tugging” on web, shaping our entire belief system as a whole. Those beliefs which, over time, prove most resistant to change are given the most confidence, and undergird the web. Subjective choices, (e.g. which hypotheses should be given resources to test) are made on the basis of which beliefs have the least confidence, and whether the competing hypothesis would fit more comfortably within, and add strength to, the overall web. Alternatively, over time (gradually or abruptly) observation, experimentation, and new theories may weaken the web to such an extent that it becomes created anew – a paradigm shift. (e.g. from ptolemaic astronomy.)
Climate science isn’t “settled,” in the sense that it remains a science. Yet, there is (and this is a descriptive/sociological statement) a “web of belief” which climate science works with, and has confidence in. The practical upshot is that understanding the degree of confidence held in any particular belief (e.g. the role of water vapor versus CO2 when modeling increasing concentrations of CO2), depends on understanding the extent to which confidence is given supporting beliefs in the web.
Skepticism is indispensable – without it, we could not build a web in which we could have the most belief. The denial of data is not just reprehensible from an ideological standpoint, but from a scientific one. Yet, from the standpoint of scientific belief, (if you accept this sort of philosophy of science) skeptics carry the burden.
December 1, 2009, 5:20 pmzuch says:
They’ve published. That’s the normal order of things. Now it’s incumbent on you to show they were wrong, and how and why. They they can try to address the criticisms and fix their stuff as welll as they can. Thesis, antithesis, synthesis, and all that. That’s the process.
Note that even for very non-intuitive claims, such as those of Pons and Fleischmann, and Benveniste, that’s how it worked. Why you think that we need scepticism if not the outright rejection we see around these parts in advance for this work prior to “replication”, when in fact the known physics (”greenhouse effect” and such) is rather unremarkable and unobjectionable, is beyond me.
Cheers,
December 1, 2009, 5:21 pmzuch says:
This is really weird. I’ll say (for perhaps a fourth or fifth time) that the standards for “knowing” something (say, academic misconduct occurred or, say, global warming is happening) seems to be rather situational. Why? The quality and completeness of the evidence in one case doesn’t seem to measure up in the least to that needed in the other, despite the fact that the other conclusion is rejected (or at the very least sniffed with rather upturned nose)….
Cheers,
December 1, 2009, 5:42 pmzuch says:
… which, conveniently for the Bloggerdom Climate Swiftboaters, is by popular acclaim all of it despite the fact that most everyone claiming such actually has no idea whatsover. Magical, I tellya.
Cheers,
December 1, 2009, 6:06 pmzuch says:
Had you said “scientists”, you might have had a point. But then that would be counterfactual.
They can go dig up their own gawddam “evidence”. See, e.g., “Paluxy prints”.
Cheers,
December 1, 2009, 6:10 pmSG says:
Nobody At All:
I think Popper (or at least my vague memories from a long ago philosophy of science class) presents a reasonable frame for this discussion. CRU (in this case) bears the burden of presenting a falsifiable claim. Once they have done that, the burden then fall onto skeptics to falsify their claim, and to the extent that skeptics try and fail, the more confidence we can have in CRU’s claims.
But they haven’t held up their end of the bargain. They may be right, but their claims can’t be tested. They can’t duplicate their own results and some of the data is lost – this is not my assertion, it’s they’re claims. They haven’t met their basic burden for their claim to even be considered as science.
And again, none of this says they’re wrong. The actual correctness of their claim is tangential to the discussion. But if our plan is to use the scientific method to determine with confidence (A)GW, at this point nothing can be said one way or the other about the truth of (A)GW based on the CRU data. It’s not that these strands in the “web of belief” have been broken, they’ve simply disappeared along with that which was connected to them.
They haven’t necessarily permanently disappeared, but they can’t (shouldn’t) be assumed to be valid – they actually need to be validated in order to be woven back into the web.
December 1, 2009, 6:47 pmzuch says:
See this re the “lost data”. And the assertion that they can’t “duplicate their own results” seems to be your claim, not theirs.
Cheers,
December 1, 2009, 7:10 pmNobody At All says:
Not to get too far down into the weeds – You are mostly right about Popper. He was concerned with the distinction between science and pseudo-science, and made the observation that inductive confirmation is impossible, only inductive falsification. So, if people have a belief that is not falsifiable, they aren’t engaging in science, but pseudo-science. Here, it would be possible to falsify the beliefs held, with the data available; even assuming that the specific code adjustments could not be replicated.
More importantly, when philosophers asked the question “How does science proceed?” the Popperinian answer was that we proceed closer to the truth through a process of falsification. However, this depends on a finite number of testable theories (in order to falsify and dispose of the false ones, proceeding to the subset of true ones), and there are an infinite number of theories to explain our world. The relationship between this subjectivity and confidence under such a scenario is, in part, what inspired Kuhn, Quine, etc.
Climate science proceeds from both theory and observation. To disclaim any confidence upon any observations flies in the face of a theory, as well. One’s confidence in a particular dataset may be undermined by a lack of clarity into it’s reliability; the extent to which such uncertainty should diminish the reliability of the state of climate science as a whole, however, requires grappling with much else in the way of theory and observation. To this extent, again, the burden rests on the skeptic.
December 1, 2009, 7:15 pmzuch says:
I note that a lot of the “data” that Charles Darwin accumulated was actually drawings and notes of his observations of various creatures etc. The original specimens are of course long since gone [he did save some, but not all].
Maybe the creationists of the day should have insisted that he couldn’t reproduce his previous observations and wasn’t a real scientist. Had they done that back then, we wouldn’t have to put up with this annoying Eyvvullllution….
Cheers,
December 1, 2009, 7:17 pmRichard Aubrey says:
So once we’ve decided…what?…about the data, the lost data, the crummy codes, the agreement or non-agreement with something else, we’re left with another corpus.
December 1, 2009, 8:58 pmThat is, trying to hide cooling, discrediting skeptics, figuring out how to stonewall FOIA requests.
The second corpus should be enough to make the whole think stinky. In fact, it does, except for those with extraordinarily strong stomachs who suck it up for the team.
zuch says:
Richard Aubrey:
Once we’ve shown your first objection is ill-thought-out and bogus, we have to plow on with your next strident objections, eh? And on and on. And on and on. And while we’re at it, we have to insist that all bills be posted online for three days and read in their entirety on the floor of Congress before acting on them … oh, right, unless there’s an urgent need to put forth as many amendments as possible…. Where does it end? Do you wonder why people take umbrage and tempers get frayed?
Cheers,
December 1, 2009, 9:13 pmRichard Aubrey says:
zuch.
December 1, 2009, 9:29 pmYou haven’t shown squat. Assertions don’t count and references to congressional bills aren’t relevant.
And the second corpus does exist. Conclusions may be drawn.
Assertions to the contrary notwithstanding.
zuch says:
Projection?!?!?
As to my admittedly OT inclusion of the Republican “go slow, we need to study … oh, nevermind, let’s rush this through”, that’s just another example of conservative obstructionism (unless they need speed for political purposes) trying to make sure nothing gets in the way of their pillage and plunder while under a new Democratic administration and Congress. Same with the energy bill and health bills. And Kyoto or its progeny. It’s soooooooo predictable. And has not even a passing relationship or even a one-night-stand with science.
Cheers,
December 1, 2009, 11:00 pmSG says:
To disclaim any confidence upon any observations flies in the face of a theory, as well. One’s confidence in a particular dataset may be undermined by a lack of clarity into it’s reliability; the extent to which such uncertainty should diminish the reliability of the state of climate science as a whole, however, requires grappling with much else in the way of theory and observation. To this extent, again, the burden rests on the skeptic.
I think you’re attempting to steal a base here. Let’s be clear about what’s being received skeptically – It’s not skepticism over climate science per se, it’s the idea that man-made CO2 emissions are driving a positive feedback loop that will cause an environmental cataclysm in the medium term (next 50-100) years. That’s the positive claim that this data of uncertain providence and hazy reliability purports to show.
In the (absurd) limit, no one is skeptical that weather station X was at some temperature T at time Y. Few are skeptical that the planet is warmer now than it was at other points in its history (both human history and geological history). There is little argument about the physical properties of CO2. Personally, I’ve stated that I think it’s likely (but not proven) than man-made CO2 emissions have made the planet somewhat warmer than it would have been in their absence. Climate science as a whole is not under attack, although it’s claimed precision is as well as its predictive abilities.
I don’t think the AGW claim has been well established even accepting the CRU data at face value (“The fact is that we can’t account for the lack of warming at the moment and it is a travesty that we can’t.”). And I don’t think we’re under any obligation to accept at face value data from an organization that attempted to destroy the data (“Can you delete any emails you may have had with Keith re AR4? Keith will do likewise. [...] Can you also email Gene and get him to do the same? [...] We will be getting Caspar to do likewise.”) rather then let it be independently examined (and in violation of law to boot). Given that email alone, I think we’re entitled, even obligated, to question anything that has come from there.
Let me be clear, I don’t claim that anything has been disproven; I do think skepticism is the proper response at this point, though. Whatever the “truth” is, climate science will benefit from the added scrutiny. But right now, AGW’s proponents and defenders are making it seem an awful lot like a pseudo-science (in the Popperian sense).
December 1, 2009, 11:04 pmzuch says:
Ummm, the big brouhaha here is not over the models and the predictions, but rather on the past climatic data and the evidence for GW/AGW.
You want to start a discussion of the predictive models, ask the hosts to start a new thread.
Or if your complaint is not about the quality of the previous climate data and the present GW trend, out with it, and maybe we’ll have less noise here.
Let’s not change the subject, m’kay?
Cheers,
December 1, 2009, 11:49 pmSG says:
zuch:
Adults are talking – please keep quiet.
December 1, 2009, 11:51 pmzuch says:
I suppose e-mail is “data” in a more general sense, but isn’t this eliding the accusations of data destruction (an arguably more serious charge) into something actually less sinister. Do I really need to save every Nigerian scam letter?
Cheers,
December 1, 2009, 11:53 pmzuch says:
OK. That’s it. 4Q. And read between the lines.
Cheers,
December 1, 2009, 11:54 pmzuch says:
Care to do something other than substanceless ad hominem attack?
Cheers,
December 1, 2009, 11:56 pmNobody At All says:
SG:
My perception is that the arguments made by skeptics are generally much more sweeping both regarding theory (e.g. CO2/Water Vapor; Natural/Man CO2) and observation (e.g. little/no engagement with the published iterative methodologies for data adjustment). Whatever the case may be regarding the state of skepticism generally, addressing my remarks to you and your contentions may have been in error.
December 2, 2009, 12:50 amzuch says:
Nobody At All:
She sounds reasonable at times. Much better than most here. But she’s drunk the Kool-Aid when push comes to shove.
Cheers,
December 2, 2009, 1:51 amRichard Aubrey says:
zuch.
December 2, 2009, 8:17 amYou’ve been trying to have it both (all) ways for too long. You’re mixed up.
It’s the AGW folks who are trying to rush something through–see Copenhagen for starters.
The skeptics are asking that certain anomalies be explained first. Like it isn’t actually warming, for starters.
And then we find the source for much of the noise made over the last ten years can’t be trusted.
Skeptics are saying to slow down.
There’s enough money in this to cancel Copenhagen and still get the guys there next year, you know.
David Schwartz says:
Oh the irony!
I’m reasoning from the data I actually do have, and you’re speculating that the data we don’t have might contradict it, rendering my conclusions invalid. You are making precisely the same argument I am making (can’t trust the data we have as the data we don’t have might contradict it) and claiming it contradicts me.
The big difference — my argument is “I doubt we can’t prove with confidence there isn’t at least one bad thing we don’t know” while yours is “I believe we can prove with confidence there are no bad things we don’t know”.
December 2, 2009, 9:47 amzuch says:
Where ever did you get that? I don’t think I said any such thing.
Judging from what you said right above, I’d say some of your “data” is (most charitably) “interpolated” if not simply made up.
Cheers,
December 2, 2009, 11:20 amNewly-minted Skeptic says:
zuch:
The claim of an imminent catastrophe that must (not to mention can) be addressed by a massive, potentially world-impoverishing, intervention in the global economy is the SOLE reason for general interest in this subject, from skeptics and true believers alike. That IS the subject. SG is right, and you are weird. Cheers.
December 2, 2009, 1:04 pmJohn Moore says:
.
Yep – 2012 is really a scary problem, all right! Those Mayans were great scientists.
December 2, 2009, 8:28 pmzuch says:
Oh, I agree that alleged tax increases and increased costs for fossil products drives the RWers crazy and occasions many posts and comments in the blogosphere (including here). But the controversy at issue is the work and the data of the folks that were documenting existing GW. The complaints about the sufficiency of future predictive models had more to do with the physics/physical chemistry than with the historical temperature record (which is what the CRU is being Swiftboated for). Even Lindzer is not a GW sceptic; he claims the problem is with the models of future temperatures and what we can do about it.
Hope that’s clear.
Cheers,
December 2, 2009, 10:45 pmJohn Moore says:
The folks who were screwing up the documentation of the earth’s surface temperature history, you mean. BTW, not only is their evidence of fraud and suppression of dissent in their emails and code, but their particular area of focus is not a very good one for measuring climate change. Surface temps are subject to a wide variety of random and systematic errors, and thus even thermometer based readings are a pretty lousy proxy for he climate variable of global temperature (which is technically a measure of the thermal energy stored in and near the surface of the globe). Furthermore, the methods used to convert the proxy measurements (thermometry or tree rings) into useful global data (gridded average temperatures) are tricky, subject to conscious and unconscious bias, and require the use of a bunch of software that, from all evidence, is garbage. And, of course, CRU threw away the basic data they used to get the whole thing rolling (i.e. the information about how their gridded data was constructed from the raw data).
Ocean energy storage is far more diagnostic. Too bad it’s been going in the downward direction, isn’t it?
December 2, 2009, 11:00 pmzuch says:
Fraud? Really? Out with the “evidence”.
Assuming arguendo the truth of the latter statement, moving the goalposts here much? But it sounds like you have the makings of a good paper here … if you have the data to back it up. Go for it. Maybe a NPP in it for you. ;-)
Cheers,
December 3, 2009, 5:30 amJohn Moore says:
Zuch, it doesn’t take a paper to show that surface temps are not a great proxy. The reason the Argos project was undertaken is that ocean energy measurement is a much, much better way to get at the real warming/cooling signal.
You can go lecture the funders of that international project about how they need to convince you first that it is needed.
December 3, 2009, 5:47 pm