What is kept quiet is that the bulk of these FOIs were sent in an effort to establish the existance, or not, of documented confidentiality agreements between CRU and national meteorology offices.

The sequence was (1) request for temperature data (a) denied because of confidentiality agreements (2) request for all confidentiality agreements (b) denied as excessive (3) a project where each requestor asked for any confidentiality agreements between CRU and 5 specific nations (of a list of 150 or so.)

At this point CRU was able to find about three documents that — generously interpreted — matched the description.

No such animal exists.

If you want to reconstruct from raw data what the climate was in the United States and western Europe, you probably could.

If you wanted to use that to determine what the climate of the whole globe was, though, you’d have to pull that out of your fundament, like Jones did.

reasonable parsons:
I meant “speak truth to power”, of course.

No, you had it right the first time.

reasonable parsons:Real Scientists and decent people like pot meet kettle and I will speak power to truth until you Heed and obey! Gaia cries out!

I meant “speak truth to power”, of course.

Where they are concerned, pot meet kettle is right: Your job is to heed and obey! You are not entitled to see their data, and should not have access to it, and since you don’t have legitimate access to it, you have no evidence with which to question what they say!

Real Scientists are those who go along with the consensus, peer-review each other’s work to support the consensus! Anyone who dares to deny the Consensus is simply to be dismissed as a mediocrity, a crank in the service of corporations and an ideologue!

Real Scientists and decent people like pot meet kettle and I will speak power to truth until you Heed and obey! Gaia cries out!

What do you suppose they’d like to do in Copenhagen, and what they’re using to sell that to us? And what it’s going to cost us?

“he might be scrupulous about posting his data, but not scrupulous about actually using data properly. i don’t blame him. since he is a mediocrity, he has to massage the data to get anti global warming results so somebody will quote him and make him an expert.

mcintyre fail1

mcintyre fail2

mcintyre fail3″

It is interesting to me that all three hyperlinks go to RealClimate.org, which is the website run by the persons whose work McIntrye is criticizing. It is also a website that routinely blocks comments by persons who disagree with their position. McIntrye consistently replies to threads from realclimate.org at his own website. He replies on his own website because his comments are blocked at Realclimate, even when they are discussing his work.
Realclimate is where I go If I want to read a one-sided viewpoint expressed by AGW partisans. It has little value for anything else.

Eric Raymond, noted open-source Linux guru and author of best selling “The Cathedral and the Bazaar”, makes a convincing argument that climate science MUST be ‘open-sourced’. Link here … http://esr.ibiblio.org/?p=1436. In other postings on his blog, Eric is looking at the the leaked FOIA.zip source code, finding the emails a distraction from the underlying climate model programming. Check it out!

jccamp: Could anyone with the requisite background comment on the actual code and the much bandied-about “Harry Read Me” notes to himself from the last programmer (Is that correct?). Is it accurately described that entering almost any data set of random numbers — the ages of all the dogs on the block, numeric values for my great-grandparents’ middle names, etc — will still provide a solution that suggests fairly consistent global warming trends? Or is it more precise to say that no one really knows what the code might provide, since it is so unintelligible, but that it does deviate from computation and rely on unverified and untested tables for some questioned (and questionable) sub-sets of values (those pesky post-1960 things)?I confess to a certain shattered arrogance about this. I was confident I would be able to follow the technical discussion, but was quickly disabused of that impression.Addition/Edit: I neglected to add there may be more than my two suggested explanations. I would appreciate whatever is accurate.

RHSwan: Neurodoc, In the US, any data created or discovered by Federal Government funds (I’m not sure about the states, but it probably is similar) is by default public data and is available to anybody who wants it.

What authority do you have for that broad claim?

This document prepared in 2001 may be helpful here. Speaking of OMB A-110, a highly relevant document where federal research money is concerned, it says inter alia:

Under the recent revisions to A-110, FOIA reaches a grantee’s research data (a defined term in A-110) if it relates to published research results that were used by a federal agency to develop an agency action that has the “force and effect of law.” http://www.fptt-pftt.gc.ca/doc/owndata.doc

Laura(southernxyl): Neurodoc, it’s not just that they won grants, although I’d expect to see some provision for reporting what grant winners produce with government money.It’s more like a drug company offering a new drug for human use, and declining to share any data about it. “We did studies — take our word for it, the drug is very effective and safe. No, we won’t share the data. Take us to court, but in the meantime, we’re selling the drug.”

Sorry, but I don’t understand what you are saying either as to federal research grants or a drug company with a new drug for human use.

Do you know it know it to be the case that grant recipients are obliged as a condition of their grants to make public all the data that they collect in the course of their sponsored research save for that data which might be exempted under a FOIA exception, e.g., privacy of individuals?

Drug companies that want approval to marked a new drug need the approval of the FDA to do so, and to get that approval, they must turn over to the FDA all the data that the FDA requires of them in order to determine safety and efficacy, to decide on labeling and advertising claims, to monitor experience with the drug after it is on the market, etc.

After the year is up, the data are supposed to be released to all comers.

IMHO, there’s no reason for a researcher to release (say) tree-ring data he’s collecting, until he’s analyzed it and written a paper on it. The data should be released simultaneously with the paper’s publication, however. In a scientific field that depends on data-gathering rather than experiment, the data is everything. Results are not reproducible without the original data. If it’s not reproducible, it’s not science.

Klee12

What a load of pure, unadulterated poppycock. “Safety of the human race” was never at question here, these guys were more worried about the “safety of their exhaulted position and assurance of their cash stream”

If these “climate scientists” are so concerned about the human condition then they should release their data, code and methodology for full public peer review (not a behind-closed-doors review by their buddies either). Good science isn’t something that a small group of “smart guys” hoards and dispenses to the the rest of the world because they “know better and have a higher calling”.

How about a little humility and a good dollop of “I could very well be wrong”? Their numbers didn’t fit the decline which means that they still didn’t understand the modalities at work! All of this “rush out and save the world” BS is exactly that, BS. Just a smokescreen to cover up their very human traits of dishonesty, arrogance and selfish greed.

Robert Ayers: Much more difficult for result-replication will be all the data-dependent processing steps, like reproducing the CRU triangulation of the world’s reporting stations and myriad interpolations.

Tangentially, a paper that I found interesting: Tim Mitchell & Phil Jones, “An Improved Method of Constructing a Database of Monthly Climate Observations and Associated High-Resolution Grids (2005)” (warning: PDF).

I do not know whether this methodology was used, e.g. in the Harry_Read_Me.txt file, but interesting nonetheless.

(edited link.)

Let me add here that I thank “Nobody” for his conversation. And this small discussion (which I started) is really a sideshow. I agree it is likely that I could get “real as observed” (noon, whatever) values from the US stations with some digging and begging. Much more difficult for result-replication will be all the data-dependent processing steps, like reproducing the CRU triangulation of the world’s reporting stations and myriad interpolations.

Robert Ayers: For creating a time series, I’d settle for monthly averages of particular observation times: “January average of all 31 noon observations”. But that doesn’t seem to be on offer at the USHCN.

You’re right, it isn’t. To my best of my knowledge, however, climate data does not rely upon noontime readings rather than average daily temperatures, nor is there is reason to believe that the former is more indicative of that day’s temperature than the latter. (e.g. (a) a storm passes overhead at 11 am, suppressing sunshine, but is gone by 1 pm; (b) a storm passes overhead at 1 pm, suppressing sunshine, but is gone by 3 pm. Unclear as to why a noontime temperature would be a more reliable indicator of that day’s temperature.)

In any case, if you wish to create a time-series, the data is available.

Comments:

1. Yes, data in experiments with human subjects must be kept securely and confidentially at every stage. This is not relevant to the CRU data set.

2. I suggest that you turn your proprietary information question inside out: The CRU claim their data is proprietary. Does this implicitly define their commercial endeavor?

If you think about it that way, their stonewalling makes a lot of sense.

Their data set sounds like a nightmare, and it is normal fiduciary diligence for them to prevent any competitor to view it out in the sunlight, where all the makeup and cosmetic surgery is revealed in all its nastiness.

Just have a programmer at UCLA have a look through the HARRY_READ_ME.txt file. Every one of us number crunchers have toiled on complex programming projects and made our own pleading log files like the HARRY_READ_ME.txt log. It shows a mare’s nest of tinkering and finagling that cannot be unwound to get back to the initial data set. To a scientist, this is a severe fault, causing Harry’s heart to cry out in that log file. But CRU is not a small time egghead project — they are global players in this multi-trillion dollar Kyoto/Copenhagen scheme, spectacularly leveraged on this data set.

Over-leveraged, it looks like. That data set is now crumbling.

Robert Ayers: The above says that the temperature at station 145856 in January 1961 was 31.2. Since I used a file with “raw-avg” in its name, the README tells me that the value is “the average of un-adjusted mean monthly maximum and minimum temperatures”. (I could instead have chosen a “tob-avg” file: “the average of mean monthly maximum and minimum temperatures adjusted only for the time of observation bias”)

That all sounds fairly “raw”. And it is certainly much better than data with all manner of “adjustments”. But real “raw” would be like “the observation at station 145856 on 3 January 1961 at 12:02pm was 38.8″.

The FTP site offers me the choice of a mean monthly “minimum” or “maximum” or (as I read it) (min+max)/2. For creating time-series, I would really like to deal with a sequence of observed values (like the temperature as-observed every noon) rather than extrema.

That’s “real” raw data. I could be wrong (and happily retract, in advance, if I am so shown), but I don’t think that the lack of temperature data for every minute of January 1961 is due to scientists acting badly, but rather the technological limits of data storage in 1961.

Here’s GHCN data, here’s USHCN data, and here’s the GISTEMP source code; you may also be interested in the page of internet-accessible data sources compiled by the scientists.

I saw that UXHCN data before I posted. It is valuable. It is not as-observed data. The USHCN link that “Nobody” provides is to the FTP site ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/v2/monthly. A typical file at that FTP location contains data “rows”. Each row contains twelve monthly values for one US station and year. Like this: 14585631961 312 374 422 476 591 707 769 761 623 557 372 246 517

The above says that the temperature at station 145856 in January 1961 was 31.2. Since I used a file with “raw-avg” in its name, the README tells me that the value is “the average of un-adjusted mean monthly maximum and minimum temperatures”. (I could instead have chosen a “tob-avg” file: “the average of mean monthly maximum and minimum temperatures adjusted only for the time of observation bias”)

That all sounds fairly “raw”. And it is certainly much better than data with all manner of “adjustments”. But real “raw” would be like “the observation at station 145856 on 3 January 1961 at 12:02pm was 38.8″.

The FTP site offers me the choice of a mean monthly “minimum” or “maximum” or (as I read it) (min+max)/2. For creating time-series, I would really like to deal with a sequence of observed values (like the temperature as-observed every noon) rather than extrema.

Police Round Up Notorious Gang “The Hockey Stick CRU” in International Sting Operation

You’re missing the beauty of the scientific method – funding and motive are not relevant. Reproducibility and falsifiability are what’s important. If you make a positive claim, you have to provide enough information for others to be able to independently reproduce it. If you put forth a hypothesis to explain your data, it should generate testable conditions. This holds independently of what your funding source is, what’s your motivation in looking at the problem, etc.

Note that on the these grounds, a lot of climate science fails to make the grade.

I recollect reading NASA gent James Hansen’s testimony to congress, and thought that he presented some valid concerns.

We now have had enough time to see how his predictions panned out. To put it bluntly: they did not. Of his various scenarios, the one which most closely follows actual CO2 levels since then (Scenario B) showed a 1 C increase in earth temperature. Actual increase: 0 C.

The scenario which most closely follows actual temperature (scenario C) was a “zero increase in CO2 emissions” scenario — something which obviously did not happen.

In the late 1980s, Hansen predicted a rise in sea level of several feet, heat increases which could be felt, ubiquitous high winds, etc.

Summary. The predictions made by Hansen have failed. His predictions have erred in the pro-AGW direction. Conclusion: his predictions are not to be trusted.

MR: Those who you’ve read say that the ‘skeptics’ just want the raw station data are wrong or misleading. Raw station data is ubiquitous, what they are saying is that they want to know WHICH subset of data was used (it’s important) and the specific process used to create the final results.

When the term ‘Raw Data’ is used, typically, it is in reference to the raw data that was USED by GISS, CRU & whoever else. It does NOT meant in reference to the total superset of raw data available in the world. To put another way, USHSC’s raw data is not the same thing as GISS’s or CRU’s raw data and that is a critical difference.

This is not the way that “Raw Data” is used in the context of this thread (nor in Adler’s most recent thread). The three comments preceding yours (excluding mine) all precede on the basis of unavailable station data, as does the comment right after yours.

But, as you say: raw station-level data is ubiquitous. If someone wished to independently participate in the science of determining changes in measured temperature, a lack of data availability does not prevent them.

For the record, I’m not saying that GISS, or CRU, or anybody else’s calculations are wrong. I’m a big believer in the scientific process and believe that information should be made available so that other’s can verify the work. It’s too important not too.

We’re on the same page, here. Were I to quibble, I would point out that there is a large set of applicable literature by many authors, and that many authors do, in fact, defend their normalization, etc. procedures, and that this literature is largely in agreement as to the overall trend.

Legitimate scientists (published climate researchers not McIntyre) sent in FOI requests for the data.

The CRU refused saying “we have non-release agreements”

When FOI requests were made for the agreements the CRU stated that they couldn’t find the agreements ( actually some claptrap about how we moved our offices, never standardized how differnt departments file them and it would take more than 18 hours to find the agreements. The CRU could not even state who the agreements are with.

When asked for all the data that was available that would not violate the agreements with parties that could not be enumerated the response was just not possible (probably because they couldn’t filter the not to be disclosed to third party data out because they did not know which data belonged to the non-disclosure set, and they couldn’t find the agreemnts to find out what was and was not allowable…..

This is all documented on Climateaudit.org under the FOI topics in the frame to the left, I think this is the link http://go2.wordpress.com/?id=725X1342&site=camirror.wordpress.com&url=http%3A%2F%2Fwww.climateaudit.org%2F%3Fcat%3D53 but that site is pretty overloaded so I can’t verify that

There is a good post on Willis Eschenbach’s FOI Request, on camirror.wordrpess.com
http://camirror.wordpress.com/2009/11/25/willis-eschenbachs-foi-request/

The last I checked Gavin Schmidt & company at RealClimate.org are still spewing “non-disclosure” agreements as the reason the data can’t be released, and RealClimate.org isn’t saying who the “non-disclosure” agreements are with. I imagine that if asked the response would be:
We can’t say who it is we have an agreement, or we can’t find those.

We seem to be speaking past each other and I’m not sure how we can get past that. Let me put in a metaphor:

The scientific process includes being able to replicate results.

The argument that all the raw data station data is the equivalent of saying to a baker that all the raw ingredients to bake a cake are available in a grocery store. That statement would be correct, but not helpful.

However, you could not replicate the baker’s cake without knowing what ingredients were used to make the cake, what proportions were used, and how the cake was cooked.

Same with these temperature studies, we don’t know what stations were used, if enough stations were used in each grid to adequately replicate the varous microclimates, what adjustments were made, and how the data was baked into it’s finished product. (Yes, I avoided using the word ‘cooked’.)

On point 3:

Those who you’ve read say that the ‘skeptics’ just want the raw station data are wrong or misleading. Raw station data is ubiquitous, what they are saying is that they want to know WHICH subset of data was used (it’s important) and the specific process used to create the final results.

When the term ‘Raw Data’ is used, typically, it is in reference to the raw data that was USED by GISS, CRU & whoever else. It does NOT meant in reference to the total superset of raw data available in the world. To put another way, USHSC’s raw data is not the same thing as GISS’s or CRU’s raw data and that is a critical difference.

Horse. Beaten. Dead.

For the record, I could take the same raw station data, pick the stations I wish to use, adjust the numbers so they seem right to me, run my grid calculations, and “prove” the earth was cooling.

It would be a load of TOTAL BS, but without revealing what stations I chose, what adjustments I made, and what calculations I used…you couldn’t prove me wrong. Better still, the differences we’re talking about are miniscule in statistical terms so the final results would look very similar but with a slightly different slope.

And when pressed for details, I could just point to the larger superset of station data and say “my raw data is readily available from other sources”.

For the record, I’m not saying that GISS, or CRU, or anybody else’s calculations are wrong. I’m a big believer in the scientific process and believe that information should be made available so that other’s can verify the work. It’s too important not too.

Some of the things I have read lead me to believe that many of the journals that have such policies don’t vigorously enforce them. Perhaps if they did, it would put pressure on scientists to put pressure on the national meteorological organizations to be more liberal where such important research is concerned.

Robert Ayers: Aaron Bergman wrote: “All the NASA data is available, for example, and all their code is, too. I believe this is the correct link. http://data.giss.nasa.gov”
I visited the GISS three days ago to see if that was true.Based upon my trawling of GISS the site I believe that a) the original USA data, although not lost, is not obtainable at GISS or USHCN (see cdiac.ornl.gov); only “adjusted” data is available for download and b) algorithms and code is not available.
I wrote to the “contact” email address at USHCN on 26 November and asked them whether the raw station records and the algorithms/programs were available. We’ll see if they reply.

Here’s GHCN data, here’s USHCN data, and here’s the GISTEMP source code; you may also be interested in the page of internet-accessible data sources compiled by the scientists.