A. Criminal:
“Why most published research findings are false”

I love how misleading that article is. Any given study, yes, has an extremely low likelihood of gathering data that will produce a perfect, flawless, and universal model. So yes, from that perspective, most research findings are “false.”

But as Chem2L said, totality of data. A published paper might say “In A population, with B sampling methods, with C testing methods, conducted by researcher D, using analysis method E and conforming to definitions F and G, we found H.” The appropriate response to this is “Huh, that’s interesting” and to repeat the experiment in other labs, with other methods, in other cultures, under other conditions, longitudinally, to learn what biases are influencing the study and to eventually form an overall model.

So any given study will be “false,” but when you bring them together and conduct follows-ups, eventually an overarching theory may emerge. Developing that theory is the important part, not getting each individual study perfect.

Chem2L:
And don’t even get me started on the terms “environmental chemicals,” “natural,” or my personal favorite, “toxins.”I spared you my speech on that and you’re all happier for it, trust me.

I’m not sure I can trust you, Chem2L. This could be a really cool rant. Among other things, I want to compare notes with my own rant on “natural” and see what important points I’m missing.

Bad practice, skirting on unethical.

Scientists are people. They get invested in their position on a given issue. In fact, they are expected, once they reach a certain level in their careers, to have an opinion; strong opinions drive the debate forward, while weak ones are uninteresting. You have to fight for funding. Your reputation is the life of your laboratory and your institution, and all the people that depend on it, and becomes a valid end in itself. Under this pressure, it is difficult to maintain integrity – and by this I mean criticality – when your personal narrative, your social status, and the ability to continue your work are on the line. Recall the Hwang Woo-suk cloning controversy, and the historical case of Piltdown Man.

One of my former coworkers at a drug discovery company – a really bright and critical Ph.D., whom I still respect – once told me that he thought it should be all right for climate scientists to manipulate their data in order to better support the theory of global warming, because hey, the other side was clearly doing it, and the future of the planet was on the line. He wasn’t suggesting it was ok to falsify data, mind you, but to statistically interpret in a goal-directed way, such as through the strategic exclusion of outlying data points… was necessary. I was appalled. The choice of exclusion of outliers is an exercise that is performed in all scientific studies; outliers often indicate a limitation of the model or method, not a valid data point. You have to do it. It is part of being critical. It was his admission of bias as an active force in these choices that bothered me. I argued that the public had a right to the truth, even if it was not a compelling story. He agreed that the public had a right to the truth, but to a truth they could use – he argued that the public didn’t have the capacity to appreciate the totality of the data, and he had an obligation to represent what it was in terms they could digest, to speak for the data and its implications. We are both skeptics, in different ways.

These experiences tell me that when a scientist’s opinion and the data diverge, it is better to pay attention to the data, even if the only answer it gives is, “We really don’t know yet.” Though a viewpoint or theory may in fact prove to be correct, unsupported opinions should be actively identified and pilloried until the data can back them up. Credentialed plausibility is insufficient. Kristof is a layperson in this case, looking for answers that can be put to good use in living one’s life, on our behalf. We are all laypersons in most areas of expertise. But he is lending credence to suspicions that people will feel they have a right to rely on as fact. I think that’s wrong. I trust the big picture, not the charismatic leader.

Perhaps a different color of Kool-Aid, but there you have it.

And don’t even get me started on the terms “environmental chemicals,” “natural,” or my personal favorite, “toxins.” I spared you my speech on that and you’re all happier for it, trust me.

Kevin P. says:

You have a misunderstanding of peer review. Peer review does not involve proving or reproducing anything.

I do peer review every day. It is true that reviewers rarely repeat experiments. That makes peer review just the first step in scientific scrutiny. It is interesting that global warming deniers rarely get past even this screening process. If an interesting promising idea appears in a refereed journal, others will be motivated to test it further, repeat experiments, etc.

some additional criticisms

I suspect lots of researchers are more confident of their results than they can reasonably say in a formal presentation, such as a journal article.

Peer reviewed articles are sometimes wrong? Big deal. What should we do? Let first-year graduate students decide what’s worth publishing? Oh. Wait.

“Why most published research findings are false”
http://www.ncbi.nlm.nih.gov/pubmed/16060722

There’s not a small amount of irony in treating this article as gospel, without any discussion of the underlying assumptions, methods, or claims.

lgm
“Prove it or shut up” is peer review, with the extra feature that the person youprove it to is an expert in the field, not Glen Beck or some yokel hired by the Heritage Institute.

You have a misunderstanding of peer review. Peer review does not involve proving or reproducing anything. It is a review of a claim or article or publication without any attempt to reproduce the results independently. Any yokel could tell you this. Even Wikipedia has it generally correct: http://en.wikipedia.org/wiki/Peer_review

Peer review seems to be becoming increasingly irrelevant as a tool to winnow the good science from the bad.

It can hardly “become increasingly irrelevant” for something that in never did in the first place.

Far too many people confuse “peer review” with “independent confirmation of results”. The former really only answers the question, “Does this paper pass the laugh test” (i.e. will we end up embarrassed to have published it in our journal?) That’s all, nothing more. It has never (outside the mind of some clueless journalists) had anything to do with proving or validating the claims made.

It’s a lot worse than that (which is why ‘climategate’ didn’t surprise me):

“Why most published research findings are false”
http://www.ncbi.nlm.nih.gov/pubmed/16060722

Austism and climate are simiarly difficult cases because:
– they rely on historical measurements for comparisons.
– the measurements are difficult.
– there’s a lot of things to measure.
– it’s difficult or impossible to have proper control groups.
– emotional and political content and funding.

And yeah, most MSM accounts of science are bad.

Houston Lawyer: Since there are now benefits to be received for being diagnosed with many maladies, suprise, the number of cases diagnosed has increased. This is in addition to our ability to correctly diagnose illnesses that we could not diagnose in the past. Neither of these factors indicate that the number of people suffering from these maladies has actually increased.

For more information, try this: http://www.google.com/search?q=sowell+autism

Surely, we have all met someone who is generally non-crankish, but believes, e.g., that Obama was not born in the U.S., or the U.S. government was behind the 9/11 attacks.

Actually, I’ve found the beliefs that Obama wasn’t born in the U.S., and that the U.S. government was behind the 9/11 attacks, to be very reliable predictors of general crankishness. I don’t doubt there are exceptions, but no, I haven’t met any.

An article in a forthcoming issue of a peer-reviewed medical journal, Current Opinion in Pediatrics,

Publication in a “peer-reviewed” journal doesn’t mean that the article’s conclusions are necessarily well-supported. Dr. Andrew Wakefield, M.D., also has an impressive resume, and published his (in)famous article in a first ranked peer-reviewed journal, Lancet. In article, Dr. Wakefield was, likewise, “full of cautionary language”.

This sounds like a distressingly familiar pattern.

Jiffy says:
Both Kristof’s article and the abstract of the Landrigan paper to which it links refer to “proof of concept studies.” Does anyone know what that means?

The abstract you linked to answers that:

Summary: Expanded research is needed into environmental causation of autism.

Or, as Audrey said in Little Shop of Horrors:

Feed Me!

For a readible review of the language frequently found in abstracts and journal articles, I suggest A Layperson’s Guide to the Scientific Literature – Part 3 July 23rd, 2008, at
http://photoninthedarkness.com/?p=148 It explains the terms Meta-analyses, Pilot Studies, and Case Reports / Case Studies. A “Proof of Concept Study” looks like a type of Pilot Study — a study to determine whether the available evidence supports a more formal, better funded, study. (Keep in mind that studies that have negative results neither get published nor get additional grant fundings. So, there is a financial interest in finding something worth further funding to study even more).

Throbert McGee says:

Why are there more diagnoses of X?:

Erm, because all other factors being equal, parents would rather hear “Your child is autistic” than “Your child is retarded”?

Generally not true. However, if you want to maximize educational resources available for your SPED child, a lot more will be available with an ASD diagnosis than with a MR diagnosis. The additional services justify additional funding for the school and its teachers, aids, etc. So, actually, there is greater benefit for service providers for an ASD diagnosis than for parents. However, as the parent of an autistic child, I’ll confess that getting a diagnosis that increased the services he receives (in and out of school) isn’t something I discouraged.

Further, many people with ASD diagnoses are very intelligent. Today, Albert Einstein would likely receive an ASD diagnosis, and Bill Gates demonstrates many characteristics that could support an ASD diagnosis. Before autism was included as a diagnosis in 1980 in DSM-III-R (and the diagnostic criteria expanded significantly since then), lots of folks were diagnosed as “Geeks.” One of the rules for life that Bill Gates once laid out in a speech to some graduating High School Seniors was “Be nice to the Geeks in your class. In 10 years you’ve probably be working for one of them.”

For more information on the (alleged) autism epidemic, and the corresponding decrease in MR as a diagnosis, see Five Easy Graphs, January 28th, 2009
http://photoninthedarkness.com/?p=158 ; Another Perspective on the “Autism Epidemic”, October 26th, 2007 http://photoninthedarkness.com/?p=117 ; and,
Myths and Legends of Autism: Part 3 September 7th, 2007 http://photoninthedarkness.com/?p=107

The author is a research scientist with 2 Ph.D.s in different fields of biology, and the father of an autistic child.

SenatorX: Except it hasn’t.

Erg you might wish to check your source, all it disproves is that the hypothesis that children who actually were borderline autistic but not tested were skewing the numbers by being compared with those which were being tested. This just reinforces that it is indeed autism testers who cause autism, and not an artifact of who is and isn’t being tested.

(of course some might argue that autism specialists are more likely to establish themselves in areas where thereis a higher incidence of autism, reversing causality. certainly this is worthy of research as is discovering the mechanism by which autism testers cause autism, but as a precautionary measure until we can determine the truth we should do something to keep autism testers away from children.)

Impressive, no? She was still dead wrong about Bendectin. Don’t confuse a resume with insight on any given medical issue, especially when it comes to speculative issues of causation.

One time this doctor said he thought I might have diabetes. He totally turned out to have been wrong. Now I just ignore them all (white coats are no marker of expertise!), especially when it comes to speculative diagnoses.

It’s unfortunate that A. Zarkov (and David Bernstein) couldn’t have saved us all the effort of googling Dr. Landrigan and discovering that he does have a strong epidemiological background.

Here’s the web page of Dr. Shanna Swan, who was testified against all evidence that Bendectin causes birth defects. Among other things, “Dr. Swan is currently Professor in the Department of Obstetrics and Gynecology and Director of the Center for Reproductive Epidemiology at the University of Rochester School of Medicine and Dentistry.” Impressive, no? She was still dead wrong about Bendectin. Don’t confuse a resume with insight on any given medical issue, especially when it comes to speculative issues of causation.

If you look at reporting trends, it appears that increases in diagnoses are showing up in more educated communities (http://abcnews.go.com/Health/AutismNews/education-level-environment-tied-autism-clusters/story?id=9514773). Researchers believe that this is simply because parents with more education are more likely to have access to facilities to diagnose these disorders.

If you even look at pop culture from as recently as the 80′s, there are children portrayed as rambunctious, who would now be placed on the autism spectrum. (Look at the movie Parenthood, Steve Martin’s youngest son has some very telling personality traits.)

I’m not willing to say that environment does not have any affect on increased rates of autism, but certainly advancements in our understanding of human development has led to increased levels of diagnoses.

There is increasing concern that pollutants in our environment are leading to increased cancer rates, and various forms of birth defects and early-childhood-development issues. This is pretty clear. In some places, however, this isn’t just a hypothetical concern but a known risk. I lived downwind from Hanford for a few years, and it was well known that there was an area around where I was living that had a substantially increased risk of thyroid cancer for the elderly.

I guess I may be confused about Prof. Bernstein’s point. Is he arguing that pollutants are clearly not an issue and no cause for concern? Or is he saying anyone addressing anything relating to autism is a crank? If not, and he is willing to admit that there is a possibility that some of these things may be pollutant-caused, then is he agreeing with Kristof’s piece but uncomfortable saying so?

After all both seem to be saying “we don’t know for sure, but this is a real possibility….”

Matt: It’s unfortunate that A. Zarkov (and David Bernstein) couldn’t have saved us all the effort of googling Dr. Landrigan and discovering that he does have a strong epidemiological background.

Excellent. You did the work for me. Landrigan’s training is somewhat unusual for a pediatrician. Assuming his bio is accurate, he does seem well qualified to identify possible causes of childhood developmental disorders.

Matt:
Throbert understands neither mental retardation nor autism.

Google on “retardation”, “classification”, and “autism.” The hypothesis that people who in past decades would’ve been categorized with some non-specific cognitive impairment are now being statistically subsumed into the “Autism Spectrum” is not something I just pulled out of my ass to make conversation on VC. The possibility that this kind of diagnostic reclassification may partly explain the apparent rise in autism has caught the attention of some reputable researchers in the field.

If you want tarot-card reading, you go to Miss Cleo. She will know how to read tarot cards. He will not know how to read tarot cards, but he’ll know about things related to pediatrics and preventive medicine. Or is that too obvious to even state?

Apparently it’s NOT overly obvious that the doctor’s expertise in pediatrics and preventative medicine do not guarantee that he is a reliable expert on epidemiology or statistical analysis.

UPDATE: Whoops. I stand corrected. Thanks, Matt.

A. Zarkov: Pediatricians deal with the treatment of childhood diseases and developmental disorders. I don’t think their training includes much in the way of epidemiology, which tends to be heavily statistical. Dr. Landrigan might be very good at treating children and even offering advice to parents on how to avoid specific causes of childhood diseases, but how does that qualify him to do epidemiology?

It’s unfortunate that A. Zarkov (and David Bernstein) couldn’t have saved us all the effort of googling Dr. Landrigan and discovering that he does have a strong epidemiological background. From his bio page:

Specialty: Occupational Medicine , Pediatrics
Clinical Interests: Environmental Pediatrics; Occupational Medicine; Epidemiology
Education and Training MD, Harvard Medical School
M.Sc., London School of Hygiene and Tropical Medicine
Diploma of Institute Health, University of London
Residency, Epideminology, Center for Disease Control
Residency, Pediatrics, Children’s Hospital Boston
Residency, Medicine/Pediatrics, MetroHealth Medical Center
Awards 2009
Best Doctors
New York Magazine
2006
Lifetime Achievement Award
Children’s Health Environmental Coalition
2006
Children’s Environmental Health Champion Award
Environmental Protection Agency
2005
Health Achievement in Occupational Medicine Award
American College of Occupational and Environmental Medicine
2005
J. Lester Gabrilove Award
Mount Sinai School of Medicine
2003
David P. Rall Award for Advocacy in Public Health
American Public Health Association
2002
James Keogh Award
National Institute for Occupational Safety and Health
2002
Jacobi Medallion
Mount Sinai School of Medicine
2001 – 2006
Top Doctor. Preventive Medicine. New York Metropolitan Area and United States
Castle Connolly Ltd
2000
William Steiger Memorial Award
American Conference of Governmental Industrial Hygienists

Philip J. Landrigan, M.D., M.Sc., the Ethel Wise Professor and Chair of the Department of Preventive Medicine, is a pediatrician, epidemiologist, and internationally recognized leader in public health and preventive medicine. He has been a member of the faculty of Mount Sinai School of Medicine since 1985 and Chair of the Department of Preventive Medicine since 1990. Dr. Landrigan is also the Director of the Children’s Environmental Health Center.

Dr. Landrigan graduated from Boston College in 1961 and from Harvard Medical School in 1967. He completed an internship in pediatrics/medicine at Cleveland Metropolitan General Hospital and a residency in pediatrics at Children’s Hospital Boston. In 1977, he received a Diploma of Industrial Health from the University of London and a Masters of Science in Occupational Medicine degree from the London School of Hygiene and Tropical Medicine. He then served for 15 years as an Epidemic Intelligence Service Officer and medical epidemiologist at the Centers for Disease Control and Prevention (CDC) and the National Institute for Occupational Safety and Health (NIOSH). While at CDC, Dr. Landrigan served for one year as a field epidemiologist in El Salvador and for another year in northern Nigeria. He participated in the Global Campaign for the Eradication of Smallpox. Dr. Landrigan directed the national program in occupational epidemiology for NIOSH. He was awarded the Meritorious Service Medal of the US Public Health Service.

In 1987, Dr. Landrigan was elected as a member of the Institute of Medicine of the National Academy of Sciences. He served as Editor-in-Chief of the American Journal of Industrial Medicine and Editor of Environmental Research. He has published more than 500 scientific papers and 5 books. He has chaired committees at the National Academy of Sciences on Environmental Neurotoxicology and on Pesticides in the Diets of Infants and Children. The NAS report that he directed on pesticides and children’s health was instrumental in securing passage of the Food Quality Protection Act, the only environmental law in the United States that contains explicit provisions for the protection of children. From 1995 to 1997, Dr. Landrigan served on the Presidential Advisory Committee on Gulf War Veteran’s Illnesses. In 1997-1998, Dr. Landrigan served as Senior Advisor on Children’s Health to the Administrator of the U.S. Environmental Protection Agency and was instrumental in helping to establish a new Office of Children’s Health Protection at EPA. From 2000-2002, Dr. Landrigan served on the Armed forces Epidemiological Board. Dr Landrigan served from 1996 to 2005 in the Medical Corps of the United States Naval Reserve. He retired in 2005 at the rank of Captain. He continues to serve as Deputy Command Surgeon General of the New York Naval Militia, New York’s Naval National Guard.

Dr. Landrigan is known for his many decades of work in protecting children against environmental threats to health, most notably lead and pesticides. His pioneering research on lead toxicity at low levels persuaded the US government to mandate removal of lead from gasoline and paint, actions that have produced a 90% decline in incidence of childhood lead poisoning over the past 25 years. Dr. Landrigan has been a leader in developing the National Children’s Study, the largest study of children’s health and he environment ever launched in the United States. He has been centrally involved in the medical and epidemiologic studies that followed the destruction of the World Trade Center on September 11, 2001. He has consulted extensively to the World Health Organization.

Pediatricians deal with the treatment of childhood diseases and developmental disorders. I don’t think their training includes much in the way of epidemiology, which tends to be heavily statistical. Dr. Landrigan might be very good at treating children and even offering advice to parents on how to avoid specific causes of childhood diseases, but how does that qualify him to do epidemiology?

If I as your peer look at your datasets and confirm that it meets the criteria for good science, there is value in a peer reviewing.

If peer review involved looking at datasets, I’d be more impressed by it.

People may not like where that leads.

Based on what I’ve seen, read, and personally experienced, the possibility exists that exposure to too much Oxytocin during development may increase the risk of Autism.

Kevin P.:
Performing a review of someone else’s article isn’t proof.Having someone else being able to reproduce the work by performing their own experimentation or empirical work is proof.Peer review may still be useful but it seems to be degenerating into: “Yeah, it looks good and smells good, sounds good to me!”I wonder if this has always been the case, and if only recent politically charged scandals have brought this inherent weakness into the light of day.

Um, reviewing your proof is good science on my part as well. If I as your peer look at your datasets and confirm that it meets the criteria for good science, there is value in a peer reviewing. I agree that running a new series of experiments with my own data, controls, and variables is better than peer review, but peer review absolutely has value.

I also agree that there is real danger from consensus-science-by-commitee ala the climategate folks where actual results get suppressed as a result of bad scientists and grant-challenged administrators.

Yes – considering that there has been a long line of unfounded fears about environmental “toxins”, chemicals, power lines etc. causing any number of ailments, and many (if not most) of those promoting these fears have been cranks with PhD’s. The recent brouhaha about vaccines is a good example. Maybe this particular person is totally sincere and totally correct. Unfortunately for him, the atmosphere for debating these things has been poisoned by the cranks that came before him. I’m going to want to see a whole lot of evidence and replication by other researchers before I take him out of the “presumed crank” category.

Matt: Of course, what Bernstein doesn’t mention is the long line of mounting evidence that environmental toxins may be contributing to brain damage in the developmental phases of life. Perhaps, and just PERHAPS, that is why Dr. Landrigan expresses a stronger view to Kristof than that concluded in his piece.

From that: “It also revealed that no more than 56 percent of the estimated 600-to-700 percent increase, that is, less than one-tenth of the increased number of reported autism cases, could be attributed to the inclusion of milder cases of autism. Only 24 percent of the increase could be attributed to earlier age at diagnosis.”

Unless they wrote that in a horribly poor fashion, it sure looks like they’re saying that 80% of the “increase in autism diagnoses” might be related to “chemicals or microbes”.

That’s not a strong indication that we should be worrying about “toxins” (?) causing autism.

Whenever someone jumps to handwaving about unspecified “toxins” causing something, it’s an instant alarm bell – good scientists won’t call it a “toxin” until they’re sure what it is and that it’s actually causing a toxic effect rather than causing autism as a side effect maybe sometimes somewhere. (That being a negative effect, but not obviously a toxic one.)

(I also would not assume that a Chair at a “Prominent School” (appeal to authority?) is necessarily well-versed in the totality of the research “landscape” – or that being a Chair somehow makes one immune to unconscious cherry-picking or other biases. Because it doesn’t.)

Steve: There is nothing particularly strange or even remarkable about a scientist whose experiments are consistent with his hypothesis becoming increasingly confident that his hypothesis is correct.

Of course there isn’t. But why should we care? His personal confidence is utterly irrelevant. Science is not about handing down wisdom from a priesthood in lab coats; it’s about research. If he has research which strongly confirms his theory, he should publish it. If he hasn’t done so, then it’s just bloviating. (Not that there’s anything wrong with him saying it, but why is Kristoff publishing it as though it had meaning?