In Reviews in American History the country’s leading historian of homicide, Randolph Roth (OSU) has an article that deals with two recent disputes in American history: how many guns were listed in early American probate records and whether the Old West was violent (I dealt with the latter issue in an earlier post). On these two related issues, revisionist historians had suggested that there were few guns and relatively little violence.
Roth first indirectly addresses a question that puzzled me. How could historians be so easily misled about the quality of Michael Bellesiles’s fall 2000 book, Arming America?
Roth writes (link may not work for many non-university readers):
As historians, we take pride in our openness to new ideas. That is why, I think, so many of us were enthusiastic about Michael Bellesiles’s Arming America. Its thesis—that few Americans used, owned, or cared about guns before the mid-nineteenth century—was certainly congenial to scholars who supported gun control, because it suggested America’s homicide problem was caused by an increase in gun ownership and the creation of a “gun culture.” But Arming America had a more fundamental appeal for historians. It proved that they could make important discoveries, that they could confound received wisdom with bold hypotheses and careful research, and that they could play a crucial role in public life. Unfortunately, Arming America was wrong about early America.
Why do we as a profession have trouble deciding which quantitative studies are sound and which are not? Why do we mistake faulty studies for good ones and vice versa? Like everyone else, including statisticians, we are not very good at judging probabilities. Humans are proficient at many things, like counting, but as psychologists have discovered time and again, we have trouble figuring odds or risks or rates on the fly. Our common sense is a hindrance. That is why we need tools to help us determine which quantitative studies are wrong and which are right. Thanks to statisticians, those tools are at our disposal. It’s simply a matter of getting acquainted with them in a statistics course, keeping a few textbooks at hand, and thumbing through them occasionally for formulas that can help us answer quantitative questions.
Bellesiles claimed, for instance, using evidence from probate records, that only 15 to 21 percent of the wealth-holding population owned guns in the late colonial and early national period [even that estimate was an increase over Bellesiles’s original estimate in Arming America of 14.7%]. But James Lindgren and Justin Heather, who reanalyzed a sample of probate records gathered by Alice Hanson Jones, found that half of all wealth-holders had guns in 1774. Who is right? It takes only a few minutes to decide, once the data are in hand and the right formula is found.
The historical consensus today, even among many of Bellesiles’s former supporters, appears to be that Bellesiles was wrong on most of his main points about guns in early America (see Fall From Grace). Yet I still occasionally encounter people who believe that, though Bellesiles may have made some major mistakes or even may have acted improperly, his significant errors were limited to the probate data, leaving his primary thesis of very few guns and little or no gun culture in early America basically intact. Yet in two widely read articles (one co-authored with a student, Justin Heather), I detailed hundreds of mistakes in Arming America that significantly undercut nearly all of the major claims in the book.
Most historians just found it hard to believe that one of their own had made such basic errors, especially once the book had been praised by some of the country’s best historians in the New York Times and the New York Review of Books (both reviewers later changed their opinions). And because we had used sophisticated statistical techniques, some historians were slow to realize that the errors we identified were easily identifiable in the records and the raw data itself.
I urged that, before entering the public discussion, historians should spend an hour in a good research library checking the published probate records of Providence that Bellesiles had used for some of his counts. They could quickly see that nearly every claim he made about those records was false (of course, we did not expect historians to go into the many unpublished archives to read the original probate records we also analyzed for other times and places). Although Bellesiles viciously attacked our work on the Providence records in the Wall Street Journal and online, he then quietly capitulated on almost every one of our points regarding Providence, removing most of his mistaken statements from his book in 2001.
In his 2007 article, Roth recounts guns in one of the sources we used, the 1774 inventories published by Alice Hanson Jones:
Nearly half of the 919 estates in her sample included guns, even though at least 28 inventories in her sample were incomplete, and 81 estates were owned by women, 13 of whom owned guns. . . .
The formula shows that there is a 99% chance that the actual proportion of wealth-holders who owned guns (π) was between .600 and .462. There is only a 1 in 200 chance that the proportion of wealth-holders who owned guns was below 46.2 percent. Lindgren and Heather are right: probably half of all wealth-holders owned guns.
Assuming that Jones’s sample was properly executed—and it has been verified by subsequent scholars—how likely is it that the actual proportion of wealth-holders who owned guns was only 15 to 21 percent, as Bellesiles claimed? . . .
Bellesiles’s estimate of 21 percent is 12.0 standard deviations below the sample proportion (53.1 percent) and his lower estimate of 15 percent is 14.2 below. That is an enormous distance, statistically. The chance that Bellesiles’s estimates are correct is nil—less than 1 in a zillion-zillion. . . .
Thanks to these statistics, we can reject Bellesiles’s findings with confidence. We do not have to examine his methods or his selection of counties. His estimates of gun ownership are impossibly low.
To someone with even a modicum of statistical sense, Bellesiles’s estimates were mathematically impossible.
It is gratifying to have our work replicated and confirmed, an act for which I heartily thank Randy Roth.
By the way, two participants in the debate over Arming America (Clayton Cramer and Saul Cornell) have published books in the last 15 months.