BACKGROUND
When the Heller case came out, Carl Bogus pointed to a “careful study” by Colin Loftin, et al., (325 New Eng. J. Med. 1615 (Dec. 5, 1991)), showing a huge drop in homicide resulting from the DC gun ban. I cited criticisms of that study. Tim Lambert responded on his own blog, apparently thinking that I had been unfairly hard on the Loftin study.
Now that I’ve had a chance to look at some of the DC data, I think that in my original post I was much too easy on the Loftin study. There are even more problems with the Loftin study than I originally thought.
In response to Bogus’s endorsement of the Loftin study, I cited an analysis by Dean Payne:
According to Loftin’s numbers, adjusted to a per-capita basis, the District’s post-ban benefit vanishes altogether. Its proportionate rate reductions are smaller than those achieved by its neighbors. . . .
Loftin’s report dismisses a number of confounding factors, but fails to present adequate justification for doing so. Despite claims to the contrary, the presented measure of lives saved by the District’s restrictive handgun policy is structured such that it is inherently contaminated by:
– lives saved by a region-wide drop in homicide and suicide rates from other causes, affecting both study areas;
– lives saved by the population exodus from the District;
– killings in which non-firearms means were substituted for firearms.
My analysis suggests that essentially all of the benefit perceived by Loftin is the result of this or similar contamination.Finally, the study period ends in 1987, just as Washington DC began suffering a continuing homicide wave that earned it the dishonor of being the Murder Capital of the United States.
LAMBERT’S DEFENSE
Tim Lambert defended the Loftin study. First, Lambert claimed: “the study did not conclude that the reductions were caused by the ban since it is possible that some other factor was the cause.”
Actually, I think Lambert is wrong here. The Loftin study rejected other explanations as implausible, concluding without any qualification that “the law reduced gun-related suicides and homicides substantially and abruptly.” The study also claimed, “The data from the District of Columbia provide strong evidence that restrictive licensing of handguns reduced gun-related homicides and suicides” in DC, but not necessarily in other jurisdictions.
Second, Lambert wrote:
While the numbers given in Loftin’s paper were for numbers of homicides and suicides rather than rates, Loftin et al state that they get similar results if age adjusted per-capita rates are used. (This corrects for change in the age structure of the population as well as its size.) The results for age adjusted per-capita rates were given in [unpublished] supplementary material.
Lambert then presents these rates, which were directly standardized for a control variable, age.
But the claim I endorsed was about the necessity of using homicides rates, rather than counts, without making any claims about other control variables, such as age. Lambert does not present any Loftin data on rates without age adjustments, so I have no reason to think that Payne was incorrect in the claim that I cited.
THE FAILURE TO CONTROL FOR RACE
So what’s wrong with adjusting rates for control variables like age, gender, and race that might affect the conclusions? Nothing at all; indeed, the failure to control for major demographic changes is one of the two chief defects of the Loftin study. Again, it’s embarrassing that a scholarly journal would publish a study on changes in homicides without adjusting for important demographic changes in DC, in particular race.
Let me explain: African Americans have much higher homicide victimization rates than Whites (nationally, about 5-7 times as high in the 1976-87 period). According to the Current Population Survey data, in the pre-gun ban period, there averaged about 97,000 more African Americans in DC than in the post-ban period. That 18% drop in African Americans between periods should be associated with a drop nearly that large in homicides in DC, which needed to be controlled for, but it wasn’t in the Loftin study.
This one error by itself could account for the results reported by Loftin et al. Indeed, had Loftin controlled for race, it is likely that the DC gun ban would have been associated with a statistically significant increase in homicides, rather than the opposite.
CHOICE OF PERIODS
The other big problem with the Loftin study is the choice of periods. As Payne noted, the study period ends in 1987 just as homicides in DC are exploding. DC homicides went from 147 in 1985 to 194 (1986) to 225 (1987) to 369 (1988) to 434 (1989) and to 472 in 1990. If data through 1990 (or even 1989) had been included in the post-gun ban period (the study appeared on December 5, 1991), the reduction effect reported would have disappeared and almost certainly the opposite association (increased homicide) would have been shown.
In other words, the results reported in the Loftin study are merely artifactual. They are artifacts of the failure to control for racial demographic changes and the choice of study periods. Making either of these changes should have been sufficient to eliminate or reverse the direction of the effect reported by Loftin et al. in their paper (whether expressed in counts or rates or age-adjusted rates).
DATA NOT WELL BEHAVED
Another serious problem, one that may prevent any persuasive time series analysis, is that the DC data are not “well behaved.” The huge increase in homicides in the late 1980s is probably not the result of the gun ban (but rather crack cocaine and gangs), yet without looking at comparable major cities (which the Loftin study does not do), it is impossible to get even a rough sense of whether Washington, DC, fared worse than other cities without gun bans.
Yet that is not the only troublesome pattern in the data. After peaking in the 1971-74 period at an average of 266 homicides in DC, homicides dropped precipitously to 235 in 1975 and 188 homicides in 1976 (the law banning guns took full effect after legal challenges on Feb. 21, 1977; it was also enforced for over two months in 1976: September 24, 1976 – December 8, 1976).
Thus, there was already a strong downward mini-trend in force when the gun ban took effect. Indeed, for the first six years of the gun ban (1977-1982), in only one year (1979) did the number of homicides fall below the 188 homicides recorded in 1976, a year mostly before the ban took effect Is this mini-trend real or just artifactual? If it’s real, then the gun ban might have fairly quickly put an end to a then recent, but strong downward trend in the data. But one can’t really tell whether this mini-trend is real, even by looking at monthly data. What all this suggests is that the often-difficult decision about what curved shape to fit to time series data can determine whether one gets an upward or downward deviation from the trendline.
Actually, I think that, until the mid-1980s, the DC homicide counts fairly nicely fit the numbers of African Americans in DC, rather than Loftin’s gun ban hypothesis. As homicides jump substantially from 178 in 1967 to 266 per year in the 1971-74 period, the number of African Americans in DC makes a similar jump from 385,000 in 1967 to 583,000 in the 1971-74 period, thus increasing the population most highly victimized. By 1976, the number of homicides has fallen to 188 while the number of African Americans has also fallen substantially to 444,000 (remaining within 36,000 of that figure through 1987).
CONCLUSION
In short, I was too easy on the Loftin et al. study when I blogged about it in June.
The reported results are merely artifacts of serious failures in its research design. The failure to control for race and the choice of time periods to study entirely drive the direction and the strength of the reported effects. Unfortunately, however, the data are ultimately not well enough “behaved” to justify a full, careful time series analysis that might persuade a cautious researcher of the probable real effect of the DC gun ban – whatever that might have been.
UPDATE: Carl Bogus responds to my first post on this issue here.