“Guns Did Not Protect Those Who Possessed Them from Being Shot in an Assault.”

So reports a press release posted at ScienceDaily.com, which also says,

The study estimated that people with a gun were 4.5 times more likely to be shot in an assault than those not possessing a gun.

“This study helps resolve the long-standing debate about whether guns are protective or perilous,” notes study author Charles C. Branas, PhD, Associate Professor of Epidemiology. “Will possessing a firearm always safeguard against harm or will it promote a false sense of security?” …

Penn researchers investigated the link between being shot in an assault and a person’s possession of a gun at the time of the shooting. As identified by police and medical examiners, they randomly selected 677 cases of Philadelphia residents who were shot in an assault from 2003 to 2006. Six percent of these cases were in possession of a gun (such as in a holster, pocket, waistband, or vehicle) when they were shot.

These shooting cases were matched to Philadelphia residents who acted as the study’s controls. To identify the controls, trained phone canvassers called random Philadelphians soon after a reported shooting and asked about their possession of a gun at the time of the shooting. These random Philadelphians had not been shot and had nothing to do with the shooting. This is the same approach that epidemiologists have historically used to establish links between such things as smoking and lung cancer or drinking and car crashes.

This was promptly echoed in the Philadelphia Daily News.

Conspicuously missing from the press release and the news story were two critical limitations that were admitted in the original study. These qualifiers mean that the press release headline, as well as all the other statements and implications of causation, were quite mistaken. Perhaps defensive possession and carrying of guns helps protect the possessor and carrier, and perhaps it doesn’t. But the study sheds virtually no light on the subject.

1. To begin with, there’s the obvious causation/correlation problem. Maybe, as the authors speculate, carrying a gun increases your chances of being shot with a gun (as suggested by the framing of the issue as “whether guns are protective or perilous”), or at least fails to decrease them (“guns did not protect”). Or maybe a third source — perhaps some people’s being the targets of death threats, or being in a dangerous legal line of work, or being gang members or drug dealers — causes both higher gun carrying among those people and higher risk of being shot.

By way of analogy, we don’t suggest that pacemakers cause heart attacks, or don’t protect against heart attacks, just because we find a correlation between the presence of pacemaker and the incidence of heart attacks. Obviously, people might get pacemakers precisely because they’re at risk of heart attacks. Well, people might get guns precisely because they’re at risk of attack. (Stewart Baker makes a similar point.)

One can try to control for this in some measure — but while the study controls for some relevant attributes (race, sex, age, neighborhood, having a “high-risk occupation,” and having at least one arrest on one’s record), it leaves a vast range of factors uncontrolled. You’d think that gang members are more likely than others to carry guns and to get shot, even controlling for the presence of an arrest record. (Lots of law-abiding people carry guns, but I expect that more gang members do.) But the study doesn’t control for that, or for many other things.

Let me illustrate this with a deliberately oversimplified model. Let’s begin by assuming a total population of 100,000, that’s divided into two groups, a 10% high-risk group and a 90% low-risk group. Let’s say that the high-risk group has a 60% risk of being attacked, and as a result 40% of its members carry guns. And let’s say that the low-risk group has a 5% risk of being attacked, and as a result 3% of its members carry guns. Let’s also imagine a total population of 100,000 (just to make the numbers easier), and let’s assume that possessing a gun has a modest protective effect for both groups — it reduces the risk of being injured when attacked from 75% to 60%.

Here’s what this turns out yielding, with “A” meaning “armed subgroup” and “U” meaning the unarmed subgroup.

The result: The armed subgroup has 3.5 the risk of injury compared to the unarmed subgroup, and the relative odds ratio between them is 4.29. And this is so even though in the model gun possession decreases the injury risk for both the high- and the low-risk group.

Naturally, this is just a model; the real numbers are likely very different from the ones I give here, and in fact no-one knows what the real numbers are. (The model also doesn’t precisely fit the numbers in the study, though I’m pretty sure one can make a similar model that would fit them more closely.) My point is that one just can’t infer from an odds ratio of over 4 to the judgment that “guns did not protect those who possessed them,” much less that they were actually “perilous” to the possessors. The high odds ratio is just as consistent with the model I describe as with a model where gun possession increases the risk of injury.

2. But wait, there’s more. The research model works only to the extent that you actually know who possesses guns and who doesn’t. Both the cases (people who were shot) and the controls (people who were called on the phone) might want to conceal their gun possession. The cases might have thrown away their guns before the police arrive (sometimes easy, sometimes hard or dangerous). The controls might have lied to the stranger who calls them to ask them, “Where were you at 10:30 pm two nights ago?,” and “Were you possessing a gun at the time?” (always easy and safe).

And both the cases and the controls might have plenty of reasons to lie. They might have been possessing guns in public without a concealed-carry license. They might have been felons who didn’t have the right to possess a gun even at home. (People with arrest records made up 53% of all cases and 37% of all controls; the study doesn’t tell us how many had felony conviction records, but I suspect that quite a few of those with arrest records did.) Or they might not be sure what the questioner is getting at. And that’s true even if the questioner claims that he’s just an academic researcher.

Fortunately, the study helpfully tells us what would happen if there’s concealment of gun ownership by some fraction of cases and controls — though of course the press release and the newspaper article are silent about this. If only 1% of controls and cases who are reported not to have had guns are randomly recoded to having guns, two of the three results (“all gun assaults,” “gun assaults where victim had at least some chance to resist,” but not “fatal gun assaults”) end up yielding statistically insignificant results. If 3% are so recoded, all three results lose statistical significance.

If we assume that 1% of controls were concealing their gun possession and 0% of cases were concealing it — not implausible, since it’s easier for a control to conceal gun possession than for a case to do so (since the cases may be too injured to get rid of the gun, may need a gun for continued self-defense, and in any event are the subjects of a police investigation in which the police might learn the truth) — all three results lose statistical significance. The numerical value of the non-significant odds ratios falls as well; if 5% of controls conceal their gun possession buy 0% of cases do, the odds ratio falls to 2, which of course reflects a considerably lower relative risk. And these are pretty low percentages of false reporters, given the incentives that many people might have to hide their gun possession.

And all this is in addition to the possible confounding factors discussed in item 1 above. If there were no such confounders, then perhaps even a low odds ratio might be telling, or perhaps even a statistically insignificant odds ratio above 1 might in some measure undermine the “guns as protective” theory. But these two problems put together — the possibility that the result stems from the existence of a high-risk group whose members are especially likely both to carry guns and be the targets of attack, and the possibility of even slight misreporting dramatically affecting the results — make the study highly uninformative.

So it’s possible that gun possession was “perilous,” in the sense of increasing the risk of the possessor’s being injured. It’s possible that it “did not protect those who possessed guns,” in the sense that it didn’t reduce the risk of the possessor’s being injured. But it’s also possible that it was “protective,” in that it reduced the risk of the possessor’s being injured, but this result is swamped by the other phenomena I point to. The study doesn’t give us much extra information about which theory is correct. And yet it is publicized, and it’s reported, as if it did robustly show the causal relationship.