On Mental Health Commitments and the Virginia Tech Shooting:

Was the tragic incident at Virginia Tech the result of a failure of Virginia’s mental health system? Slate recently posted Seung-Hui Cho’s commitment papers and they are revealing: the magistrate who heard Cho’s case determined that he was “an imminent danger to himself as a result of mental illness,” but determined that there were “alternatives to involuntary hospitalization.”

After the shooting, Sally Satel at AEI argued that Virginia needs to reexamine its involuntary treatment laws and adopt a lower threshold for commitment, more in line with states like Arkansas and Hawai’i. Others, like Brian Jenkins at RAND, contend that the tragedy probably could not have been prevented and might not have a solution.

It’s impossible to make sense of the debate, though, without understanding the extent to which we’ve dismantled our mental health system in this country. Brick-by-brick, cell-by-cell, we deconstructed what was once a massive mental hospital complex and built in its place a huge prison.

The sheer magnitude of transformation is absolutely remarkable. It is visually shocking — especially given the exponential rise in incarceration in this country in the past thirty years. The following figure gives a snapshot. It shows the aggregate rate of institutionalization in the United States for the period 1934 to 2001, with disaggregated trend lines for mental hospitalization on the one hand and state and federal prisons on the other.

FIGURE: Rates of Institutionalization for Residents in All Mental Institutions and State and Federal Prisons in the United States (per 100,000 adults)

Although our current rates of imprisonment in state and federal prisons are extraordinarily high, the level of total institutionalization (in prisons and asylums) was higher during the 1940s and 50s.

We all know that prison populations skyrocketed nationwide beginning in 1970, rising from under 200,000 persons to more than 1.3 million in 2002. That year, in 2002, our prison rate exceeded for the first time the 600 mark (600 inmates per 100,000 adults) — by far the highest rate and raw number of incarcerated persons in the world.

But what is far less well known is that the United States as a whole institutionalized people at even higher rates in the 1940s and ’50s. If you look at rates of persons in mental hospitals and prisons per 100,000 adults, in the period between 1935 and 1963 the United States consistently institutionalized at rates above 700 per 100,000 adults — with highs of 778 in 1939 and 786 in 1955.

In a recent study, I collected state-level data on mental hospitalization for the twentieth century, and my findings are staggering. For one thing, there was a wide range of institutions. In addition to state and county public mental hospitals, there were public and private institutions for “mental defectives and epileptics” and for “the mentally retarded,” psychiatric wards in general and VA hospitals, “psychopathic hospitals,” city hospitals, and private mental hospitals.

There was also an entire parole system for persons institutionalized in mental hospitals and the parole numbers were non-negligible: on December 31, 1933, for example, 46,071 mental patients were on parole or otherwise absent, representing almost 10 percent of the total institutionalized patient population of 435,571.

One of the most perplexing things I discovered is that there is a strong statistical relationship between aggregated institutionalization (in asylums and prisons) and serious violent crime. Using state-level panel regressions spanning the entire period from 1934 to 2001, including all 50 states, and controlling for economic, demographic, and criminal justice variables, I found a large, robust, and statistically significant relationship between aggregated institutionalization and homicide rates. The findings are not sensitive to weighting by population and hold under a number of permutations, including when I aggregate jail populations as well.

I say “perplexing” because the populations in asylums and prisons were very different: residents in mental hospitals were more white, female, and older than our current prison populations. In 1966, for example, there were 560,548 first-time admissions to mental hospitals, of which 310,810 (55.4%) were male and 249,738 (or 44.6%) were female. In contrast, new admittees to state and federal prison were consistently 95% male throughout the twentieth century. There were also sharp differences in racial and age compositions.

Because of these sharply different populations, it’s not clear yet what to conclude from my study — and it’s far too early to draw public policy implications. But a few things are clear.

The first is that we should not be surprised that there are so many persons with mental illness behind bars today. We deal with perceived deviance differently than we did in the past: instead of getting treatment, persons who are viewed as deviant or dangerous are going to jail rather than mental hospitals.

The second is that we should not be surprised that our mental health systems are in crisis today. The infrastructure is simply not there. This is evident in states across the country where persons with mental illness are being housed in jails rather than treatment facilities.

What is also clear is that Seung-Hui Cho probably would have been institutionalized in the 1940s or 50s and, as a result, the Virginia Tech tragedy may not have happened. According to the New York Times, the director of the campus counseling services at Virginia Tech said of Cho: “The mental health professionals were there to assess his safety, not particularly the safety of others.” It’s unlikely we would have taken that attitude fifty years ago.

But the problem is, we would also be institutionalizing another huge swath of humanity — and it’s simply not clear how many of those other lives we would be irreparably harming in the process.

The classic texts of social theory from the 1960s — Erving Goffman on Asylums, Gerald Grob on The State and the Mentally Ill, David Rothman on The Discovery of the Asylum, Michel Foucault on Madness and Civilization, Thomas Szasz on The Myth of Mental Illness, Michael Ignatieff on A Just Measure of Pain, and many others — describe in chilling detail how closely mental institutions began to approximate the prison and the “total institution.” These critical writings should remind us of the other price that society pays when we commit and institutionalize each other.

Mental Hospitalization and Prison Rates in Western Europe:

In response to my first blog entry here, some readers asked how the rates of institutionalization in the United States compare to Western Europe. There is no doubt that the graph of institutionalization over the twentieth century immediately raises the question whether Western or industrialized countries with currently low prison populations use their mental health systems as an alternative form of social control.

One reader, for instance, notes: “Our high rate of penal imprisonment is widely-cited, but your study makes me wonder whether that's a fair picture. How do US rates of total institutionalization compare to the world? I suspect (given how we've dismantled mental health infrastructure), that we're somewhat closer to the rest of the industrialized world.”

I was curious about that too and did a bit of research – and plan on doing a lot more. I discuss it in the conclusion of the study here.

My preliminary findings are interesting and suggest that these suspicions are not entirely off the mark, though not exactly right either.

Among countries in the European Union, the highest rate regarding the number of beds in psychiatric hospitals per 100,000 inhabitants in 2000 was in the Netherlands, which had a rate of 188.5. Other highs were posted in Belgium (161.6), Switzerland (119.9), France (113), and Finland (102.9). The average for the 25 European Union countries in 2000 was 90.1, down from 115.5 in 1993.

These figures are, indeed, higher than the corresponding prison rates for the same countries, which stood in 2006 at 128 per 100,000 persons in the Netherlands, 91 in Belgium, 83 in Switzerland, 85 in France, and 75 in Finland. But they certainly do not come close to the rates of aggregated institutionalization in the United States.

These are preliminary findings, and I obviously need to conduct more research on these comparative figures. There is one country, though, that may offer some competition to the United States – strangely reminiscent of the Cold War era. The Russian Federation has a prison rate of 611 per 100,000, which, when combined with mental health institutionalization, may begin to get close to our institutionalization rates.

On a related issue, there is evidence that in the past some European countries used institutions other than the prison more than they do now to control those deemed deviant—in other words, that the trends identified in the United States may bear some resemblance to trends in Europe.

The Republic of Ireland, for example, had much higher rates of institutionalization in a wide range of facilities, including psychiatric institutions and homes for unmarried mothers, at mid-century—in fact, eight times higher—than at the turn of the twentieth century. Eoin O’Sullivan and Ian O’Donnell have an interesting new paper on that in Punishment & Society. It’s called “Coercive confinement in the Republic of Ireland: The waning of a culture of control,” and it’s in Volume 9(1) at 27-48 (2006).

In Belgium, the number of psychiatric hospital beds per 100,000 inhabitants fell from 275 in 1970 to 162 in 2000; in France, it fell from 242 in 1980 to 111 in 2000; in the UK, from 250 in 1985 to 100 in 1998; and in Switzerland, from 300 in 1970 to 120 in 2000. Again, this requires more research, but there may be a parallel here in terms of the rise and fall of mental health rates.

So overall, important differences, but some parallels. Apart from Russia, though, the numbers should not be much comfort for the United States.

Mental Hospital, Prison, and Homicide Rates: Some More Analyses.

Yesterday’s post triggered a lot of comments regarding this graph – it’s on page 23 of the new study on asylums and prisons that I was discussing previously. The figure graphs two time-series using national-level data: the overall rate of institutionalization in the United States (in mental hospitals and prisons) and the homicide rate over the period 1934 to 2001. The institutionalization trend line is scaled to the left-hand side and is high throughout the 1930s, 40s, and 50s; the homicide trend line is scaled to the right-hand side and rises sharply in the 1970s and 80s.

FIGURE: Rates of Aggregated Institutionalization and Homicide in the United States (per 100,000 adults).

In an earlier paper, I analyzed these data using a Prais-Winsten regression model to correct for autocorrelation in the time-series data. I found a large, robust, and statistically significant relationship between aggregated institutionalization (asylums and prisons) and homicide rates at the national level, holding constant three leading structural covariates of homicide (youth demographics, unemployment, and poverty).

The problem with using time-series data for a single jurisdiction (in this case, the entire United States) is that they typically provide weak power to rule out alternative explanations for the patterns observed in the data. This is something I’ve observed and written about in the context of Giuliani-style policing. (In an article with Jens Ludwig testing the broken-windows policing hypothesis, we showed that the time-series data for crime in New York City was not just compatible with a broken-windows policing theory, but also with what we call the “Broken Yankees Hypothesis” (BYH). It turns out that the strong performance of Billy Martin’s Yankees teams during the late 1970s coincided with a drop in homicides, and the consistent excellence of Joe Torre’s squads beginning in the late 1990s accompanied an even greater decline in homicides).

In order to test the national-level findings, I collected state-level panel data and ran clustered regressions. The results were truly remarkable. Using state-level panel data spanning the entire period from 1934 to 2001, including all 50 states, and controlling for economic, demographic, and criminal justice variables, I again found a large, robust, and statistically significant relationship between aggregated institutionalization and homicide rates. The findings are not sensitive to weighting by population and hold under a number of permutations, including when I aggregate jail populations as well.

To help visualize the relationship, I plotted the predicted values of homicide in the final model (Model 6) against the aggregated institutionalization rate. These, then, are the predicted values of homicide from the model including all the independent variables (aggregated institutionalization, real per capita income, demographics, execution rate, proportion urban, proportion black, and state and year fixed effects). The data are clustered by state, resulting in what appear to be some strings of observations.

Some readers have suggested that the study should include a model with the prison rate and the mental hospitalization rate as separate independent variables. John Lott recently wrote to me “I don't understand why prison population and [mental hospitalization] only seemed to be entered in as a sum." Eric Rasmusen similarly argues here that “There is another regression you absolutely must do: regress murder on [aggregated institutionalization], prison, and asylums all in one regression. That will separate out the effects.”

These are interesting points and something my superb colleagues at the University of Chicago, Tom Miles and Jake Gersen, had batted around with me earlier. My concern is the contribution of aggregated institutionalization and I am really not concerned about the relationship of the parts. I had included some of these regressions in the study, but for the sake of completeness, I just now reran the regressions using every possible permutation of aggregated institutionalization, mental hospitalization alone, and prison rates alone. Every possible permutation — all three, each alone, and every dual-combination.

Here’s a table summarizing my results. I’ll just note for those who are not steeped in stats that the first model, which includes all three independent variables is going to drop one of them. It’s actually impossible to use all three in the same regression. If you include the sum of two variables and each of those two variables, there is a co-linearity problem (since the sum is of course a linear combination of the two). Statistics programs fix this problem by tossing out one of the variables. In this case, STATA dropped the mental hospitalization alone variable. So Model 1 is really identical to Model 6.

But I’ve presented them all for full and complete disclosure. They do not affect my conclusions. Models 2 and 4 are in the draft of the study. Model 5 represents a race-horse comparison of mental hospitalization and prison rates. Notice that mental hospitalization alone is slightly less significant, but still significant, whereas prison rates alone are not. Again, my concern is not with the relative contribution of the parts, but of the whole. Model 6 includes aggregated institutionalization and prison rates – and here too, aggregated institutionalization remains statistically significant with a coefficient about the same size (slightly larger).

New Table: Harcourt Results on State-level Panel Data (All Permutations)

These additional specifications do not change the bottom line: Aggregated institutionalization is the best predictor of homicide rates. In studying the prison today, we need to aggregate mental hospitalization and prison rates.

Not only that, but there is in all likelihood an endogeneity problem that actually attenuates the relationship that I am finding in my data. The fact is, there is, if anything, simultaneity bias. The relationship between crime and institutionalization is likely to be two-way. Although increased institutionalization is likely to decrease crime rates through incapacitation, increased crime is also likely to increase institutionalization through convictions and sentencing.

As a result, the incapacitation effect of institutionalization on crime is probably diminished and the statistical estimates are likely to understate the effect. The effect of the bias would be to underestimate the effect of aggregated institutionalization on crime. This would only increase the effect of aggregated institutionalization on homicide.

A former student of mine who also studied under Gary Becker, John Pfaff at Fordham, has a terrific new paper on the methodological problems in the prison literature. He extensively reviews the existing "first generation" studies and raises a number of methodological problems — from endogeneity to omitted variable biases and colinearity.

To be sure, like those other studies, the statistical analyses in my study may be missing some control variables. Few if any of the studies that John reviews in his paper go as far back as the 1930s and the fact is, it is practically impossible to find any more reliable data at the state level that go back that far — though I am continuing to search for more.

But the findings are nevertheless remarkable — actually astounding. These regressions cover an extremely lengthy time period (back to 1934) for all fifty state, resulting in a large number of observations (almost 3,300), controlling for economic, criminal justice, youth and demographic variables, and the results remain robust and statistically significant in the most complete models. That is amazing.

One final point. At a conference last week at Yale where I first presented this work, some participants argued that I have to guide the use of this research and address the policy implications.

I resisted the invitation then, but want to emphasize why here. The reason is that the policy implications of this study could lead in any number of directions. Some readers could argue that my findings show there is no reason to have prisons. Instead of prisons, we should have treatment facilities. Others could argue that we should incapacitate more women — remember, there were far more women in mental hospitals, almost 50 percent. Some might argue that we are now at the right level of institutionalization. But this study tells us nothing about the costs and trade-offs to society involved in imprisoning so many people, and whether the harm to the individuals affected by incarceration does not outweigh the harms to the victims of crime.

So I want to emphasize that we all need to proceed with caution. A study finding correlations is not enough to start drawing policy conclusions.

Institutionalization vs. Imprisonment: Are There Massive Implications for Existing Research?

Andrew Gelman at Columbia University writes on his statistics blog here that the findings I discussed yesterday seem correct and don’t surprise him in the least. (He also has some entertaining reactions to some of the comments from yesterday).

I’ll confess that I not only find the results surprising, I also think that, if they are indeed right, they have farreaching implications for our existing (and future) research on prisons and their effect on unemployment, crime, education, and poverty, as well as our research on gun laws (think of the right-to-carry debates here, here, and here), the effect of abortions (think of the Donohue/Levitt thesis), the deterrent effect of the death penalty (think of the recent debates here), social control and disorganization theories, collective efficacy – and the list goes on.

In practically all those studies, we have used the imprisonment rate to measure society’s level of incapacitation. But the prison rate alone may not capture what we were trying to measure. The most straightforward interpretation of my findings is that neither the rate of imprisonment alone, nor the rate of mental hospitalization alone are good predictors of serious violent crime over the period 1934-2001. In contrast, the aggregated institutionalization rate (aggregating the mental hospitalization and prison rates) is a strong predictor of homicides. This suggests that there is something going on in the relationship between mental hospitalization and prison — perhaps a form of substitution — that should make us rethink entirely how we measure social control and incapacitation.

But since practically none of our studies on prisons, guns, abortion, education, unemployment, capital punishment, etc., controls for institutionalization writ large, most of what we claim to know about these effects may be on shaky ground.

Here’s a good example. My colleague Steve Levitt at the University of Chicago has a great paper on the crime decline of the 1990s published in the Journal of Economic Perspectives called Understanding Why Crime Fell in the 1990s: Four Factors that Explain the Decline and Six that Do Not. In the paper, Levitt identifies the prison-population build up as one of the four factors that explains the crime drop of the 1990s.

Levitt estimates that the increased prison population over the 1990s accounted for a 12% reduction of homicide and violent crime, and an 8% reduction in property crime — for a total of about one-third of the overall drop in crime in the 1990s (see pages 178-79). The paper and its progeny have given rise to fascinating debates over the role of the police (Malcolm Gladwell takes Levitt to task in an interesting post here), the abortion thesis, and the role of broken-windows policing vs. the crack epidemic.

What interests me here, though, is that when Levitt extends his analysis to discuss the period 1973–1991, he sticks to the prison population exclusively and does not consider the contribution of the declining mental hospital population (see pages 183-86). As a result, Levitt is surprised that the drop in crime did not start sooner (see page 186). Regarding the period 1973–1991, Levitt writes:

"The one factor that dominates all others in terms of predicted impact on crime in this earlier [1973–1991] period is the growth in the prison population. Between 1973 and 1991, the incarceration rate more than tripled, rising from 96 to 313 inmates per 100,000 residents. By my estimates, that should have reduced violent crime and homicide by over 30 percent and property crime by more than 20 percent. Note that this predicted impact of incarceration is much larger than for the latter [1990s] period." (page 184)

Based on prison data alone, Levitt is left with a significant gap between projected and actual crime rates for the period 1973–1991. Levitt concludes: “The real puzzle in my opinion, therefore, is not why crime fell in the 1990s, but why it did not start falling sooner” (see page 186).

The unexplained difference, though, vanishes if we include mental hospitalization with the prison rate in an aggregated institutionalization variable. I do the math in this paper here at page 1775. The increase in confinement from 1973 to 1991 would have been smaller (because of deinstitutionalization) and, based on Levitt’s estimates, this would have translated into a 12% decrease in homicides, not a 35% decrease. Levitt’s revised estimate for the total effect of his ten factors on homicide during the 1973–1991 period would be an increase in homicides of 3%, which is not far from the actual reported change in the UCR of a positive 5%.

In other words, using aggregated institutionalization data rather than prison data would eliminate Levitt’s disparity regarding the change in homicides. This is just one example that explains a gap. But think of all the other areas where the difference might undermine the results.

Here's another example from the death penalty deterrence debates. The fact is that none of the existing extensive research on the deterrent effect of capital punishment has included mental hospitalization within an aggregated institutionalization rate. Instead, all the studies use prison rates only to get at a measure of incapacitation.

My study includes, as a control variable in the regressions, the execution rate for each state over the period 1934 to 2001. So we can get some idea of what happens when you use aggregated institutionalization rather than the prison rate. The results are interesting: in my fourth model (Model 4 of Table III.1 at page 33), the execution rate is positively related to homicide and statistically significant at .05, suggesting that, controlling for aggregated institutionalization, there may be evidence of a brutalization effect from executions: more executions, more homicide. The statistical significance does not withstand the introduction of demographic and urban variables, and in my most complete model (Model 6 same page) the coefficient is positive but unreliable.

Much has been written recently about the deterrent effects of capital punishment. John Donohue and Justin Wolfers have reviewed the recent studies, including state-level panel data analyses, and conclude that “none of these approaches suggested that the death penalty has large effects on the murder date” (page 841). When I include mental hospitalization, my findings are consistent with these conclusions, but in the process they undermine a lot of other research.

Practically all our criminology has failed to connect the prison to the asylum. For instance, Alfred Blumstein and Joel Wallman, in their account of crime trends in the introduction to The Crime Drop in America — generally perceived as an authoritative compilation — never address aggregated institutionalization. With regard to the sharp increase in crime in the 1960s, Blumstein and Wallman hit on all the usual suspects — the baby-boom generation, political legitimacy, economics — and include later the usual explanations for the 1990s crime drop — changing drug use patterns, decreased gun violence, New York-style policing, the federal COPS program, and increased incarceration. Notably absent in all of this, though, is the relationship between mental health and prison populations.

With the exception of research that specifically explores the interdependence of the mental hospital and prison populations, including some public health studies and some empirical research into the causes of the prison explosion (for instance here, here and here) published empirical research does not conceptualize the level of confinement in society through the lens of aggregated institutionalization (asylum + prison) but rather simply through imprisonment rates.

Even the most rigorous, recent analyses of the prison-crime relationship use only imprisonment data. Though a tremendous amount of empirical work has been done on long-term crime trends, structural covariates of homicide, unemployment, and the prison expansion, none of this literature conceptualizes confinement through the larger prism of institutionalization, and none of it aggregates mental hospitalization data with prison rates.

So in contrast to Andrew Gelman, I’m not only surprised by the results of the regression, I’m also extremely concerned about the implications regarding the state of our current knowledge and existing research. And, unhappily, in contrast to Gelman's just-so post, I expect a huge amount of resistance.

Asylums and Prisons: Race, Sex, Age, and Profiling Future Dangerousness.

In Madness and Civilization, Michel Foucault documented a remarkable continuity of confinement through different stages of Western European history, from the lazar houses for lepers on the outskirts of Medieval cities, to the Ships of Fools navigating down rivers of Renaissance Europe, to the establishment in the seventeenth century of the Hôpital Général in Paris — an enormous house of confinement for the poor, the unemployed, the homeless, the vagabond, the criminal, and the insane.

“Leprosy disappeared,” Foucault writes, “the leper vanished, or almost, from memory; these structures remained. Often, in these same places, the formulas of exclusion would be repeated, strangely similar two or three centuries later. Poor vagabonds, criminals, and “deranged minds” would take the part played by the leper . . . . With an altogether new meaning and in a very different culture, the forms would remain—essentially that major form of a rigorous division which is social exclusion but spiritual reintegration.”

Social exclusion unites the asylum and the prison. The question that I ask in my research is whether we should think of the two populations as somehow linked. Is it possible that today’s category of the “criminally deviant” is tied to yesterday’s category of the “mentally defective”? In our social research, should we think of the two populations as a whole, rather than as two separate parts?

I am by no means suggesting that the same people have been moved from one institution (the asylum) to another (the prison). That is far too simplistic – for at least three important reasons.

Mental Illness: First, although the rate of mental illness among prison inmates is probably higher than among the general population and although the problems surrounding mental illness in jails and prisons have reached crisis proportions, it’s not the case that our prisons today are overwhelmingly housing persons with mental illness. For one thing, the war on drugs has taken an enormous toll on African-American communities, and has contributed to an unconscionable increase in black male incarceration that has nothing to do with mental illness. Bruce Western at Princeton documents this better than anyone in his new book, Punishment and Inequality in America.

Estimates of the number of mentally ill inmates vary. According to a 1999 report by the DOJ, about 283,800 inmates in prisons and jails suffered from mental illness at the time – which represented about 16% of jail and state prison inmates. A more recent 2006 DOJ study reported that 56% of inmates in state prisons and 64% of jail inmates across the country reported mental health problems within the past year. Steven Raphael at Berkeley has a fascinating paper and he finds that deinstitutionalization from 1971 to 1996 resulted in between 48,000 and 148,000 additional state prisoners in 1996, which according to him, accounted “for roughly 28 to 86 percent of prison inmates suffering from mental illness.”

A new paper by Steven Erickson and his colleagues reviews the literature on prison mental illness and shows that the estimates for mental illnesses, broadly defined, range from 16% to 90% and for severe mental illness from 6.4% to 39%. “These rates,” they suggest, “are well above those found in the general population of approximately 30% for mental illness and 6% for severe mental illness.” The paper is extremely skeptical of these estimates and casts doubt on the surveys based on methodological shortcomings.

For sure, it is exceptionally difficult to compare mental hospital residents of the 1950s to prison populations of the 1990s because the definitions, diagnoses, and medical routines have changed so much. Remember, the whole infrastructure of our mental health system has collapsed – making it unrealistic to measure the key attribute of “prior mental health contacts.” Moreover, drug use and psychotropic medications have changed enormously. But despite all that, the two populations must differ along somewhat-objective criteria of mental illness.

Race, Sex and Age: Second, the demographics of the two populations are different, as I discuss here in the Texas Law Review at pages 1781-1784. The prison population today is, overall, younger, much more male, and more African-American than the mental hospital populations at mid-century. In 1966, for example, there were 560,548 first-time admissions to mental hospitals, of which 310,810 (55.4%) were male and 249,738 (or 44.6%) were female. In contrast, new admittees to state and federal prison were consistently 95% male throughout the twentieth century. In 1978, African Americans represented 44% of newly admitted inmates in state prisons. That same year, minorities represented 31.7% of newly admitted patients in mental hospitals.

(But note that those populations were also changing internally. Henry Steadman and John Monahan report in a 1984 study that, in their sample, “the mean age at hospital admission decreased from 39.1 in 1968 to 33.3 by 1978. The percentage of whites among admitted patients also decreased, from 81.7% in 1968 to 68.3% in 1978.” There was a similar shift in the prison admissions data: “the mean age of prison admittees was 29.0 in 1968 and 28.1 in 1978" and the percentage of whites among prison admittees decreased from "from 57.6% in 1968 to 52.3% in 1978").

The War on Drugs: Third, a large portion of our current prison population consists of non-violent drug offenders. The war on drugs has helped fill our prison populations, especially our federal prisons. The following graph, from a forthcoming book with Frank Zimring at Berkeley on the Criminal Law and Regulation of Vice, traces the total sentenced population of the federal prison system and the number of federal prisoners for whom the most serious offense was a drug offense:

As I noted before, drug use may intersect in complicated ways with mental health issues, and some users may well be self-medicating. But the numbers associated with the war on drugs clearly transcend these possible connections.

As a result, the story is not simple trans-institutionalization. It is not simply substitution from one institution to another. But that does not mean that the populations are not sufficiently connected or similar in more important ways to be thought of as one – or counted as one. Could it be that we use the categories to socially exclude people we perceive as marginal, disorderly, abnormal? Do we use the categories to sift out those who offend our sensibilities and who we perceive as dangerous? Michel Foucault observed in Madness and Civilization that “There must have formed, silently and doubtless over the course of many years, a social sensibility, common to European culture, . . . that suddenly isolated the category destined to populate the places of confinement. To inhabit the reaches long since abandoned by the lepers, they chose a group that to our eyes is strangely mixed and confused. But what is for us merely an undifferentiated sensibility must have been, for those living in the classical age, a clearly articulated perception.”

Today, the categories of “mental illness” and “criminal deviance” seem very distinct. With the exception of those inmates who are diagnosed as suffering from mental illness, it seems wrong or confused to lump together the insane and the criminal, to mix the two categories. But is it? Will later generations question our own inability to see the continuity of social exclusion and confinement?

One place where the categories seem to be melding together is in the prediction instruments that we use to identify future dangerousness. We are now profiling the criminally dangerous, the mentally instable, and future sexual offenders in very similar ways. I trace the history of our profiling instruments in a new book, Against Prediction: Profiling, Policing, and Punishing in an Actuarial Age.

We’ve seen a rash of new actuarial instruments intended to predict future violent behavior. In terms of sexual violence, these include the Static-99, the Violence Risk Appraisal Guide (VRAG), the Hare Psychopathy Checklist-Revised (PCL-R), the Minnesota Sex Offender Screening Tool (MnSOST-R), the Sex Offender Risk Appraisal Guide (SORAG), the Sexual Violence Risk-20 (SVR-20) and the HCR-20 — as well as, for the very first time, released in 2005, violence risk-assessment software, called the Classification of Violence Risk (COVR).

How accurate are these prediction instruments and how will they affect the profiled populations? John Monahan, a leading authority on prediction instruments, a proponent of these instruments (in fact co-author of the new COVR software), and the director of the MacArthur Violence Risk Assessment project, offers a nuanced assessment. Writing in The Observer, Monahan asks: “How good are psychiatrists and psychologists at distinguishing which people with a mental illness will be violent? Research shows professionals are better than pure chance, but not much. Predicting harmful behaviour is like predicting bad weather. An inaccurate prediction doesn't necessarily mean the clinician or the meteorologist has 'missed something'; it may just mean the science of forecasting has a long way to go.”

Not much better than pure chance. Virginia just adopted a Sexually Violent Predators Act (“SVPA”) in April 2003 that provides for the civil commitment of sex offenders identified based on the Rapid Risk Assessment for Sex Offense Recidivism (RRASOR) – an actuarial instrument. The RRASOR consists of four items (prior sexual offenses, age at release, victim gender, and relationship to victim) and scores as a sum these four items. A score of 4 or more on the RRASOR (the higher scores) is associated with a 5-year sex offense recidivism rate of 37% and a 10-year sex offense recidivism rate of 55%.

Fifty-five percent — and remember, these are persons who have previously been convicted (rightly or wrongly) of a sexually violent offense. That leaves almost half the relevant population misidentified, at least for that 10-year span. Accuracy and inaccuracy may be in the eye of the beholder. (I discuss the reliability of other actuarial instruments in Against Prediction, reviewing studies like these here and here). The question is, how will these new actuarial methods and predictions of future dangerousness shape the people in our total institutions?

UPDATE MAY 4, 2007: John Monahan tells me that Virginia last year changed it's Sexually Violent Predators statute to require not the RRASOR but the Static-99. The Static-99 has slightly different cut-off scores depending on the age of the victim.

Concluding Thoughts on Total Institutions: Future Directions and Critical Reflections.

I conclude this series of web logs where I began — with that haunting graph of the asylum and prison populations in the United States during the twentieth century (rates per 100,000 adults):

Chris Uggen, chair of the sociology department at Minnesota, asks on his public criminology blog: “Shouldn't sociological criminologists be able to offer some explanation for the figure at left, showing the aggregate rate of institutionalization for prisons and mental hospitals?”

The figure does, indeed, call out for explanation. But my sense is that the graph itself and the state panel regressions do more to undermine confidence in our conventional explanations and accepted wisdom, than they do to stabilize them.

It may simply be too early to offer answers. We may need first to rethink and study afresh the notion of total institutions. So let me suggest here a few directions for further research and a question regarding the larger theoretical framework.

First, I think we need to place the demographic differences between the two populations in a richer historical context. Many readers immediately question my findings because of the demographic contrast between the asylum and prison populations. But there may be more to the picture. On the issue of racial compositions, for instance, the national counts may mask important differences at the state and regional level.

The early surveys by the Census Bureau are revealing in this respect. Aggregated to the national level, African-Americans represented a small fraction of residents in mental hospitals enumerated on January 1, 1923 — 7.6% to be exact — and had a relatively low institutionalization rate (192 persons per 100,000 African-Americans). Whites in contrast represented 92.9% of mental hospital residents and had a significantly higher ratio of 259.8 per 100,000 whites. But things look very different within and between states and regions.

The New England and Pacific regions had high rates of black institutionalization, in fact far higher than white institutionalization in those regions, and also far higher than black institutionalization in the South. The Census Bureau in 1932 explained why: “This is undoubtedly due to the lack of adequate hospitals for negroes in the South. In the parts of the country in which negro patients are admitted to State hospitals without discrimination, the rate for negroes generally exceeds that for whites. In Massachusetts, for example, the rate for resident negro patients is 644.4 and for resident white patients, 408.8.” Here are some other state breakdowns (in rates “Per 100,000 of same race”):

Notice how the comparative rates differ as between states and regions. Clearly, racial demographics varied at the state level and will require more nuanced analysis.

Second, we need to explore in greater depth the relative magnitudes of the different possible effects. In the state panel data regressions, there are interesting clues about other potential explanations. I've already discussed here the issue — or non-issue — of capital punishment. But there are other explanations to investigate.

The size of the youth population seems to play an important role in my regressions, which is consistent with what many criminologists have argued (see here and here). (Some economists do not agree, see here and here). What is particularly interesting about my results is that the effect shows up with the 20 to 24 cohort in the most complete models, but not with the 15 to 19 cohort. This suggests that the actual ages chosen may have significant impacts on the results.

The race effects are also remarkable and, in all likelihood, have to do with high victimization rates in the African-American community, as Lawrence Bobo at Stanford suggests here in "A Taste for Punishment." The negative effect of urbanization is surprising, but may be an artifact of a very loose definition of urbanity. The Census Bureau defines “urbanized areas” very broadly to include areas that have a density of 500 persons per square mile. The gradual lowering of the urban threshold may account for these surprising results.

The key question, though, is how these potential explanations compare in magnitude to each other and, more importantly perhaps, as compared to sociological theories of neighborhood effects, social disorganization, social control, subcultural theories, etc. How does aggregated institutionalization compare to Robert Sampson's measures of collective efficacy and social cohesion in the size of its effect on crime or unemployment or education? Is it possible that institutionalization may actually dwarf those other effects, especially when we investigate a lengthy time period such as 1934 to 2001?

This raises a larger theoretical question about kinds of explanations. The fact is, aggregated institutionalization is about social physics. The term "social physics" may sound nineteenth-century — rightly so. It was first used by Auguste Comte to refer to what is now the discipline of sociology, though Comte abandoned the term when the statistician, Adolphe Quetelet, started using it in a more narrow statistical sense to refer to the “homme moyen” (the average man).

In reappropriating the term here, I would define social physics narrowly as social theories that are necessarily true as a result of the physical nature of our mortal existence, in contrast to theories that depend on the intermediation of human consciousness and decision-making. By way of illustration, consider six theories now central to sociological and economic criminology: (1) rational choice theory, (2) the broken-windows theory, (3) legitimacy theory, (4) incapacitation theory, (5) youth demographics, and (6) the abortion hypothesis. The first three operate through the intermediary of human consciousness. In each case, the theory depends on actors believing certain things and conforming their behavior to those beliefs. In contrast, the last three theories involve only social physics — physical restraint in the case of incapacitation or cohort size in the latter two cases.

Depending on the magnitude of the effects, it may turn out that social physics explain far more than theories of rational choice and social influence. (I discuss this idea here in a a new essay). This raises a troubling question: If social physics have far greater explanatory power, then why have we spent so much of the twentieth century developing socio-cultural and political explanations of deviance — theories of deviant subcultures, disorderliness, social disorganization, collective efficacy, anomie, social conflict, to name but a few? If the dominant factor is simply the rate of total institutionalization qua incapacitation or the size of youth cohorts, then why have we spent so much time trying to identify and shape social relations and social processes?

The answer to this puzzling question — should it arise — may lie in our schizophrenic relationship to punishment that is so glaringly reflected in the arresting figure of asylum and prison populations.

Carceral Notebooks, Volume 2: Exploring the Carceral Zone with Nussbaum, Sunstein, Stone, Leitzel, McAdams, and Others.

Before unplugging and returning to my luddite existence, I wanted to share one last bit of information about the space where deviance gets criminalized. The Carceral Notebooks explore precisely that liminal space between morality and crime – focusing on the puzzles surrounding the criminal and legal enforcement of morality.

If you are interested in that space zoned carceral, you may be interested to know that a new volume of The Carceral Notebooks is just out, available to browse on the web here and also in print.

The new volume has several extremely provocative essays about the regulation of morality, especially the regulation of sexuality, as well as terrific response essays from some of our leading social thinkers. Martha Nussbaum has a wonderful essay on Narcissism and Objectification, Cass Sunstein on Equal Sex, Geof Stone on Placing Consent in Cultural and Historical Context, Jim Leitzel on Secret Deviants, and Richard McAdams on Guilt and Crime.

There is also a virtual art exhibit that accompanies the volume with remarkable artwork by Virgil Marti, Mia Ruyter and others. The artwork is arresting, and I invite you to visit the space.

Thank you to Eugene Volokh and his conspirators for inviting me to discuss my work on asylums and prisons. I’ll now return to the library and to that little green paperback Census Bureau volume, Patients in Hospitals for Mental Disease, 1923.