I had always assumed that scholarship reviews for tenure promotions were anonymous. This is certainly the case at many (most?) schools, but I've learned it is not the universal practice. At my school, external reviews are anonymous and heavily redacted. Internally generated scholarship reviews can be anonymous as well. While I would love to know the identities of my reviewers -- especially those whom I've never met and yet had very positive things to say about my work -- I have no particular quarrel with the system. At other schools, however, external reviews of a tenure candidate's scholarship are not redacted. At still others, I've gathered that the reviews are officially anonymous, but that tenure candidates regularly learn their reviewers' identity.
The case for anonymity is straightforward: Giving reviewers a promise of anonymity encourages greater candor. Insofar as a tenure candidate's scholarship is deficient, this is more likely to be uncovered through anonymous reviews, as reviewers will feel more comfortable criticizing a fellow academic's work if they can do so from behind the veil of anonymity. Among other things, anonymity may make it easier to criticize a peer whom one knows personally. It may also lesson the risk of reprisal (though this concern is less serious in the legal academy where most journals are faculty edited and relatively few scholars recieve peer-reviewed grants).
Not all academics think anonymity is the best approach, however. Among other things, anonymity may enable some academics to settle personal scores and attack those with whom they disagree. (Yes, this would be wrong for any academic to do, but that does not mean it does not happen.) Others believe that academics should be able to offer honest critiques of others' academic work without concealing their identity.
When I was up for promotion to associate professor (without tenure), one reviewer mailed me a copy of his letter with a note. We had never met, and he was critical of one of my articles (though he still recommneded my promotion. In his note he said he did not believe in anonymous review. In his opinion, anything critical of my scholarship that he would be willing to say to my senior colleagues, he should be willing to say to me. As a tenured professor, he explained, he had little to fear from a junior academic, though he could threaten my career prospects with a negative review. He also suggested that I had a right to know who was criticizing my work. In a sense, he suggested it was cowardly for an external reviewer to hide his or her identity when criticizing someone else. [Insert ironic reference to my own current anonymity here.]
Was my external reviewer correct? Should external reviews be anonymous? I would be interested to learn what VC readers think, and to know more about what is done at other schools and in other disciplines.