TriggerFinger


Bellesiles v Lott


Introduction

A recent article on the website http://jointogether.org/ proposes to examine the similarities between two researchers on the topic of firearms and the results of allegations concerning their personal behavior and the validity of their work. It makes the claim that certain similarities exist between the case of Michael Bellesiles, an anti-gun historian and author of the controversial Arming America, and the case of John Lott, an economist and author of More Guns, Less Crime and The Bias Against Guns. In both cases, allegations about improper behavior were made, and the JoinTogether organization wishes to claim their subsequent treatment differed -- presumably the result of their differing views on gun control.

Who is John Lott?

John Lott is an economist who has studied the intersection of firearms and crime extensively, including two books on the topic (More Guns, Less Crime, and The Bias Against Guns). He is probably the most-cited authority on the issue. His opinion pieces on the subject are published in major newspapers; he freely shares his data with anyone who cares to ask for it. While a few researchers have challenged some of his conclusions, none have ever credibly accused him of falsifying data, and his responses to his critics are well-reasoned. In short, his academic credentials are impeccable. Anti-gun forces have been searching for some way to attack or discredit his work for years, and failed to find any real basis in his research itself. Unfortunately, they have found other grounds.

The first problem originated some years ago, when Lott claims to have conducted a poll on a gun-related issue, which he then cited in several editorials and interviews. When questioned, he was unable to produce the data in support of his poll; he claimed the data had been lost in a computer crash. He has been unable to produce clear evidence that the poll ever took place, but has posted evidence supporting that claim. Since that time, he has redone the poll at his own expense and produced substantially the same results.

The second issue is somewhat more recent; accusations have arisen that he used a fake online identity to post glowing reviews of his own books to bookstores (primarily amazon.com) and participate in online debates. He has admitted to doing this and responded to the criticisms (at the very end of the linked file, which is a zipped PDF).

Finally, the article quotes Chris McGrath of "Handgun-Free America" saying that Lott's The Bias Against Guns contains unsubstantiated claims, specifically regarding:

  1. claims that one-third of the post-1997 public school shootings had been stopped by armed citizens before police could arrive
  2. claims that concealed-carry permit holders "virtually never" commit crimes

McGrath admits that Lott provided page numbers from his book to support the first claim, but asserts that the page numbers cited don't actually support it. I don't have Lott's book available right now, so cannot verify the claim either way (I will post an update when I can do so).

But I can say that McGrath is wrong about his second accusation; "virtually never" does not mean "never", so a single counterexample is not sufficient to disprove it. Concealed-carry holders as a group commit far fewer crimes than the general population; for example, in Florida, concealed-carry permit holders have a crime rate of .02% (2 hundredths of a percent), or 2 crimes committed per 10,000 permit holders. That's "virtually never" in anyone's book.

And in late-breaking news, another controversy has appeared. This one, if accurate, does in fact cast doubt on Lott's results and integrity. Whether the accusations are accurate is still in question; Lott has posted responses here and here.

Who is Michael Bellesiles?

Michael Bellesiles was a history professor who wrote the controversial book Arming America, claiming that an examination of the historical evidence revealed that early Americans had far fewer guns than is generally thought. His research included studies of probate records (ie, inventories of possessions, typically performed after the owner dies in the course of executing a will), anecdotal accounts from the period, and similar sources. Upon initial publication, his book received rave reviews from the press, and was awarded the prestigious Bancroft Prize.

Shortly after the publication of his work, questions concerning its accuracy in certain areas begin to arise. Those questions originated primarily from independent individuals investing their personal time to check the sources Bellesiles was citing, and concerned the core of Bellesiles' work in several different areas. The most in-depth refutation of Bellesiles' work comes from Clayton Cramer.

The most significant allegations concerned the probate records. Bellesiles claimed to have spent years visiting various archives and libraries around the country, taking painstaking handwritten notes from the probate and court records by means of "tick marks" on a series of notepads. A variety of problems arose from this:

  1. Bellesiles, when asked to provide his data, claimed to have lost the records. After several different excuses, he finally settled on claiming that the records were destroyed during a flood of his offices. While the flood is a verified event, university officials indicated that very little actual damage was done.
  2. Independent individuals seeking to duplicate his results from the same original sources found huge errors, both misreporting firearms as "old or broken" when there was no such indication, and in simply miscounting the numbers listed. While there is room for difference in methods and interpretation from such searches, the errors found were far more substantial than could be explained in that manner, and were universally in favor of Bellesiles' premise.
  3. In at least one case, Bellesiles claimed to have examined probate records that do not exist. After this was pointed out, he made several different attempts to correct his citation by changing the location of the records he had examined. After Bellesiles finally settled on the location of the records he claimed to have examined, publicizing their "rediscovery" to answer his critics, independent investigators contacted the staff of the archive. There was no record of Bellesiles' claimed first visit, and the records at that location did not match Bellesiles' description of them.
  4. In cases where primary sources were cited, investigations uncovered several mis-stated quotations. Taken individually these could easily be honest mistakes; taken together, and the alterations universally supporting Bellesiles' premise, they are vulnerable to accusations of deliberate misrepresentation.

Throughout the length of the controversy, Bellesiles repeatedly denied almost all allegations (to his credit, a subsequent printing of the book did contain some minor corrections). He changed his story about his missing data (the compiled probate records) in particular several times, and focused his defenses of the work primarily on attacking his critics.

After almost two years of non-stop criticism from a number of separate sources, and increasingly poor defenses from the author himself, his employer (Emory University) opened an investigation into the accusations by a board of independent investigations. The report from that investigation indicated that although substantial errors in citations and evidence were found, there was insufficient evidence to prove deliberate misrepresentation in the areas they were asked to investigate.

In other words, the investigative committee stopped a hair short of accusing a prize-winning professor of history and nationally-known author of deliberate fraud. Reading between the lines, it is clear that they did so not because they thought Bellesiles was innocent of the charge, but rather that they felt they could not prove it. As the evidence (Bellesiles' records) had been destroyed, and the question of deliberate fraud versus inadvertent misrepresentation is a matter of intent and notoriously hard to prove in borderline cases, the committee was careful to stick to accusations it could support rather than engaging in speculation on Bellesiles' mental state.

Even without an accusation of deliberate fraud, the report amounted to damning evidence of incompetence. Shortly following the publication of the report, Bellesiles resigned, presumably under pressure from the university. While his letter of resignation, also published, did not admit wrongdoing, it also did not attempt to address any of the accusations in the report.

Telling the difference

So what is the difference between the cases? Well, the difference in outcomes is obvious; Bellesiles resigned his position and has had his work thoroughly discredited. Lott has been attacked by gun control lobbying groups, but has kept his job, continued to publish articles in newspapers concerning the implications of his research, and in general has remained a player in the debate.

But is this difference valid? The Join Together article seems to be suggesting it is not; in other words, that Lott should be subject to penalties similar to those imposed upon Bellesiles. The NRA is mentioned several times throughout the article as leading the attacks upon that historian, giving the (false) impression that it was behind most of the criticism and somehow applied political pressure to force a negative outcome. This is supposed to simultaneously vindicate Bellesiles as a victim of the gun lobby, discredit Lott by continuing to publicize the accusations, and impugn the honesty of those who support gun rights by suggesting their criticism of Bellesiles was politically motivated rather than based on the merits of the claims.

The problem with that premise is simple: it's complete nonsense.

The differences between the cases of Lott and Bellesiles derive from fundamental differences in the accusations, and are then magnified by the response to those allegations. The fact is, the allegations against Bellesiles came from a multitude of sources, independently verified several times over, and concerned the core of his work. Bellesiles' responses to the allegations have consistently been inadequate.

Moreover, rather than a single error or even a few, multiple errors were uncovered, indicating a pattern of at best sloppy record-keeping -- and at worst, deliberate deception. Bellesiles is open to the accusation of deliberate fraud because his errors are both systemic and always seemed to occur in the same direction; that is, misquotes were always misquoted in ways that favored his premise, guns in probate records were always undercounted rather than overcounted, and so on.

By contrast, Lott was accused of losing data for a single survey that did not form part of his core work. He has been consistently willing to provide the data upon which he based the majority of his work to his critics (who, indeed, have used his own dataset in their criticism). He has responded to the criticisms, and taken steps to replicate the missing data from the survey, producing substantially the same results.

In short, while Bellesiles' errors were systemic, suggestive of bias in their direction, and undermined the foundations of his premise, Lott's missing survey represents a tiny fraction of the data he used for his books. It's removal from consideration in no way damages the core premise of his work, and even his critics do not claim otherwise.

In fact, Lott's critics are generally content to debate interpretation of the data, rather than challenging the data itself or the validity of his methods; indeed, many of those critics are dependent upon Lott's own data for their criticisms, even while they refuse to provide their own.

The secondary accusation regarding Lott's creation of an internet persona to post positive reviews about his work, while not particularly flattering, does not in any way impugn the validity of the work itself. Nor does Lott's response to these accusations suggest an ongoing credibility or ethical problem. (In an recent and interesting twist, the Emory Wheel has published a story claiming Bellesiles invented his own internet persona to defend his work; as with the Lott case, this is hardly flattering but not actionable fraud).

That lack of a solid academic basis for complaint about Lott is sufficient to explain the difference in treatment, but there is another factor, perhaps equally important. That factor is the difference in response to criticism.

Lott has continually faced his critics, provided them with his laboriously gathered data set for analysis, and authored substantial replies to their criticism. In cases where his behavior was less than perfect, he's admitted what he did, admitted that it was a mistake, and explained his motives. In short, he has behaved like a credible academic participating in a legitimate public policy debate.

Bellesiles, on the other hand, has not done so. He has refused to participate in forums about the flaws in his work. He makes several differing excuses concerning his inability to provide the data he used. Upon investigation of these claims, they were found to be lacking. He has yet to substantially reply to most of the criticisms leveled against his work. His work has been investigated and found lacking by an impartial committee of historians, and the evidence was sufficient to warrant a resignation.

Conclusion

While the cases appear similar on the surface, the details differ substantially, and from those details the difference in result is derived. Had Bellesiles been revealed only to have used a fake identity to promote his work, he would have been embarrassed, no doubt; but he would not have lost his job. Had Lott failed to provide data supporting his core premise or consistently misrepresented his results, doubtless his own position would have been at risk. But the cases are not identical, and we should not be surprised when the results also differ.

The controversy surrounding Bellesiles has mostly died down. Bellesiles resigned his position at Emory, had his Bancroft Prize rescinded, and the original publisher is no longer publishing his book. With the exception of the article to which this is a response, there has been little mention of him for some time now.

However, the controversy surrounding Lott continues to evolve, with new allegations being made. Lott continues to respond to these allegations. While I can't predict what the final outcome will be, as events are developing even as this paper is written, there is a clear pattern with respect to the Bellesiles case: Lott is freely providing his data and making constructive responses to critics. Bellesiles could never manage a coherent response; just excuses.

No one should expect their statements to be taken on faith. People will check up on claims that are made to influence public policy, and they will be able to determine whether those claims are valid or not. A responsible advocate will have that information ready to have to provide to his critics, even if that provides his critics with more ammunition against him, because the search for truth is more important.

This philosophy is anathema to many advocacy organizations, primarily because they have gotten away with deceiving the public for years, supported in many cases by collaborators in the media who, as with Bellesiles, find the statements of those they agree with too comfortable a fit with their own worldview to aggressively search for the facts. But public policy is too important for comfortable assumptions.

I have listed some of the resources I used in writing this article in my weblog.

Note: This is an older article I am reposting to make it available via the weblog.

This entry was published Tue Jul 04 23:25:05 CDT 2006 by TriggerFinger and last updated 2006-07-04 23:25:05.0. [Tweet]

comments powered by Disqus

Subscribe to Atom Feed

I am not a lawyer, and nothing on this site should be taken as legal advice.

This site is run on custom blog software and is being actively developed. Please be forgiving of errors.


This website is an Amazon affiliate and will receive financial compensation for products purchased from Amazon through links on this site.