Introduction
A recent article on the website http://jointogether.org/
proposes to examine the similarities between two researchers on the
topic of firearms and the results of allegations concerning their
personal behavior and the validity of their work. It makes the claim
that certain similarities exist between the case of Michael Bellesiles,
an anti-gun historian and author of the controversial Arming America, and the case of John Lott, an economist and author of More Guns, Less Crime and The Bias Against Guns.
In both cases, allegations about improper behavior were made, and the
JoinTogether organization wishes to claim their subsequent treatment
differed -- presumably the result of their differing views on gun
control.
Who is John Lott?
John Lott is an economist who has studied the intersection of firearms and crime extensively, including two books on the topic (More Guns, Less Crime, and The Bias Against Guns).
He is probably the most-cited authority on the issue. His opinion
pieces on the subject are published in major newspapers; he freely
shares his data with anyone who cares to ask for it. While a few
researchers have challenged some of his conclusions, none have ever
credibly accused him of falsifying data, and his responses to his
critics are well-reasoned. In short, his academic credentials are
impeccable. Anti-gun forces have been searching for some way to attack
or discredit his work for years, and failed to find any real basis in
his research itself. Unfortunately, they have found other grounds.
The first problem originated some years ago, when Lott claims to
have conducted a poll on a gun-related issue, which he then cited in
several editorials and interviews. When questioned, he was unable to produce the data
in support of his poll; he claimed the data had been lost in a computer
crash. He has been unable to produce clear evidence that the poll ever
took place, but has posted evidence supporting that claim. Since that time, he has redone the poll at his own expense and produced substantially the same results.
The second issue is somewhat more recent; accusations
have arisen that he used a fake online identity to post glowing reviews
of his own books to bookstores (primarily amazon.com) and participate
in online debates. He has admitted to doing this and responded to the criticisms (at the very end of the linked file, which is a zipped PDF).
Finally, the article quotes Chris McGrath of "Handgun-Free America" saying that Lott's The Bias Against Guns contains unsubstantiated claims, specifically regarding:
- claims that one-third of the post-1997 public school shootings had been stopped by armed citizens before police could arrive
- claims that concealed-carry permit holders "virtually never" commit crimes
McGrath admits that Lott provided page numbers from his book to support
the first claim, but asserts that the page numbers cited don't actually
support it. I don't have Lott's book available right now, so cannot
verify the claim either way (I will post an update when I can do so).
But I can say that McGrath is wrong about his second accusation;
"virtually never" does not mean "never", so a single counterexample is
not sufficient to disprove it. Concealed-carry holders as a group
commit far fewer crimes than the general population; for example, in
Florida, concealed-carry permit holders have a crime rate of .02% (2
hundredths of a percent), or 2 crimes committed per 10,000 permit
holders. That's "virtually never" in anyone's book.
And in late-breaking news, another controversy has appeared. This one, if accurate, does in fact cast doubt on Lott's results and integrity. Whether the accusations are accurate is still in question; Lott has posted responses here and here.
Who is Michael Bellesiles?
Michael Bellesiles was a history professor who wrote the controversial book Arming America,
claiming that an examination of the historical evidence revealed that
early Americans had far fewer guns than is generally thought. His
research included studies of probate records (ie, inventories of
possessions, typically performed after the owner dies in the course of
executing a will), anecdotal accounts from the period, and similar
sources. Upon initial publication, his book received rave reviews from
the press, and was awarded the prestigious Bancroft Prize.
Shortly after the publication of his work, questions concerning its
accuracy in certain areas begin to arise. Those questions originated
primarily from independent individuals investing their personal time to
check the sources Bellesiles was citing, and concerned the core of
Bellesiles' work in several different areas. The most in-depth
refutation of Bellesiles' work comes from Clayton Cramer.
The most significant allegations concerned the probate records.
Bellesiles claimed to have spent years visiting various archives and
libraries around the country, taking painstaking handwritten notes from
the probate and court records by means of "tick marks" on a series of
notepads. A variety of problems arose from this:
- Bellesiles, when asked to provide his data, claimed to have lost
the records. After several different excuses, he finally settled on
claiming that the records were destroyed during a flood of his offices.
While the flood is a verified event, university officials indicated
that very little actual damage was done.
- Independent individuals seeking to duplicate his results from the same original sources found huge errors,
both misreporting firearms as "old or broken" when there was no such
indication, and in simply miscounting the numbers listed. While there
is room for difference in methods and interpretation from such
searches, the errors found were far more substantial than could be
explained in that manner, and were universally in favor of Bellesiles' premise.
- In at least one case, Bellesiles claimed to have examined
probate records that do not exist. After this was pointed out, he made
several different attempts to correct his citation by changing the
location of the records he had examined. After Bellesiles finally
settled on the location of the records he claimed to have examined,
publicizing their "rediscovery" to answer his critics, independent
investigators contacted the staff of the archive. There was no record of Bellesiles' claimed first visit, and the records at that location did not match Bellesiles' description of them.
- In cases where primary sources were cited, investigations
uncovered several mis-stated quotations. Taken individually these could
easily be honest mistakes; taken together, and the alterations
universally supporting Bellesiles' premise, they are vulnerable to
accusations of deliberate misrepresentation.
Throughout the length of the controversy, Bellesiles repeatedly
denied almost all allegations (to his credit, a subsequent printing of
the book did contain some minor corrections). He changed his story
about his missing data (the compiled probate records) in particular
several times, and focused his defenses of the work primarily on
attacking his critics.
After almost two years of non-stop criticism from a number of separate
sources, and increasingly poor defenses from the author himself, his
employer (Emory University) opened an investigation into the
accusations by a board of independent investigations. The report from that investigation indicated that although substantial errors in citations and evidence were found, there was insufficient evidence to prove deliberate misrepresentation in the areas they were asked to investigate.
In other words, the investigative committee stopped a hair short of
accusing a prize-winning professor of history and nationally-known
author of deliberate fraud. Reading between the lines, it is clear that
they did so not because they thought Bellesiles was innocent of the
charge, but rather that they felt they could not prove it. As the
evidence (Bellesiles' records) had been destroyed, and the question of
deliberate fraud versus inadvertent misrepresentation is a matter of
intent and notoriously hard to prove in borderline cases, the committee
was careful to stick to accusations it could support rather than
engaging in speculation on Bellesiles' mental state.
Even without an accusation of deliberate fraud, the report amounted
to damning evidence of incompetence. Shortly following the publication
of the report, Bellesiles resigned, presumably under pressure from the
university. While his letter of resignation, also published, did not
admit wrongdoing, it also did not attempt to address any of the
accusations in the report.
Telling the difference
So what is the difference between the cases? Well, the difference in
outcomes is obvious; Bellesiles resigned his position and has had his
work thoroughly discredited. Lott has been attacked by gun control
lobbying groups, but has kept his job, continued to publish articles in
newspapers concerning the implications of his research, and in general
has remained a player in the debate.
But is this difference valid? The Join Together article seems to be
suggesting it is not; in other words, that Lott should be subject to
penalties similar to those imposed upon Bellesiles. The NRA is
mentioned several times throughout the article as leading the attacks
upon that historian, giving the (false) impression that it was behind
most of the criticism and somehow applied political pressure to force a
negative outcome. This is supposed to simultaneously vindicate
Bellesiles as a victim of the gun lobby, discredit Lott by continuing
to publicize the accusations, and impugn the honesty of those who
support gun rights by suggesting their criticism of Bellesiles was
politically motivated rather than based on the merits of the claims.
The problem with that premise is simple: it's complete nonsense.
The differences between the cases of Lott and Bellesiles derive from
fundamental differences in the accusations, and are then magnified by
the response to those allegations. The fact is, the allegations against
Bellesiles came from a multitude of sources, independently verified
several times over, and concerned the core of his work. Bellesiles'
responses to the allegations have consistently been inadequate.
Moreover, rather than a single error or even a few, multiple errors
were uncovered, indicating a pattern of at best sloppy record-keeping
-- and at worst, deliberate deception. Bellesiles is open to the
accusation of deliberate fraud because his errors are both systemic and always seemed to occur in the same direction;
that is, misquotes were always misquoted in ways that favored his
premise, guns in probate records were always undercounted rather than
overcounted, and so on.
By contrast, Lott was accused of losing data for a single survey that did not form part of his core work. He has been consistently willing to provide the data
upon which he based the majority of his work to his critics (who,
indeed, have used his own dataset in their criticism). He has responded to the criticisms, and taken steps to replicate the missing data from the survey, producing substantially the same results.
In short, while Bellesiles' errors were systemic, suggestive of bias in
their direction, and undermined the foundations of his premise, Lott's
missing survey represents a tiny fraction of the data he used for his
books. It's removal from consideration in no way damages the core
premise of his work, and even his critics do not claim otherwise.
In fact, Lott's critics are generally content to debate interpretation
of the data, rather than challenging the data itself or the validity of
his methods; indeed, many of those critics are dependent upon Lott's
own data for their criticisms, even while they refuse to provide their own.
The secondary accusation regarding Lott's creation of an internet
persona to post positive reviews about his work, while not particularly
flattering, does not in any way impugn the validity of the work itself.
Nor does Lott's response to these accusations suggest an ongoing
credibility or ethical problem. (In an recent and interesting twist,
the Emory Wheel has published a story claiming Bellesiles invented his own internet persona to defend his work; as with the Lott case, this is hardly flattering but not actionable fraud).
That lack of a solid academic basis for complaint about Lott is
sufficient to explain the difference in treatment, but there is another
factor, perhaps equally important. That factor is the difference in
response to criticism.
Lott has continually faced his critics, provided them with his laboriously gathered data set for analysis, and authored substantial replies to their criticism. In cases where his behavior was less than perfect, he's admitted what he did, admitted that it was a mistake,
and explained his motives. In short, he has behaved like a credible
academic participating in a legitimate public policy debate.
Bellesiles, on the other hand, has not done so. He has refused to
participate in forums about the flaws in his work. He makes several
differing excuses concerning his inability to provide the data he used.
Upon investigation of these claims, they were found to be lacking. He
has yet to substantially reply to most of the criticisms leveled
against his work. His work has been investigated and found lacking by an impartial committee of historians, and the evidence was sufficient to warrant a resignation.
Conclusion
While the cases appear similar on the surface, the details differ
substantially, and from those details the difference in result is
derived. Had Bellesiles been revealed only to have used a fake identity
to promote his work, he would have been embarrassed, no doubt; but he
would not have lost his job. Had Lott failed to provide data supporting
his core premise or consistently misrepresented his results, doubtless
his own position would have been at risk. But the cases are not
identical, and we should not be surprised when the results also differ.
The controversy surrounding Bellesiles has mostly died down. Bellesiles
resigned his position at Emory, had his Bancroft Prize rescinded, and
the original publisher is no longer publishing his book. With the
exception of the article to which this is a response, there has been
little mention of him for some time now.
However, the controversy surrounding Lott continues to evolve, with
new allegations being made. Lott continues to respond to these
allegations. While I can't predict what the final outcome will be, as
events are developing even as this paper is written, there is a clear
pattern with respect to the Bellesiles case: Lott is freely providing his data and making constructive responses to critics. Bellesiles could never manage a coherent response; just excuses.
No one should expect their statements to be taken on faith. People will
check up on claims that are made to influence public policy, and they
will be able to determine whether those claims are valid or not. A
responsible advocate will have that information ready to have to
provide to his critics, even if that provides his critics with more
ammunition against him, because the search for truth is more important.
This philosophy is anathema to many advocacy organizations, primarily
because they have gotten away with deceiving the public for years,
supported in many cases by collaborators in the media who, as with
Bellesiles, find the statements of those they agree with too
comfortable a fit with their own worldview to aggressively search for
the facts. But public policy is too important for comfortable
assumptions.
I have
listed some of the resources I used in writing this article in my
weblog.
Note: This is an older article I am reposting to make it available via the weblog.