easily find a large number of books on the subject of baloney detection,
pseudoscience, and pathological science and deception. A complete bibliography
would fill several pages. Michael Shermer, Robert Park and Carl Sagan
and others have written books and articles about these subjects. All
have produced lists of things to look for and questions to ask in order
to detect bogus claims or even outright fraud.
is a synthesis of a number of these. All writers will say that no scheme
provides completely foolproof detection of bogus claims; however, if
you poke and prod the claim with the questions that follow, the odds
are in your favor. The questions are in no particular order, so you'll
have to mine out the applicable ones.
is the claim/discovery announced?
A scientific discovery is checked and reviewed before being accepted
for publication in a peer-reviewed journal. If a claim or discovery
is announced by holding a press conference, beware. Such a claim may
be unverified - not checked by an independent investigator. The cold
fusion debacle is often cited. Robert Park's description of this affair
in "Voodoo Science" is recommended reading. Also beware of "discovery"
announcements that ask you to subscribe (pay) to some newsletter.
this source often make such claims?
If this particular source frequently makes claims that bear no resemblance
to the best current knowledge, beware. There was someone in Dallas who,
as the year 2000 approached, made a series of predictions based on the
book of Revelations. He described how the sky would open up, trumpets
would sound, and the world would end. Obviously, nothing happened.
anecdotal evidence cited?
Anecdotes, or individual stories, are not scientifically useful. Such
a story suggests only that the individual did so-and-so (assuming no
fakery). No thing about larger populations can be inferred from it.
Any description of an invention or product which uses large numbers
of anecdotes or testimonials is suspect. You also must remember that
testimonials can be bought; you can pay someone to do it for you. Also
- the Federal Trade Commission will prosecute false claims, but testimonials
are not regulated.
the source claim that "the establishment is trying to suppress this
Such a claim does not provide any evidence that the "discovery" is really
valid. You need to poke these with other questions to find out more.
the claim fit in with what we know about the world?
Does it fit into the larger context of knowledge? Shermer describes
the claim that the Earth is only 6,000 years old. Accepting that claim
requires that you also concede that everything learned about the Earth
and solar system through astronomy, physics, geology, biology and other
sciences is completely wrong.
the "discovery" made in isolation?
The reality of this world is that someone not trained in the science
and its methods and working alone in a garage is not likely to make
any kind of significant scientific discovery. Such a person making a
great "discovery" is likely to be misinterpreting a known effect. They
will likely persist with their belief in their discovery even after
being shown that it is a well-known effect.
anyone tried to disprove the claim?
It is very important to find out if other investigators have tried to
replicate the work. Toward this end, real scientists will publish complete
information about the work, enough for another competent investigator
to attempt replication. Others will repeat the work and publish the
this source offering a new explanation for observed phenomena, or simply
attacking the existing explanation?
As with claims of suppression, this claim does not offer any evidence
that the discovery is valid. Anyone attacking an existing explanation
had better offer evidence. Absent evidence, all you know is that they
do not like the existing explanation, for whatever reason.
the source claim that "this knowledge has been around for so long it
must be good?"
A lot of interesting "discoveries" were made a long time ago. These
include such gems as "the Earth is flat", "fire is a result of the phlogiston
escaping from the burning material", and "the primary components of
everything are Earth, air, fire and water." That they have been around
a long time doesn't make them right. Again, what is the evidence?
Is the observed effect very small and accompanied by lack of success
in increasing it?
Along this line, is the experiment a multi-trial statistical one or
a direct measurement? Park's description of mind-over-matter experiments
is recommended. Although balances capable of measuring microgram (very
tiny) forces exist, mind-over-matter experiments use statistical trials,
possibly because no claimant has ever been able to move a micro-balance
with thoughts. The small effect observed in statistical trials is likely
due to some small systemic bias in the experiment itself. The polywater
affair is an example of a small effect that could not be increased.
Does the evidence for the "discovery" not get any better with time?
See number 10 above. If the observed affect is really a tiny bias in
the experiment, nothing the investigator can do will increase the effect.
Another thing to check is the results of better experiments. If, as
the experimental methods improve, the observed effect gets smaller,
it is likely that a perfect experiment will show no effect at all.
What does the bulk of evidence point to - the new claim or something
Is the source focusing on one small thing and ignoring a huge base of
accumulated evidence that points to something else? Be sure to look
at the claim in the context of current knowledge.
What kind of reasoning has been used?
Beware of "Post hoc, ergo propter hoc" thinking; this implies confusion
about causation. This Latin adage translates as "after this, therefore
because of this." If A happened then B happened, then obviously A caused
B. This is a very bad assumption. A claim that A caused B must be accompanied
by credible evidence showing how A caused B. To put it simply, correlation
does not mean causation.
of such things as questionable techniques (like hypnotic regression),
anecdotal evidence (see 3 above), conspiracy theories (like government
cover-ups), poor quality evidence (low-resolution photos) and simple
Does the new claim offers a new explanation for something, does it account
for as many phenomena as the old one did?
Einstein's relativity explained more phenomena than did Newtonian mechanics.
At velocities much less than the velocity of light, relativity reduces
to Newtonian mechanics.
Is there any indication that the source's personal beliefs and biases
are driving the conclusions?
Has contradictory evidence been ignored or pushed aside? It is necessary
to consider all the evidence, not just the favorable bits. The process
of science tends to sort this out.
Is it possible to test the claim?
Obviously, a statement that "God did it" cannot be tested in any way.
Such a claim is an untestable construct. Explanations that do not make
any testable predictions are of no use. They do not add to knowledge.
Is a chain of evidence (links) offered?
If chain of evidentiary links for a claim is presented, every link must
be solid. A proposed chain of evidence fails completely if even one
link fails. If someone claims that A causes B, B causes C, C causes
D, and D causes E, they had better be prepared to demonstrate EVERY
link. If, for example, all links are proven except that C causes D,
which cannot be in any way demonstrated, then there is no proof that
A results in E.
In extreme cases, such as claims about UFOs, can hoax be confidently
Evidence for "flying saucers" is usually photographic. Modern image
processing technology is so good that it is no longer possible to reliably
detect a hoax photograph by examining it. A claim that a photograph
has "passed all photographic analyses" may mean nothing more than that
the hoax is very well done. If hoax cannot be ruled out, there remains
a small probability that the evidence is fake.
Would a claimed invention violate the laws of thermodynamics?
Be aware of the laws of thermodynamics. The universe operates according
to a set of physical laws that we understand reasonably well. Those
laws govern what can and cannot be done. No amount of cleverness or
piety will allow anyone to violate those laws.
Law of Thermodynamics says, simply, that "you can't win." Energy is
conserved; it is not magically created. Always remember the old phrase
" Non gratuitum prandium", which means "no free lunch." No thermal energy
can be extracted from a single source - a heat flow from hot to cold
Law of Thermodynamics says "you can't even break even." Any energy conversion
process has losses - the output energy will be less than the input.
The difference is wasted and contributes to increasing the entropy of
the universe. No one has ever found a way to get around this.
Is the claim/discovery really spectacular?
Scientists will tell you that "extraordinary claims require extraordinary
evidence." They mean exactly that. If the claim is that life has been
found on Mars or that someone has found a way to eliminate aging, really
extraordinary evidence will be required to demonstrate it. A vague observation
or a few anecdotes will not do the job.
Beware of special pleadings.
Be especially wary of any claim or excuse that the claimed effect cannot
be measured for some reason. It may be that "the presence of a skeptic
contaminates the effect", or "attempting to measure the effect destroys
it", or "bad vibrations interfered", or something like that. An "effect"
that cannot be measured likely does not exist.
If the effect is measured from a sample, how was the sample obtained?
Obtaining a proper random sample for a statistical measurement is more
difficult than most people realize. There are many problems that can
occur with sampling, problems that can seriously bias the results.
suppose you want to take a poll about something and you want a random
sample of the population of your city. How to do it? You might think
of randomly choosing names from the phone directory - but that will
exclude all people who either have no phone or have unlisted numbers.
Try going downtown and randomly picking people who pass by. That will
exclude all people who do not go downtown. Getting a real random sample
is not easy.
Beware of "It can't be, so it isn't" thinking.
In the Cottingley Fairies incident, in 1917, two young girls took photographs
which supposedly showed fairies cavorting with them in the woods. The
two girls who photographed the fairies were considered incapable of
perpetrating such a hoax, and thus followed the conclusion that they
had not. The photos were therefore fawned over as genuine pictures of
fairies. Truth was that the girls had indeed cut the "fairies" out of
paper and photographed them in a prank which got out of hand when others
saw the photos. This was not figured out until 1978.
Look at spelling and grammar.
When reading any web site, flyer, or anything else, be aware of how
well the stuff is written. A web site or printed item that is filled
with misspellings and bad grammar is highly suspect.
of excellent books on the subject of pseudo-science exist. Here are
some good ones. Please remember that there are many more. The book on
hoaxes is included because it is a good collection of historical (and