The Science of Quackometrics
Thursday, June 7, 2007
So, how does the Quackometer work?
The quackometer counts words in web pages that quacks tend to use. The more quack words, the more quackery is suspected. That is Quackometrics.
The basic problem is that spotting the suspect words that many sites use, such as ‘vibrations’ or ‘energy’ is just not good enough as ‘good science’ sites are quite at liberty to use them. Even spotting these words in close conjunction with health terms, such as ‘healing’ or ‘nutrients’, is not quite good enough. My own background was research within in nuclear medicine group and the researchers had lots of legitimate reasons to mention ‘magnets’ and ‘health’ in (almost) the same breath.
So – the site uses an algorithm roughly like this:
The quackometer counts words in web pages that quacks tend to use. The more quack words, the more quackery is suspected. That is Quackometrics.
The basic problem is that spotting the suspect words that many sites use, such as ‘vibrations’ or ‘energy’ is just not good enough as ‘good science’ sites are quite at liberty to use them. Even spotting these words in close conjunction with health terms, such as ‘healing’ or ‘nutrients’, is not quite good enough. My own background was research within in nuclear medicine group and the researchers had lots of legitimate reasons to mention ‘magnets’ and ‘health’ in (almost) the same breath.
So – the site uses an algorithm roughly like this:
- Keep a number of different dictionaries for use in tallying words in a web site
- Load the suspect web page and strip as much out as possible, HTML tags, scripts, punctuation etc.
- Count the number of words in each of the following dictionaries:
a) altmed terms: such as ‘homeopathic’, ‘herbal’, ‘naturopath’
b) pseudoscientific: clearly suspect terms that scientists rarely use such as ‘toxins’, ‘superfoods’.
c) domain specific words from biomed, physics or chemistry such as ‘energy’, ‘vibration’, ‘organic’.
d) skeptical words: words that no sincere homeopath would ever use, such as ‘placebo’, ‘flawed’, ‘crank’ or ‘prosecution’.
e) commerce terms that would indicate that something is for sale, such as ‘products’, ‘shipping’, or ‘p&p’.
f) Run a few other checks on pomo terms and religious terms, although not much is done with these. - Compare the ratio of frequency usage of these various types of terms and compare them to preset thresholds. If a threshold is exceeded then append the test’s associate sentence to the response. The tweaking I have been doing to the site has been adding words to dictionaries and varying the thresholds for matches.
This does not always work, Some quacks are very clever and avoid the obvious quack words. Nonetheless they still have completely hatstand ideas.
So, if anyone else has suggestions, then I would be very greatful. Just need to give up my real job to concentrate on this now.
posted by Le Canard Noir at
2:01 PM
3 Comments:
I'm curious: does this work in the manual way described by the FAQ or is it more like a quack-term-specific bayesian classifier in the style of akismet?
Could I suggest the word 'scholar' would be a good one to add to your list of naughty search words LBD? It seems to be beloved of various quacks , creationists and other assorted religious nutters??
Power to you and Positive Internet by the way!!!
I see a flaw (only because I've just used the quackometer and was concerned by it's result).
The problem occurs in this situation:
A reputable person (in this case Ursula James - Visiting Teaching Fellow at Oxford University Medical School) is listed on an index site, but that index site has a long list of therapies offered by other members including some quackery, all down one side.
Because the quackometer sees the names of all the other therapies it associates them with the subject of the search (Ursula James) and decides that she is a quack.
This is clearly a flaw so I pass it over to you guys to investigate.
Post a Comment
Links to this post:
Create a Link
<< Home