How should we read a scientific paper?
A new study, published last week1, would suggest that it is a very bad idea to simply read the abstract and conclusions. Worse, it can be very misleading to rely on reports derived from press releases about the research. The study looked at how press releases included ‘spin’ about the results of a medical paper. The results were rather alarming.
The authors defined spin as the overestimation and overinterpretation of treatment benefits from RCTs. Unsurprisingly, they found press reports of medical trials contained significant spin. What was disturbing was that this spin was most closely correlated with spin found in the original papers’ abstracts.
We might expect newspapers to make medical research stories more sensational than they really are. Miracle cures and breakthroughs make better stories than mundane marginal improvements or negative results that are more likely in research. But what this story suggests is that we cannot reliably apply the tactic of looking for the original research as a corrective to media spin as the original papers are also likely to include the spin.
As most research is behind a paywall, most people only have access to abstracts of papers. A patient researching a treatment on the basis of a media report could well indeed remain misled if they can only access the abstracts of papers. Even, if we can access the full paper, it is a significant task to have to carefully read the full paper, appraise the presented data and come to our own conclusions regardless of what the abstract says.
In fact, this situation is all rather terrifying.
A good example of this came recently with a new systematic review2 of studies looking at acupuncture for chronic pain. It concluded that “acupuncture is effective for the treatment of chronic pain and is therefore a reasonable referral option”.
The conclusion went on to say,
Acupuncture is effective for the treatment of chronic pain and is therefore a reasonable referral option. Significant differences between true and sham acupuncture indicate that acupuncture is more than a placebo. However, these differences are relatively modest, suggesting that factors in addition to the specific effects of needling are important contributors to the therapeutic effects of acupuncture.
And, as you might expect, acupuncturists and other supporters of superstitious and pseudoscientific treatments are proclaiming that they have been vindicated.
The study looked at 29 randomized clinical trials of acupuncture for various sorts of pain, including back, neck and headache. It looked at studies that used sham and no-acupuncture controls. Sham acupuncture is where the patient thinks they are getting acupuncture, but typically the needle is not being inserted, merely pressed against the skin. Vickers et al. reported that when acupuncture was compared with no- acupuncture, reported reduction in pain was fairly large, about 30%. However, crucially for supporters of acupuncture, they reported a moddest improvement of pain when ‘real’ acupuncture was compared with sham acupuncture.
And as such, the media reports of this finding (with a few honourable exceptions) have followed the abstracts conclusion that acupuncture is a valid treatment option for chronic pain. David Gorski at Science-based Medicine lists a number of these spun headlines.
Is it is as simple as that? Should we accept that indeed acupuncture can indeed by a useful treatment for pain. As you might expect, it is a lot more complicated than headlines would suggest.
First of all, if true, then this analysis shows something startling. Acupuncture claims to work through the needles unblocking something called qi that is supposed to flow through something called meridians. There are tens of thousands of specific points on the body that are supposed to respond to a fine needle being placed in them and wiggled around. This is straightforward pseudoscientific nonsense. Qi and meridians are vitalisitic notions without any evidence to support them. Modern acupuncture is largely a New Age invention that draws on much simpler Chinese ritualistic surgical techniques.
Indeed, one can think of acupuncture as a placebo-ised pre-scientific surgery. Just as homeopathy is a placebo-ised pre-scientific pharmacology, acupuncture is a stylised and minimalised form of blood letting, cauterisation, lancing and slicing. Indeed, the early texts that modern supporters of acupuncture claim to show it is thousands of years old describe a variety of blades and points that look indistinguishable from many societies primitive surgical kit bag.
So, for acupuncture to actually demonstrate specific effects would suggest that there is indeed some truth in what acupuncturists claim about how the body works and causes of illness.
There is good reason to believe that despite the study showing positive effects for real acupuncture over sham acupuncture that these are not due to genuine physiological effects. Blinding in trials of acupuncture is notoriously hard. Most people tend to know if a needle has been stuck in you. Sham stage-dagger-style acupuncture may be the most convincing placebo version of acupuncture we have. But the practitioners are not blinded as is usual in blinded experiments (hence ‘double-blind’). Any small leak of the blinding, either by the patient guessing whether the treatment is real or sham, or by clues given off by the acupuncturist, will lead to biases towards the real acupuncture appearing in the results. A small positive difference in the real acupuncture arm can be explained by small leaks in the blinding. The authors acknowledge this problem in their paper, but crucially, not in their abstract.
Similarly, while we considered the risk of bias of unblinding low in most studies comparing acupuncture and sham acupuncture, health care providers obviously were aware of the treatment provided, and, as such, a certain degree of bias of our effect estimate for specific effects cannot be entirely ruled out.
So, let’s step back. The acupuncture paper is indeed written by supporters of acupuncture. As Steven Novella writes,
I took a close look at the study and find that the authors display considerable pro-acupuncture bias in their analysis and discussion. They clearly want acupuncture to work. That aside, the data are simply not compelling, and the authors, in my opinion, grossly overcall the results, which are compatible with the conclusion that there are no specific effects to acupuncture beyond placebo.
Indeed, the journal the paper was written took the steps of publising an independent commentary3 on the paper, so surprising were the results. Avins notes correctly that the specific effects of acupuncture are of ‘dubious clinical relevance’,
the authors state that the overall observed estimate of approximately 0.5 SD “is of clear clinical relevance,” but this assertion is difficult to substantiate.
Avins then attempts to move the debate on by arguing that despite the effects of acupuncture being placebo, that it may yet have relevance in clinical practice.
The conclusion, that most of acupuncture’s observed clinical benefit is mediated by placebo effects, will probably lead the factions to take their usual positions, with acupuncture advocates focusing on the usual-care comparisons and its detractors stressing the placebo nature of its effects. These latter critics correctly note that a new pharmaceutical agent that fails to show superiority over placebo will not be approved. Why should the bar be lowered for acupuncture? The authors acknowledge this discomfort, stating “ . . . many clinicians would feel uncomfortable in providing or referring patients to acupuncture if it were merely a potent placebo.” In response, they cite the small observed specific effect of acupuncture. But this line of argument is not persuasive: medications are developed and approved for their specific effects, and the small specific effects of acupuncture are of dubious clinical relevance. If we hold fast to the time-honored approach of placebo comparisons, then acupuncture’s detractors rightfully rule the day. But perhaps this debate should prompt us to take a careful look at the assumption that only specific biological effects matter and that the only valid estimate of effect is the comparison with placebo rather than with usual medical care.
This is a more nuanced position than was popularly reported in the press. The authors wanted the coverage to lead with the observed specific effects and that is the coverage they largely got. Acupuncturists are not happy with their techniques just being a placebo. They want more than that.
Nonetheless, despite this paper clearly showing the placebo nature of acupuncture is there still a role for it?
Much confusion surrounds discussions of placebo effects. Such effects may indeed lead to genuine subjective improvements in the experience of pain and so lead to a better quality of life – something to be welcomed. But placebo effects can also be little more than,measurement artifacts. For example, patients may be inclined to score their pain less in order to reciprocate good will towards their acupuncturist. The expectations of improvement and natural desire to please may introduce biases in score results towards positive effects. This does not mean that the patient will continue to receive benefits over any meaningful period of time, if they received any at all. Care is required in understanding just what has occured when a placebo effect is reported.
And then there is the difficult problem of offering placebo treatments in general. Either the practitioner has to intentionally deceive the patient and so undermine some pretty fundamental issues of trust, autonomy and informed consent in medicine. Or the acupuncturist has to genuinely believe in the specific effects of acupuncture and so, in a real sense, be systematically incompetent in their practice.
The problem of True Believers offering a placebo treatment such as acupuncture has other risks. Acupuncturists believe that their needles can indeed treat all sorts of conditions from infertility, smoking cessation, tinnitus to depression. Pain is a placebo responsive condition. Leading patients to believe that acupuncture has specific effects for pain, may reduced their ability to make informed decisions about other health care issues.
And then there are the risks of acupuncture. Any improvements gained from acupuncture, placebo or not, need to be weighed against the risks. A recent paper4 shows that NHS acupuncture indeed carries significant risks from pneumothorax to needles being left in the body. Although rare, the authors worry that adverse events from acupuncture are under-reported. There is an intrinsic belief in acupuncture (like a lot of alternative medicine) that the treatments are inherently safe and natural. The need for systematic reporting of problems is not seen as a priority.
Even with this review of acupuncture that failed to show convincing specific effects for chronic pain, we are likely to see a large increase of, not just patients, but also healthcare providers that now view acupuncture as a genuine referral option. The reasons for this are not because of convincing and thorough data, but because of specific spin placed on the study by its authors, dutifully copied by journalists around the world.
And whose fault is this? The Yavchitz paper on misprepresentaion of RCTs is clear: journal reviewers and editors,
Overall, these findings highlight the important role that journal reviewers and editors play in disseminating research findings. These individuals, the researchers conclude, have a responsibility to ensure that the conclusions reported in the abstracts of peer-reviewed articles are appropriate and do not over-interpret the results of clinical research.
1 Yavchitz A, Boutron I, Bafeta A, Marroun I, Charles P, et al. (2012) Misrepresentation of Randomized Controlled Trials in Press Releases and News Coverage: A Cohort Study. PLoS Med 9(9): e1001308. doi:10.1371/journal.pmed.1001308
2 Vickers AJ, Cronin AM, Maschino AC, et al. Acupuncture for Chronic Pain: Individual Patient Data Meta-analysis. Arch Intern Med. Published online September 10, 2012. doi:10.1001/archinternmed.2012.3654.
3 Avins AL. Needling the Status Quo: Comment on “Acupuncture for Chronic Pain”. Arch Intern Med. Published online September 10, 2012. doi:10.1001/archinternmed.2012.4198.
4 Risks of acupuncture range from stray needles to pneumothorax, finds study
BMJ 2012; 345 doi: 10.1136/bmj.e6060 (Published 7 September 2012)
Follow Up
6th March 2013
Incredibly, the authors of this paper have written another paper moaning about the response of bloggers and skeptics.
The abstract reads,
In September 2012 the Acupuncture Trialists’ Collaboration published the results of an individual patient data meta-analysis of almost 18 000 patients in high quality randomised trials. The results favoured acupuncture. Although there was little argument about the findings in the scientific press, a controversy played out in blog posts and the lay press. This controversy was characterised by ad hominem remarks, anonymous criticism, phony expertise and the use of opinion to contradict data, predominantly by self-proclaimed sceptics. There was a near complete absence of substantive scientific critique. The lack of any reasoned debate about the main findings of the Acupuncture Trialists’ Collaboration paper underlines the fact that mainstream science has moved on from the intellectual sterility and ad hominem attacks that characterise the sceptics’ movement.
Indeed, a paper that actually makes the sceptics’ case for them is spun by its authors as if the opposite was true.
However, I do leave a little door open with acupuncture because I have not been able adequately to deal with the following issue. Chinese ‘traditional’ acupuncture is based on points determined by the fictional meridian system. This is clearly bollocks and any resemblance to real biology is just lucky coincidence. The reported effects are clearly just due to biases and spontaneous processes, plus an element of placebo effect the real value of which I suspect is very small. The trials have been done. It’s Game Over.
On the other hand there is something called ‘dry needling’, which, as far as I can tell, seems to consist of sticking a needle straight into the hurty thing. Mrs Monkey has had ‘frozen shoulders’ and now probably has the same in her hips. She gets a lot of pain that seems to come from muscle spasm around the hips. A colleague of hers did this dry-needling business on her and she instantly went from not being able to sit cross-legged to being able to do so.
Anecdote, blah, blah. Yes. But, my point is that this looks like a direct physical action that is different from acupuncture as we usually know it. Big proviso, the effect at the time was instantaneous and impressive. The chronic pain remains. I have no idea whether repeated procedures strung together could relieve the chronic pain. Even if it could, it’s all terribly laborious and inconvenient and I doubt there really would be any genuine cumulative benefit. But, in my ignorance, I have no idea whether the literature contains any properly controlled trials of ‘dry-needling’ with no-treatment and sham-needling controls.
I therefore do keep open that little door specifically for dry-needling acupuncture for pain, but I suspect I keep I open more for lack of evidence to allow me to shut it rather than any realistic expectation that it works.
Could you be talking about IMS? This is a technique used by physiotherapists, which causes muscles to contract and release, and is completely different from acupuncture. I don’t know about studies, but you might want to look under “Intramuscular Stimulation”.
Typically a few sessions are required to have a significant lasting effect.
I don’t know whether I am. There seems to be range of therapies that all involve poking things into the body and they have a lot of overlapping features. If you read the Wikipedia page on “dry needling” I get the sense of therapies that are relatively recently invented. A worrying feature with these sorts of therapies is when the seem to be pulled fully-formed out of thin air by an individual therapist financially tied to its successful application and promulgation. That, for me, is a big red danger sign for quackery of which Hahnemann is the type-specimen.
Does “IMS” have properly controlled trials for any claims of long-term benefit? It sounds difficult to blind.
Spot on.
It is all very reminiscent of the BJGP fiasco as recorded at http://www.dcscience.net/?p=4439
In that case too, a shocking degree of spin was used to disguise results that showed essentially no effect.
The journal editors who allow this sort of dishonesty have a lot to answer for.
The lack of reporting of adverse outcomes worries me quite a bit. If this were a NDA submitted to the US-FDA, efficacy and safety would both have to be established. Fail on both counts.
The US-FDA looks at medical devices. If acupuncture were new, would the needles even qualify as a new device?
I’m sorry to hear about Mrs Monkey.
Instead of dry needling for her muscle spasms, why not introduce some botox. That should do it!
And when it comes to blind needling, the needlers hould have been trained by actors, gamblers or magicians to keep a straight face and give nothing away. Only when they have a “Certificate of Bland Insouisannce” should they needle.
Someone needs to write an article for the BMJ on the “Bias of Bias” – exposing this issue.
Rewritten to suit, Andy’s piece would be outstanding, and should be available to a wider audience – particularly as we enter more public debate about “the chilling effect of libel” and scientific integrity. (David Nutt has just been on R4’s A Life Scientific. 09:00 to 09:30 on 18/9/12.)
Been avoiding these places for a while, but I can’t quite beleive it!! Badly shaved Monkey talking about anecdotes!!!!!! Are my eyes deceiving me? Can this be true that you “leave a door open” to something that has no hard RCT evidence but because you, or your wife, has seen “my point is that this looks like a direct physical action” and ” Big proviso, the effect at the time was instantaneous and impressive.” you are willing to maybe accept a perceived improvement without solid evidence????? Andy I hope you will strike BSM from the skeptic roll of honour this very second.
Sorry, Liam, you have misunderstood the import of what I say.
If you stick a needle in someone and obtain a direct physical effect, it is a not unreasonable inference to suggest the two are causally connected. Unlike, say, an effect reported from swallowing a homeopathic pill, the inference has fair prior plausibility. The issue that is open to question is whether this effect has any impact on the long-term state of the chronic condition. The RCT evidence for so-called traditional acupuncture is that you cannot string together anecdotes like mine under controlled conditions and any real specific effect. I am unaware of good trials for this dry needling business. I am unaware because I have never bothered to look. Of they exist then I’ll modify my stance. There are two possible outcomes- you hurt someone with a needle and beyond the instant effect, long-term relief is obtained, or you hurt someone with a needle and no long-term benefit accrues.
Perhaps you have been confused by the dismissal of anecdotes when controlled trials exist, when anecdotes are used instead of trial data as if they were sufficient in themselves.. Anecdotes offer facts. What they cannot properly permit are causal inferences. They may suggest causal connections worth testing. But if the trials turn out negative then any tentative causal inferences that were being drawn must be rejected in favour of other interpretations.
Liam, the plausibility of a proposition is not enhanced by the use of multiple punctuation marks. However it is fine to be skeptical of skepticism, as long as such skepticism is based on reasonable grounds.
P.S. In common with many sceptics, I have been reluctant to draw confusions about acupuncture because at least some of its claims have fair prior plausibility, but with the accumulating evidence I think the scope for its validity has got smaller and smaller. Dry needling for painful musculoskeletal problems is about the last remnant I might concede as having efficacy. So it’s not “ah ha! BSM believes in acupuncture!”, it’s that BSM has successively rejected acupuncture, but offers an anecdote that sits in one corner where he has not come to a firm conclusion due to the combination of fair prior plausibility and lack of knowledge of the literature.
P.P.S. before drawing formal causal inferences from the Mrs M story we’d want a whole load of questions answered. Could she still sit cross-legged 1hr later? 1 day? 1 week? 1 month? 1 year? What if we had another 50 Mrs Monkeys, is the effect consistent? What if we observed her covertly, does she sit cross-legged spontaneously when not subject to the social obligations imposed by the presence of the therapist?
Anecdotes create hypotheses and questions for testing, as I said.
BSM how lazy are you??
This took me a few seconds via Google:
http://www.ncbi.nlm.nih.gov/pubmed/22871295
Unaccustomed as I am to such “research”…..
Sham would be very difficult, as the immediate effect is quite arresting.
Pretty lazy.
But to be more serious about it, I try to avoid educating myself on an unfamiliar topic based just on such abstracts as I can find but where I don’t have access to the full text. Examining such studies requires more than a casual amount of effort. Life’s, frankly, too short.
So, your reasonable supplementary question would be to ask how much if the primary acupuncture literature have I read and the answer is very little. Fortunately there are trustworthy people who do this and I am content with that.
I have read a lot of primary literature in homeopathy, but that’s been more of an interest of mine.
Not sure about “fair prior probability”.
We should recognise that acupuncture was recommended for a wide range of conditions.
We now know the causes of many of these conditions and puncturing the skin generally won’t help.
The causes bear no relationship to the “theory” underlying acupuncture.
As such its prior probability for any condition must be close to zero.
That it is now largely being discussed for subjective symptoms like pain, should ring our skeptical alarm bell, that it has gone from treating everything to treating the things where placebos are expected to get better reports of effectiveness in the short term.
I’m prepare to believe is we look hard enough we’ll eventually find conditions where sticking needles in people makes them better (some sort of boil or cyst seems likely), but it is probably not a rational approach to the discovery of new and effective medical therapies.
‘…draw confusions…’ Hahaha! Intentional pun or Freudian slip?
Neither. Autocorrect and screen too small for state of eyesight!
Mr Monkey, I use dry needling sometimes called western acupuncture. If you attend a british medical acupuncture course you will be given some very plausible reasons as to why this form of intervention may work by a very experienced research led ex army doctor GP. Do you agree that any therapy that produces a “direct physical effect” has some plausibility? and what about if the chronic problem was given this therapy in the acute stage would it maybe not become chronic? And if we find good evidence that shows dry needling to be no more beneficial than placebo how do you explain Mrs m’s change in ROM and symptoms?
Iam not saying yes BSM beleives in acupuncture I am merely pointing out that a perceived benefit from a therapy by yourself or your wife has REAL consaquences wether the research proves it to be beneficial or not. I would be really interested in what your wife thought of any evidence that shows dry needling to be innefective, based on her experiences.
That the immediate effect was temporary and had no impact on the chronic problem. I see no difficulty with this concept.
But to repeat, we are now discussing possible effects in a narrow area of acupuncture that would not even be accepted as acupuncture by many of its adherents. And the point of this blog is that the present study showed only trivial effects for “real” acupuncture in the condition under consideration.
BSM, you say your wife has muscle problems affecting her shoulders and now her hips, and was helped by dry needling. Consultant orthopaedic surgeon told me I am my own best physio and gave me an intensive set of exercises to do by myself at home daily – I’ve got knee problems that make my muscles seize up. Has your wife gone the exercise route for her problems – after all she can’t keep a dry-needler on hand to pop out of a cupboard (like the bank manager in the advert) every time she seizes up.
To be honest, she mainly just gets on with life and tries to ignore it.
Liam, could you apply your very argument to needling?
“If you*************** and obtain a direct physical effect, it is a not unreasonable inference to suggest the two are causally connected.”
This is what I and many other have been saying for years but, is constantly met with “show me the evidence” or I don’t beleive it.
IMO its the most realistic and unbiased thing I have seen you write.
Well, it rather depends on what is under those asterisks. If it’s “howl at the moon” then we’re firmly in the territory of low prior plausibility unless, of course, the claimed effect is “being locked up as a loony”.
You have to show the physical effect. Science is a bit funny like that.
Of course the fall back from alty representaion is always…we arent doing science.
Could we have some science? Vitalism in all its forms has yet to produce anything but extremely poor EB from what appears to be doyens of “publish or perish”.
I agree that it’s very difficult to scientifically prove a physical effect. All pain scales, rom tables, disability scores etc are all perceived differences described by an individual. This is my point about the monkeys. Mrs m felt she had improved and mr m observed an improvement. Mr m finds this difficult to rule out even if the evidence for the procedure is negative. I like how mr monkey is talking about it. Most skeptics just say this rct says this so it can’t possibly work, without really investigating the subject. Find the study on google, read the conclusions then slag the therapy off.
The patient knowing that they have been stuck versus a non invasive sham methodology is an issue. I think a controlled study of acupuncture could be performed without sham controls. Do controls of acupuncture with real needle insertions – But in all the WRONG places according to the traditional Qi maps. I’m not familiar with the literature, has this ever been done?
http://www.livescience.com/3254-acupuncture-works-placebo.html
This link could take you to the study. Sorry don’t have time to check further, as a free lunch is about to begin.
Yes, the sham comparison was just as successful as the acupunture. The sham was renamed as “TCM” to avoid embarrassment.
I think it has but will have to look, If I remember correctly the BMA course used a piece of research which looked at needles in the wrong places which produced positive results which indicated the mechanism being the needle inserted rather than where it was put.
What data did they use on the NICE panel? http://www.bbc.co.uk/news/health-19622016
Benedikt Matenaer gave a good talk in Berlin http://www.youtube.com/watch?v=R_nIB-DeUUI in May.
Database of Systematic Reviews 2009, Issue 1. Art. No.: CD007587. DOI: 10.1002/14651858.CD007587
This is quoted in a reply to my obsevation about acupuncture being a placebo following an article on the BBC website today.
Is this referred to above and what is known about it.
is this the skeptic mutual appreciation society?
Yes, I think it is; a small number of idiots that think they are so clever
Why do you have the same avatar as Mario?
I don’t think Mario (or Paolo) appreciates irony.
HaHaha! coming from you that’s funny; if there is one thing you do not have on this blog is a sense of humour; you would not know irony or sarcasm if it was hitting you in the face; just a bunch of grumpy old men with delusions of competence and a lot of time to waste.
I’m always keen to learn more about the workings of my native language and, while accepting that the purpose of this blog is not education in grammar, I wonder whether you could parse the following post for me and highlight the irony and sarcasm so that my poor old brain can see the subtlety of the humour;
Paolo
September 22, 2012 at 2:08 pm
“Yes, I think it is; a small number of idiots that think they are so clever”
small grammatical mistake indeed, but i have a reasonable mastery of english, french, italian and german; i am not so sure that your old brain could utter two words in a foreign language.
Still, I think my opinion was still sufficiently clear, as Mario said: “is this the skeptic mutual appreciation society? or if you like: “è questo il reciproco apprezzamento società scettico?”
half a dozen of you making what you think are witty comments on things that nobody cares about
I bow before your mastery of language.
Do you have anything to contribute on the subject of the blog?
Paolo said:
But not punctuation, it would seem.
Love learning about new techniques thank you for sharing!