Alan Aragon Guest Article: Commercial Bias in Scientific Research
A little while back, I did a brief review of Alan Aragon’s Research Review. After hounding him for quite some time, I finally got the man to send me a guest article so that I could expose people to a little bit more of his work.
I was hoping he’d give us all the secrets to freaky lean muscle mass gains.
Instead, in keeping with his status as a research nerd (probably not as bad as me but damn close), he sent me the following piece, a look at where potential biases can enter into the world of scientific research.
For those of us who follow research (or see those long lists of references at the end of articles these days), it’s an important reminder that science can be subject to biases just as much as anything else.
In any case, if research is your gig, check out his monthly research review and enjoy the following guest article.
Don’t worry, I’ll get back to the dieting series here before too long.
****
Ghosts in the Machine: Commercial Bias in Scientific Research
By Alan Aragon, July 2008
How do we know what we know?
The acquisition of scientific knowledge is a delicate process. The range of sources is broad; from the gossip of the grapevine, all the way to the peer-reviewed literature. Ironically, the process of drawing conclusions from scientific research is more of an art than a science. While the lay press is dominated by emotionally striking topics, the scientific literature has similar underlying principles that govern publication, with a few added twists. Funding source is a significant factor influencing study outcomes. Publication bias and funding bias are often overlooked yet important considerations for sizing up the evidence. Unfortunately, in many cases these factors are pre-determinants of the fate of drugs, supplements, and other products within the sphere of human health and disease. For this reason, every aspect of research – even aspects that aren’t plainly visible – must be scrutinized through a skeptical lens.
Publication Bias
Publication bias has been defined as the submission or acceptance of manuscripts for publication based on the strength or direction of the study findings [1]. Various factors are associated with the failure to reach publication, but whether the roadblock is primarily on the part of the investigators or the journal editors is unclear. Dickersin and colleagues tracked 737 manuscripts accepted by institutional review boards, and only 6 of the 124 studies that failed to be published were reported to have been rejected [2]. In contrast, Lee and colleagues recently tracked 1107 manuscripts submitted to the British Medical Journal, The Lancet, and the Annals of Internal Medicine. 70% of the studies were rejected outright, and another 24% were rejected after peer review [3]. Only 6% actually made it to publication. In spite of the differing outcomes in these investigations, they both found ‘positive outcome bias’, a preference toward the publication of studies that showed statistically significant results.
Implications of the Problem, and a Solution in Progress
The fact that a sizable body of data goes unpublished means that a significant portion of the truth (or close to it) is left in limbo. This bias of omission gets exacerbated when transferred to meta-analyses and systematic reviews, which often do not include unpublished or “grey” research. Non-publication also can have legal implications since it breaches the agreement researchers make with study participants, ethics boards, and sponsors [4]. Realizing this problem, the scientific community has spent the last couple of decades pressuring the powers-that-be to form a centralized registry of clinical trials.
The first far-reaching breakthrough in this regard was in 2004 when the International Council of Medical Journal Editors (ICMJE) released a document issuing mandatory registration of clinical trials before participant enrollment [5]. This encouraged researchers to register their investigations, creating a portal to access unpublished data, and allowing a more comprehensive inclusion of studies in meta-analyses and systematic reviews. The World Health Organization (WHO) plans to assemble an international network of registries that will function through a web-based platform. There currently is no formal tracking system for trial registration, nor is there any system for penalizing the failure to register. Global regulation of publication bias is still in its infancy, but it’s making steps in the right direction.
Funding Bias – The Other Invisible Beast
Funding bias is the strong tendency for a commercially sponsored trial to yield positive effects caused by the sponsor’s product. A perfect model for examining funding bias is the pharmaceutical industry. A recent review by Sismondo discussed actions by researchers that contribute to sponsorship bias [6]. Tactics include ‘publication planning’, which is a calculated progression wherein drug companies and their agents influence the outcomes and publication of articles for their benefit. The details involve ghostwriters, or unnamed authors hired to assist in data manipulation, tabulation, and editing.
Fulfilling this niche, a multitude of Contract Research Organizations (CROs) advertise their expertise in efficiently shuttling manuscripts through every step of the publication process – from study design, to reporting, and finally to acceptance in the target journals. Pharmaceutical companies have transferred the majority of their clinical trial support from academic institutions to CROs, which in part function as personalized data farms. Company-sponsored research is highly susceptible to design bias, including insufficient trial durations to show side effects, treatment imbalance (unrealistic or uneven comparisons), and inappropriate doses.
“Ghost-Management” in Action
Hiring an unacknowledged third party to design and execute the full range of trial procedures can produce high-quality research, but it’s also a profound source of bias. Healy and Cattell examined the impact of Pfizer hiring Current Medical Directions (CMD) on the surrounding outcomes of sertraline (Zoloft) studies, and found a distinct advantage to outsourcing to CMD [7]. A comparison was made between CMD-assisted trials and trials done without hiring a medical writing agency. Sertraline trials managed by CMD were more numerous for the given time period (85 trials versus 41) and had a 7-fold higher impact factor, thus a higher frequency of citation by other literature. The CMD-assisted trials also had higher-profile authors and appeared in more prestigious journals. And now the kicker – only 50% of the non-CMD trials showed positive results, while 100% of the CMD-assisted trials showed positive results.
Another example of ghostwriting and compromised integrity is the case of Merck’s handling of rofecoxib (Vioxx), which the company withdrew from the market in 2004 due to safety (and legal) issues. Ross and colleagues recently examined the nature and extent of ghostwriting for rofecoxib studies from 1996-2004 [8]. Only 50% of the review articles published either a disclosure of Merck sponsorship or a disclosure of any financial compensation to the authors from the company. These articles were often prepared by ghostwriters, yet were attributed to academically affiliated investigators who often did not disclose financial support by Merck.
Investigator Affiliations & Other Slippery Slopes
At this point I’ll shift gears from the drug discussion and delve into sports supplements, an arena similarly rife with commercial influence. One of the most blatant scenarios of investigator bias is when the lead researcher of a given trial is employed by the company sponsoring the trial, and whose product is being tested in the trial in question. For example, Ronald Stanko is the principal investigator in the majority of trials showing pyruvate’s effectiveness for fitness-related purposes [9-12]. He also holds patents on the pyruvate-based compounds used in the studies.
A similar situation is seen with Steve Nissen, who has led the majority of research on beta-hydroxy-beta-methylbutyrate (HMB). Nissen is also a patent holder of HMB’s use as a sports supplement. I found it amusing that Nissen did a meta-analysis on supplements used for lean mass and strength, concluding that only creatine and (surprise) HMB had sufficient data supporting their use [13]. I was tickled when Decombaz and colleagues took issue with this and brought it to light. Let me quote their concern regarding the inclusion of HMB in the same league of evidence as creatine [14]:
“Demonstrations of efficacy produced by only a small number of separate sources are intuitively less persuasive than those from a large number of independent laboratories. Among reasons for this, we believe that investigations led by connected authors or institutions are likely to share similar approaches to a given question and use a smaller variety of methods than truly independent ones.”
A final investigator affiliation worth mentioning is Paul Cribb and the supplement company AST Sports Science. Cribb is AST’s research director, and is also the principal investigator in the AST-sponsored trials involving their whey (VP2) and creatine (Creatine HSC). Thanks to the partnership of Cribb and AST, these products have spanked some serious Pubmed ass [15-19]. I’m not suggesting that we should completely write off all research produced by company-investigator alliances. However, I am urging everyone to remain skeptical and think critically. Scientific research is like any other tangle of information – it needs to be combed for bugs.
References
1. Dickersin K. The existence of publication bias and risk factors for its occurrence. JAMA. 1990 Mar 9;263(10):1385-9.
2. Dickersin K, Min YI, Meinert CL. Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards. JAMA. 1992 Jan 15;267(3):374-8.
3. Lee KP, et al. Predictors of publication: characteristics of submitted manuscripts associated with acceptance at major biomedical journals. Med J Aust. 2006 Jun 19;184(12):621-6.
4. Krzyzanowska MK, et al. Factors associated with failure to publish large randomized trials presented at an oncology meeting. JAMA. 2003 Jul 23;290(4):495-501.
5. Abaid LN, et al. Reducing publication bias through trial registration. Obstet Gynecol. 2007 Jun;109(6):1434-7.
6. Sismondo S. How pharmaceutical industry funding affects trial outcomes: causal structures and responses. Soc Sci Med. 2008 May;66(9):1909-14. Epub 2008 Mar 4.
7. Healy D, Cattell D. Interface between authorship, industry and science in the domain of therapeutics. Br J Psychiatry. 2003 Jul;183:22-7.
8. Ross JS, et al. Guest authorship and ghostwriting in publications related to rofecoxib: a case study of industry documents from rofecoxib litigation. JAMA. 2008 Apr 16;299(15):1800-12.
9. Stanko RT, et al. Body composition, energy utilization, and nitrogen metabolism with a 4.25-MJ/d low-energy diet supplemented with pyruvate. Am J Clin Nutr. 1992 Oct;56(4):630-5.
10. Stanko RT, et al. Body composition, energy utilization, and nitrogen metabolism with a severely restricted diet supplemented with dihydroxyacetone and pyruvate. Am J Clin Nutr. 1992 Apr;55(4):771-6.
11. Stanko RT, et al. Body composition, nitrogen metabolism, and energy utilization with feeding of mildly restricted (4.2 MJ/d) and severely restricted (2.1 MJ/d) isonitrogenous diets. Am J Clin Nutr. 1992 Oct;56(4):636-40.
12. Stanko RT, Arch JE. Inhibition of regain in body weight and fat with addition of 3-carbon compounds to the diet with hyperenergetic refeeding after weight reduction. Int J Obes Relat Metab Disord. 1996 Oct;20(10):925-30.
13. Nissen SL, Sharp RL. Effect of dietary supplements on lean mass and strength gains with resistance exercise: a meta-analysis. J Appl Physiol. 2003 Feb;94(2):651-9. Epub 2002 Oct 25.
14. Decombaz J, et al. HMB meta-analysis and the clustering of data sources. J Appl Physiol. 2003 Nov;95(5):2180-2; author reply 2182.
15. Cribb PJ, et al. A creatine-protein-carbohydrate supplement enhances responses to resistance training. Med Sci Sports Exerc. 2007 Nov;39(11):1960-8.
16. Cribb PJ, et al. Effects of whey isolate, creatine, and resistance training on muscle hypertrophy. Med Sci Sports Exerc. 2007 Feb;39(2):298-307.
17. Cribb PJ, Hayes A. Effects of supplement timing and resistance exercise on skeletal muscle hypertrophy. Med Sci Sports Exerc. 2006 Nov;38(11):1918-25.
18. Cribb PJ, et al. The effect of whey isolate and resistance training on strength, body composition, and plasma glutamine. Int J Sport Nutr Exerc Metab. 2006 Oct;16(5):494-509.
19. Cribb PJ, Hayes A. Effects of supplement timing and resistance exercise on skeletal muscle hypertrophy. Med Sci Sports Exerc. 2006 Nov;38(11):1918-25.
Thanks for the article. Another problem is that the conclusions and summaries written up by researchers can sometimes misrepresent the data, and just about everyone (journalists especially) take these summaries at face value not having the time, inclination, or level of nerdiness to review the study in its entirety to determine if the conclusions match the data or if the study was well-designed in the first place.
To add to that, citations beget citations, so if a flawed study receives a few citations it can start a snowball effect to the point where researchers begin to have unquestioning faith in the conclusions based solely upon the number of prior citations.
What I would like to see is a public resource that acts almost like amazon.com, where people can comment on a given study, point out design flaws and potential conflicts of interest, and so forth. Maybe this is unrealistic or a bad idea for reasons I can’t think of.
[...] Original Lyle McDonald Speaks [...]
[...] site where he delved into the issue of biased research—put your thinking caps on for this pile of gold, and no, I’m not faking it). Bottom line: our societal weight will continue on [...]
[...] article Great article by Alan I think Alan continuely gets better in his writing Lyle McDonald – Guest Blog: Article by Alan Aragon | Lyle McDonald Speaks thanks Coach [...]
As someone who works for closely with field-op CRO’s, I can tell you that many of them make an honest effort to collect efficacy and safety data closely. Obviously, what the sponsor does with the data is the real issue here but I can say that most of them are very thorough, especially when collecting AE’s. Of course, this also depends on the protocol and if the IP is post market or not. I do many phase 2 and 3 trials and I am a stickler for collecting very detailed AE information in an attempt to ensure that causality can be affirmed or dismissed once the data is cleaned and crunched. It does seem futile at times though, considering that initial AE collection is done my data mangers at the clinical sites who are not adequately trained to procure these events. So many times the principle investigator of the trial for the site will sign off on the types of events w/o checking to see if more needed to be added.
Secondly, I’m personally very suspect of Cribb and AST’s research, at least the early stuff. I remember when they were blatantly advertising their WPI as a 100% hydrosylate, which was obviously false, especially if you’ve ever tasted a pure peptide whey protein before. Then they conducted their own research (which was a poster presentation that was published as an abstract in MSSE) that showed that subjects taking 1/5 g/kg of BW whey protein (which is a shit ton) gained 11 pounds of lean tissue while losing 3 pounds of fat in 11 weeks! Please. Trained subjects add 500 calories to their diets and they lose and average of 3 pounds of fat while the casein group shit the bed of course. This was nothing more than in-house data that was presented at a meeting and was picked up for publication with other abstracts in MSSE. I don’t know. That’s more than most supp companies do but I’m very dubious of those results…
skeletal muscle tissue…
* Long-chain polyunsaturated fatty acids are also less likely to be stored as fat. Efficient Energy Production: Lean Lipids are special…