How Do You Evaluate Health Claims?

pom_wonderful_ad_540It seems like every day there is a new health fad people are talking about on Facebook, mentioning in the office, or being made into a documentary of questionable truth value. Many, if not all, of these play on our deep wish to be in control of bodies that simply refuse to do as we say. And in a society that is deeply unjust, that is facing increasing anxiety about climate change, fears of nuclear war due to a certain orange monster, and unreasonable capitalistic expectations when it comes to labour and work/life boundaries (or lack there of) these fads gain traction quickly. We all want that magic bullet that will insulate us from disease, pain, and suffering. And most of us are not trained in evidence-based medicine. Very few of us, unfortunately, are given the tools to properly evaluate health claims. So I’m offering a crash course in evaluating health (or other scientific, but I’ll focus on health) claims.

Here is the list of questions I’d recommend you run through any time you’re facing a new health claim.

1. Who is saying it?
Is it a doctor? Is it a professional organization? Is it your Aunty Patty or the lady from your mom group who calls fluoride evil? Is it a blogger who, coincidentally, has a coaching plan/supplement line/ebook that will solve this very problem you never knew was a problem?

Ideally, ideas and action plans for health issues are coming from either a doctor (or other regulated health professional trained in evidence-based medicine), a medical organization (e.g., the Centre for Addiction and Mental Health), a professional association (e.g., the American Medical Association), a regulatory body (e.g., the College of Physicians and Surgeons of BC), or a research organization (e.g., the Cochrane Collaboration). If it’s coming from “sources,” your friend who’s really into juicing, a blogger/lifestyle guru/health coach, or alternative medicine provider you should be wary and continue asking the questions below.

2. What are they saying?
Are they claiming to have found the magic bullet? Is it a diet that will solve everything from fatigue to endometriosis while preventing cancer and boosting your IQ? To paraphrase the wonderful podcast Sawbones (which you should really check out if you’re interested in this kind of thing), anything that claims to cure all probably cures nothing. Is it making a very strong, definitive statement about nutrition–especially a single food or nutrient?

3. Do they cite any evidence?
How do they support their claim? Do they support their claim? Is it anecdotal (e.g., my best friend/client/I did this thing and now I’m cured!)? Is the only support a labyrinth of links to other blogs making the same claims? Can they point to a study? A series of studies? A meta-analysis? Or is it leaning on poor “common sense” (mis)understandings of health and the body (e.g., acids are “bad”, ergo alkaline water is health-enhancing)?

4. Is it a one-off study or a meta-analysis?
Let’s say they do cite some evidence (hurray!) now we need to take a critical look at that evidence. Is it a one-off study? Is it a series of studies? Is it a meta-analysis?

The problem with a single study is that it might be a fluke, never to be repeated (and what we’re really looking for with scientific studies is repeatable findings. That tells us it wasn’t a fluke and there actually is a connection there). And/or it might be an example of p-hacking. Or it could even be a reference to a paper that doesn’t in any way support the claim made. Using the PubMed database or GoogleScholar will usually bring you at least the abstract, which will give you an overview of what they were looking for and what they found. This isn’t enough, on its own, to determine the quality of the evidence, but it will at least tell you if it’s in the ballpark of what’s being claimed.

Better than a single study (and, anecdotally, I’ll tell you that a lot of the health documentaries I’ve seen cite single studies over and over again) is a series of studies all confirming the same thing. Again, this replication of findings indicates that it’s very unlikely to be a coincidence. (Now, there is a whole other conversation to be had about the publication bias that leads to few replication studies being run, but replication is a basic and foundational thing to look for when assessing a claim).

Best of the bunch is a systematic review, which takes ALLLLLL the studies on a given subject, reviews their quality, and then synthesizes the findings to come to a conclusion about the evidence. For example, the weight of the evidence is “suggestive of a small but potentially important reduction in cardiovascular risk on reduction of saturated fat intake.” Which is counter to what all the paleobros are saying to justify eating bacon wrapped bacon with a side of bacon (which I don’t think cavemen ate?). For another example, unlike what a google search will tell you, we just don’t know if acupuncture works for insomnia or not.

But systematic reviews are big endeavours that take a long time (at least a year is what I was told in grad school), and they don’t exist for every topic under the sun. So let’s say you’ve found one or two studies that look promising in supporting this health claim.

5. What does the rest of the literature say?
Can you find other evidence that supports this claim? Is it in direct opposition to every other study published? A few months ago I accidentally got into an argument on the internet (always a bad idea) with someone flirting with AIDS denialism. Now, I work for an organization that, among other things, does HIV/AIDS research, though my area does not focus on HIV/AIDS, so I’m not immersed in the literature in the same way that I am in the field I work in, but I’m aware of the broad strokes and of recent advances. And I know how to evaluate claims and evidence. And the claims this person was making did not accord with ANY of the published evidence in thirty+ years of HIV/AIDS medicine. To me, this was a clear indication that he was deep in the weeds of moralistic, anti-scientific quackery.

6. Is this finding possible? 
So let’s say you haven’t found much evidence to support this health claim, maybe a single study or several anecdotes, but you really, really want to believe it’s true (I’m not knocking this, I have PCOS and the internet is overrun with quackery and woo, largely, I think, because there is little good science about how to treat it or even, really, what it is, and it is tempting to run with everything you find three pages deep in an internet forum). You should already be viewing it skeptically, but let’s also use some good old fashioned critical thinking here.

If it’s hailing a panacea you can probably assume it doesn’t cure anything. If it’s demonizing a single food or nutrient you can almost assuredly dismiss the health claim. See, nutrition research just isn’t really capable of that kind of specificity. Partially because nutrition research takes a LONG time (what’s the lifetime consequence of eating kale? We need a whole lifetime to look at it!) and because there are so many confounding factors (were the kale findings actually about kale or were they findings about the olive oil you cooked it in and the potatoes you served it with and the cake you ate afterward? Maybe they were about the class privilege–and thus access to healthcare and education–of people who have time to cook kale? Maybe they were findings about traditional foodways that include kale?). Plus, randomized controlled trials (the gold standard of medical research) for food are nearly impossible so most food studies are observational, relying on participant report in the form of surveys. Do you remember what you ate yesterday? How about three weeks ago?

So it’s hard to separate out a single food and make a reasonable claim about it. What do we know about nutrition, though, given that there is a new wild claim every week? Here’s the consensus of leading nutrition and food systems experts:

The overall body of evidence examined by the 2015 DGAC identifies that a healthy dietary pattern is higher in vegetables, fruits, whole grains, low- or non-fat dairy, seafood, legumes, and nuts; moderate in alcohol (among adults); lower in red and processed meats; and low in sugar-sweetened foods and drinks and refined grains. Additional strong evidence shows that it is not necessary to eliminate food groups or conform to a single dietary pattern to achieve healthy dietary patterns. Rather, individuals can combine foods in a variety of flexible ways to achieve healthy dietary patterns, and these strategies should be tailored to meet the individual’s health needs, dietary preferences and cultural traditions. Current research also strongly demonstrates that regular physical activity promotes health and reduces chronic disease risk. (Source: 2015 DGAC summary wording)

Turns out, it’s that same boring stuff we’re told over and over. Eat lots of veggies and fruits, whole grains, and lean protein. Don’t eat much sugar, refined grains, or red meat. Move regularly.

But that doesn’t get clicks or sell ebooks or diet plans or documentaries, does it?

7. Is it using equivocating language like “may”?
My final tip is to look at the language of the health claim. Any time you see someone saying something “may” do x, that means there isn’t enough evidence to say it does. Now, this could be that a correlation has been found but no causative relationship identified, it could be that it’s preliminary research and more well-designed studies need to be run, or it could be speculation about the cause of a finding with no real evidence base for the speculation. “May,” “could be,” “it’s thought that” are all red flags that we’re not dealing with evidence-based claims.

Anything I’m missing?



One thought on “How Do You Evaluate Health Claims?

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s