I recently read an article titled “When Evidence Says No, but Doctors Say Yes” . In it, David Epstein writes about how:

…it is distressingly ordinary for patients to get treatments that research has shown are ineffective or even dangerous.

Sometimes doctors simply haven’t kept up with the science. Other times doctors know the state of play perfectly well but continue to deliver these treatments because it’s profitable—or even because they’re popular and patients demand them. Some procedures are implemented based on studies that did not prove whether they really worked in the first place. Others were initially supported by evidence but then were contradicted by better evidence, and yet these procedures have remained the standards of care for years, or decades.

For over 8,000 words, Epstein explores this phenomenon of doctors prescribing everything from drugs to surgeries even when modern medicine would suggest that those treatments are unhelpful or even harmful. What I found striking as I was reading this piece was how similar the problems sound to my own field of financial planning and investment management. This series is exploring a few of the similarities.

Lesson 4: Reporting Can Lie

Epstein explains that most treatment options are either net neutral or more likely to be harmful. He writes:

“Most people struggle with the idea that medicine is all about probability,” says Aron Sousa, an internist and senior associate dean at Michigan State University’s medical school. As to the more common metric, relative risk, “it’s horrible,” Sousa says. “It’s not just drug companies that use it; physicians use it, too. They want their work to look more useful, and they genuinely think patients need to take this [drug], and relative risk is more compelling than NNT. Relative risk is just another way of lying.”

…Even remedies that work extraordinarily well can be less impressive when viewed via NNT. Antibiotics for a sinus infection will resolve symptoms faster in one of 15 people who get them, while one in eight will experience side effects. A meta-analysis of sleep-aid drugs in older adults found that for every 13 people who took a sedative, like Ambien, one had improved sleep—about 25 minutes per night on average—while one in six experienced a negative side effect, with the most serious being increased risk for car accidents.

Relative risk is a measure in the decreased risk of an adverse outcome under treatment. It is calculated by the observed incidence of an event in the experimental group under treatment divided by the incidence in the control group with a placebo.

NNT, which stands for “number needed to treat,” is how many patients would need to receive the treatment to prevent one adverse outcome from happening. It is calculated by the inverse of the incidence proportion of the control group minus the incidence proportion of the experimental group.

The ideal NNT would be 1, meaning each patient you treat avoids the adverse event. In contrast, there is no ideal relative risk reduction, because the theoretical ideal of a 100% risk reduction is impossible to achieve. Some people in the control group will avoid the adverse event or some people in the experimental group will have the adverse event for other unknown reasons.

Using the same study results, the relative risk reduction might be 22% while the NNT might be 250 . A 22% reduction in the probability of adverse events sounds fairly good while you have to treat 250 patients to have 1 person avoid the adverse event sounds like a fairly bad treatment.

Then when you look at NNT, you can find drugs where it is more common to have an adverse side effect than it is to cure the original symptom or illness you were targeting, even if the relative risk reduction is high.

This is why Sousa says that relative risk is just another way of lying. There is an incentive to lie when your methodology is bad.

The same happens in financial planning. Past performance reporting can be manipulated, and there are many ways that fund managers can design their fund to appear to beat their supposed peers. You can see an example of that manipulation here.

It is a mistake to select, review, or judge a fund based on recent or short-term returns. In order to get meaningful statistics, you need to use the longest time horizon possible. Even 30 years is not long enough to judge which investment will have a higher mean return for the next 30 years. For example, we recently had a 30-year time period where long-term bond returns beat the return for stocks even though when you look at longer time periods stocks beat bonds.

Even if you grant some “moderate predictive power” to the various ratings, rankings, and stars, lower expense ratios are better at predicting future returns.

Morningstar Stars are based on a fund’s 1-, 3-, 5-, and 10-year returns, but even 10 years is not long enough to determine a fund’s statistically superior performance. And while mutual funds and their managers will continue to tout their high ratings and rankings, you should ignore them. Every commission-based mutual fund salesman knows how to build a portfolio of six mutual funds all with an excellent 5-year track record from their company’s family of funds. Having more stars and a good 5-year track record increases sales, but it has little to do with your chance of meeting your goals.

You deserve a custom asset allocation made with your specific goals in mind. You deserve diligent contrarian rebalancing. You deserve a fiduciary standard of care. You deserve a financial advisor that passionately believes in the free market. You deserve an investment manager who is dedicated to honest reporting.

You deserve better. Call us today or find a NAPFA-Registered Advisor in your area.

Photo by Kendal on Unsplash