▲ 274 points
back
143 comments
The study was discussed a few days ago at https://news.ycombinator.com/item?id=20164809
I found this one from the table of reversals to be interesting:
Article: Efficacy of infant simulator programmes to prevent teenage pregnancy: a school-based cluster randomised controlled trial in Western Australia
Summary: The infant simulator is an example of persuasion technology or captology, where the use is intended to prevent teenage pregnancy.26 Their use is widespread in developed countries27 and is expanding into low-income and middle-income countries.28 However, in this study done in Australia, the infant simulator-based VIP program did not reduce teenage pregnancy. In fact, girls in the intervention group (n=1,267) were more likely to experience a birth (8% vs. 4%; HR=1.35; 95% CI=1.06-1.73; p=0.016) or an induced abortion (9% vs. 6%; HR=1.33 (1.00-1.78; p=0.049) than those in the control group (n=1,567) before they reached 20 years of age. This is a reversal of the practice of infant simulator programs to prevent teenage pregnancy.
A school district I contract with purchased some of these "infant simulators" a few years ago. I shared spooky photos of them entombed in foam in their storage cases (evoking visions of racks of "hibernating" astronauts from sci-fi) with my wife. She made a general observation that she thought some fraction of teenage girls would feel encouraged to have a baby after time w/ the infant simulator. Guess there was some truth to that intuition after all.
That would only explain part of the effect. It wouldn't explain why the number of abortions went up by 50%.
- dolls stimulate maternal instinct (child “gets clucky”)
- child gets pregnant
Then
- parents find out, recommend terminating pregnancy
- friends find out, recommend terminating pregnancy
- doctor finds out, recommends terminating pregnancy (“you are not ready for this”)
- unpleasant side effects set in, child decides she is not ready to handle pregnancy, or
- reality sets in, child decides she is not ready to be a mother/wants to finish school
- boyfriend abandons her
Basically increasing the number of pregnancies will necessarily increase the number of abortions.
The majority of every invention / attempt will be wrong in any human pursuit. Just think, 400 useless apps or pieces of software are probably released DAILY to the itunes store.
The only difference is the impact, but its still understandable. What if you had to write software that did the following:
* Fixed a bug in an existing "legacy system", deployed remotely without your knowledge. You have no logs or debug information, but you can speculate on cause and try it in small simulated / test environments (e.g. rodents). By introducing this fix you would also be changing the system in unknown ways.
* You only have a vague idea of how the code in the OS for this system works, your fix doesn't even change the code. Your only mechanism of change is injecting some foreign JS into a webform to cause ripple effects on the system for the user, meanwhile the system is setup to try to stop / avoid such "attacks" and will try to undo your fix if it can be detected by the system's "AI".
* if your fix fails in production for even one user, you may lose millions and potentially go jail, that person may die.
* You get one shot to release it to the public, after go-live it basically has to work or everything was for nothing. There is no "push this fix and everything gets better", even one small change is a complete restart (new molecule == new drug).
* If you accidentally release any information about the users, or lose their data, you lose millions and potentially go to jail.
* This software will take 10 years, a handful of scientists, a lab, some lawyers, lobbying, and a few billion dollars to make.
* Every user using your software will be on an entirely unique stack, processing all kinds of data in tandem with your software, and will not follow directions. Some will even have lots of other bugs in the system that are unrelated to your fix.
Have you ever thought that maybe not everything has to be seen through the lens of software? Not trying to be snarky, but it's something I see so often here on HN and it kind of bugs me.
Abstract:
> The ability to identify medical reversals [practices that have been found, through randomized controlled trials, to be no better than a prior or lesser standard of care] and other low-value medical practices is an essential prerequisite for efforts to reduce spending on such practices. Through an analysis of more than 3000 randomized controlled trials (RCTs) published in three leading medical journals (the Journal of the American Medical Association, the Lancet, and the New England Journal of Medicine), we have identified 396 medical reversals. Most of the studies (92%) were conducted on populations in high-income countries, cardiovascular disease was the most common medical category (20%), and medication was the most common type of intervention (33%).
The original study can be found here: https://doi.org/10.7554/eLife.45183.001
Supplementary File 2 contains the full list of reversals:
https://elifesciences.org/articles/45183/figures#supp2
I don’t know if this is a permalink due to the hash parameter but I’ll try:
https://elifesciences.org/download/aHR0cHM6Ly9jZG4uZWxpZmVzY...
The b64 of that crazy link directs you to https://cdn.elifesciences.org/articles/45183/elife-45183-sup... - maybe slightly more permanent?
I think it's bad policy for the FDA to screen medicines not just for safety but also efficacy. From this study it appears to be frequently unjustified but it gives the drug a seal of approval that has a tendency to replace the judgement of doctors and patients. It's not that the FDA decision makers are corrupt or incompetent. These are wildly complex issues that nobody can get right consistently. Better not to conceal that with centralized decision making.
Modern medicine can't even agree in broad strokes on the proper ratio of macronutrients for humans. We give them too much power when we uncritically accept their judgement on any given treatment. For me, prayer would have no effect. For others it could be a life saving intervention. The efficacy of a particular molecule for a particular patient isn't much easier than that to determine. Therefore decision making should be as decentral and computationally parallel as we can make it.
Efficacy is important as many drugs have significant side effects. If a drug is not going to actually reduce your chance of heart attack or help treat aggressive cancer it's likely not worth the trade-off with side effects.
Without some kind of efficacy standard, you could sell water as a cancer cure. That would be very bad for health outcomes.
However, I do agree that it's a difficult topic, in general. Off-label drug uses is one example.
> The efficacy of a particular molecule for a particular patient isn't much easier than that to determine.
How do you handle situations where the provider doesn’t know/care which is best either and just chooses the intervention that was the most heavily marketed to them?
Why do we see new patent drugs replace the “old and boring” generics without any proven efficacy benefit?
> For me, prayer would have no effect. For others it could be a life saving intervention.
In the same vein, I often get to read about an emerging approach called precision medicine.
Precision Medicine refers to the tailoring of medical treatment to the individual characteristics of each patient. It does not literally mean the creation of drugs or medical devices that are unique to a patient, but rather the ability to classify individuals into subpopulations that differ in their susceptibility to a particular disease, in the biology or prognosis of those diseases they may develop, or in their response to a specific treatment [1]
Even computationally parallel algorithms hopefully have a centralization point where the compulation completes. What's the equivalent for whether or not a drug is more effective than a sugar pill, given countless edge cases? I could imagine a case where a drug that was never effective is widely distributed before being calculated to be no more effective than placebo. In the meantime, such a widely distributed drug could cause significant harm as it is selected over medication with actual efficiacy or effective drugs avoided due to the side effects of the ineffective one.
It seems wrong to judge based on the number of false positives without weighing agianst all the snake oil they keep out.
Also sometimes a medicine's most important uses end up being off-label anyways. E.g. http://nielsenhayden.com/makinglight/archives/007140.html
Typically the efficacy stamp does come with whatever caveats came up during FDA approval - for example, my SSRIs came with two sheets of double sided paper listing in excruciating detail the potential side effects, and reminding me on every page that the pills may just make me want to kill myself so watch out for that.
SSRIs being top of the list of drugs that "sometimes work sometimes don't" I think are a good example of why the FDA rubber stamping efficacy are still good. I'm not convinced I'd had those sheets of paper if drug makers had the run of the market.
I've never actually seen evidence that a "free market" (i.e. drug market with no FDA) results in a better world for consumers, unilaterally.
“Therefore decision making should be as decentral and computationally parallel as we can make ”
This may work if the decentralized decisions were made based on results. But they are often made based on profits and there is a lot of money to be made in doing useless therapies.
Please, let's not kid ourselves about what actually is life saving intervention for a person. It downplays the need for funding and we're already dealing with a lot of suicides per day.
Not screening for efficacy will result in both legitimate cancer medication and homeopathic medicine having an FDA stamp of approval.
Nothing is stopping decentralized decision making, you just need to market whatever non-effective but safe remedy you're selling as a food supplement, a tea or whatever. You just can't advertise that it's a drug.
You lost me when you made the assertion that the FDA isn’t corrupt.
Most comments here are only focusing on medical devices; this is about the practice of medicine, not shitty companies!
In dentistry/endodontics it was believed for a long time that installing a post helped keep a crown and tooth intact for longer. But long term studies showed posts can actually reduce longevity. In another case, a study showed that a common type of knee surgery for pain is actually less helpful than a placebo surgery.
The overarching problem is that it's (currently) very hard to evaluate the real effect of practicing medicine on large swaths of the population, and then using those results to change what doctors do. By getting doctors into a continuous improvement cycle, we could identify costly, useless practices, and improve health.
Actual study: https://elifesciences.org/articles/45183
Seems to impact nearly all medical disciplines.. https://iiif.elifesciences.org/lax:45183%2Felife-45183-fig2-...
Remember, in research it is easy to conclude that some treatment does no good -- just do sloppy statistics. E.g., collect some data, flip a coin, and report the results of the coin flip. That will work because the coin flip is, in probability, independent of everything else including the data and treatment. Well that's not the only way to do sloppy statistics -- just pick some measures that are relevant but not very good. And use a small sample size.
E.g., there are some reports on how much data was needed by the Large Hadron Collider (LHC) to conclude that the Higgs Boson DOES exist -- the amount of data needed was beyond belief. Sooooo, anytime well short of that much data they could have conluded that the Higgs did not exist, that is, so far their data failed to show that it DID exist. Same for any treatment: If don't collect enough data, use poor measures, let the data get corrupted, etc., i.e., do sloppy work, then will fail to find where the treatment did work and conclude that it didn't.
I do remember watching a documentary about medical devices where it describes a FDA loophole; if the device you are trying to introduce is similar to a previous already approved product then the scientific rigor involved is significantly less. I wonder if this has something to do with it.
I wonder how many of these are still covered by insurance companies. You'd think they're in ideal (data collection) position to raise questions about what works and what doesn't. Certainly, the insurance cos have more incentive to raise red flags than Big Pharma.
upvoted this, but what happens with conflicts of interest? for example, what if insurance companies insure both patients (to cover medical expenses) and doctors (to cover malpractice lawsuit costs)? insurance company houses and systems are typically set up such that the house will statistically win out (a supposed small tax that pays for running the paper work, and manpower within an insurance company). In such a setting, where insurance companies insure both doctors and patients, the net motivation is no longer defending what works, but increasing throughput of interactions...
Challenging chemo:
https://www.icnr.com/articles/ischemotherapyeffective.html
Australia has a study that's pretty damning. Is that one of these practices listed?
the website you linked reads like homeopathic conspiratorial junk. I would be wary of anything they cite
It's hard to see this headline as something other than a result of The Irreproducibility Crisis.
Can anyone comment on the validity of this publication? I'm sceptical towards the site given it's "flashy" title and I'm not well enough schooled in distinguishing good vs. bad publications.
Everyone yelling about how the FDA is corrupt or should only screen for safety are completely ignorant of history and basic facts.
Medicine is not like trying out a new coding framework - in some cases, like cancer:
i) There is no ability to "try it out for your self" - you are either all in or not
ii) There are extreme risks to taking the drug (most cancer drugs are actually somewhat to extreme toxic...just less toxic than cancer)
iii) You only get a few tries (since, you know, cancer will kill you).
iv) People are very desperate and will try anything
v) Data sharing is very very hard given the sensitivity of the information (patient data) - making it effectively impossible to have an open data exchange. (This is also due to the billions of dollars at stake on the pharma side).
Try designing a system to overcome those challenges...and you will likely end up with a central governing body that looks like the FDA. Anything else you are going to get way way more drugs that don't work, which waste time and money and kill people, which endanger patient privacy, or which are not satisfactory to pharma or, ...
The fact is, the FDA is full of many of the smartest and hardest working doctors and scientists in all of medicine. They take their jobs extremely seriously. They are far more careful and rigorous than most academic or even pharma-level researchers.
Do they get it wrong? Yes, occasionally. Not frequently. Occasionally. But that also, is science. The FDA follows rigorous statistical procedures, but even p<0.05 (or <0.001) still means false positives can happen.
This. Having been through the FDA process myself (though we didn't make it to NDA, sob) I have to say that dealing with the Agency was great. Tedious, but great, and nothing seemed stupid except some trivialities at the margin (e.g. certain formatting issues).
Honestly the worst parts of dealing with them are direct consequences of people trying to game the system. We went out of our way to make sure nothing we did could give that impression and got excellent guidance and response.
>Do they get it wrong? Yes, occasionally. Not frequently. Occasionally. But that also, is science. The FDA follows rigorous statistical procedures, but even p<0.05 (or <0.001) still means false positives can happen.
The issue is that there is a huge lack of transparency. When a third of the studies involved in approval of anti-depressants were withheld from the public [0] and even some withheld at the insistence of the FDA [1], what good do low P-values do?
Hard to trust the science when you're not even allowed to review the papers for yourself.
[0]: https://www.nytimes.com/2008/01/17/health/17depress.html
[1]: http://www.washingtonpost.com/wp-dyn/articles/A9802-2004Sep9...
Having worked closely with FDA, I've been very impressed by the people there and how thoughtful they are despite the pressure they're under and the almost complete lack of good software tooling due to the sensitivity, regulatory requirements, and complexity and specialization of the data. Most of the medical officers there could easily be a very successful (and much higher paid) private physician or a pharma researcher somewhere but choose to remain at FDA.
On one point people are making specifically -- unlinking safety with efficacy is really tough -- an immunooncology product that causes serious AEs/hospitalizations in a quarter of the population and costs $250k and only extends life by 6 months may be approved because the alternative is almost certain mortality; whereas another drug may not be approved simply because it increases bleeding risk in a small percentage of people because there are good alternatives already.
With respect to cancer drugs, while generally no one can be prescribed medication that has not passed clinical trials, in the case of terminal patients who could potentially be saved by a drug currently in trials, there is a system for allowing them access to the medication.
It's called "Expanded Access" (also known as "Compassionate Use") https://www.fda.gov/news-events/public-health-focus/expanded...
The other reason why data sharing is hard is because it costs on average over $1,000,000,000 (1 billion USD) and 10 years of time to whittle over 10,000 drug candidates into a single new FDA approved NME (new molecular entity).
If you spent a billion dollars and a decade on a project, you will consider the data you collected to be a trade secret and would protect it with everything you can, because you have to now take this drug to market and make up the billion you spent making this drug. If you simply shared all your results, foreign competitors will simply copy your work without paying you, and will then offer your product without the cost of R&D and regulation built in: AKA they will charge 1/10 or less than you and you will struggle to profit anywhere they are in the market.
Thank you for this. So often we rip on government agencies from our perch on...Hacker News, when we really have no idea what's going on.
We do work with the Government in a different context, and I've found nothing but professionalism and deep expertise. In fact the people I've met in the agencies we're working with are some of the smartest I've encountered in my entire career.
Also it’s important to understand the context that medicine wasn’t originally “scientific “ for all the reasons you are touching on.
Historically treatments were based on some theory of how the body works, and a plausible intervention either based job that model, or based on tradition and anecdotal experience.
As people tried to implement more empiricism it hits all the difficulties you mention and others ... it’s an extremely difficult ugly challenge.
We should have lots of medical reversals. It means we are slowly getting better. Probably things could move faster if there was some giant anonymized database of every patient treatment, with documentation of symptoms, intervention, and outcome.
There are so many reasons why that is impractical, I’m astonished we are making the progress we are.
> are completely ignorant of history and basic facts
Sam Peltzman's "Regulation of Pharmaceutical Innovation" does an evaluation of this based on statistics, and concludes that the efficacy requirements are a net negative.
It's not about the FDA scientists being incompetent. It's about the increase in cost is so large that far fewer new treatments are even attempted. Conditions that affect a smaller number of people are simply not worth finding a treatment for.
>Data sharing is very very hard given the sensitivity of the information (patient data) - making it effectively impossible to have an open data exchange. (This is also due to the billions of dollars at stake on the pharma side).
There is promising work going on in this area to allow the sharing of models while maintaining privacy. (Differential privacy, homomorphic encryption, multi-party computation, etc.)
It's always worthwhile to learn the history of an organisation. That tells you a lot about it's culture.
https://en.wikipedia.org/wiki/History_of_the_Food_and_Drug_A...
This is true, but there may be moderate reforms that would work. Could drugs approved by European regulators be allowed without FDA approval? Which regulators do a better job?
>to actively seek out independent, governmental and non-conflicted clinical research
does this mean private sector pharma studies are to blame?
I read the original study. I have a few thoughts. My partner is a physician and I am an AI researcher working in medicine. I think a lot about doctors as machine learning models, and RCT results like loss terms in a complicated objective function.
What is the best learning rate for updating physicians (our models) from the results of RCT's (part of our loss)?
The authors reviewed all articles in three journals from (generally) between 2003-2017. They didn't, afaict (please point me if they did), review the time-to-correction (if any correction has been made). It takes some time before the results of an RCT end up in established practice. I'm actually surprised it's so small in many cases.
It's not like there's a database where the results of every RCT are immediately updated and the physician model is retrained overnight on the new data.
Even if there were, imagine if the learning rate (so to speak) were so high that every discipline immediately changed their published best practices on the basis of a single RCT?
Here's a cautionary paragraph from one of the excellent reversal studies they use:
Several limitations of the study warrant discussion. First, because we enrolled only 26% of eligible patients, our findings must be generalized cautiously. The most frequent reason that patients declined enrollment was a strong preference for one treatment or the other. Since patients' preferences may be associated with treatment outcome, our trial may be vulnerable to selection bias. Participating surgeons may not have referred potentially eligible patients because they were uncomfortable randomly assigning these patients to treatment; this form of selective enrollment may also create bias.26 Second, because the trial was conducted in academic referral centers, the findings should be generalized carefully to community settings. Third, we did not formally assess the fidelity of the physical therapists or surgeons to the standard intervention protocols. Finally, our study was not blinded, since our investigative group did not consider a sham comparison group feasible. [0]
I'm less concerned about RCT to Best Practice time than from Best Practice to Typical Physician Practice time. There is a cascaded model connected to the 'complex RCT loss' and it's discipline published practice down to individual physician treating patients. Compressing the time from RCT to individual physician is fraught with difficulties, but could be improved.
Finally, RCT is the gold standard, but it's not perfect and it doesn't always clearly translate to the individual physician's model of practice. Many best practices weren't established from RCT's either.
And an inconclusive result from an RCT is not the same thing as proving that there's no difference in outcomes, but a proper statistician can chime in there.
Last Week Tonight with John Oliver did a segment on this recently: https://youtu.be/-tIdzNlExrw