▲ 333 points
back
283 comments
There should be Nutrition Facts but for scientific trials. Independent agency just publishing quality assessments of the trial.
This should be an async non blocking evaluation. The statisticians who do it should be anonymous by default. There should be an appeals process for a scientist to explain why an unconventional new method is actually robust.
There should not be a single number published by this process, but rather a list of stats that speak to the overall quality of the trial on many dimensions (power, sources of bias, etc).
Only information that would not be the same on 99% of trials should be written on this label (no sec style everything is a risk word vomit disclosures).
There should not be a pre-emptive application for a label - it can only be gotten after paper submission to reduce gaming.
There should be an independent advisory org that scientists can literally call to ask for advice on structuring the trials. These calls must not be disclosed. Much like farmers can call the government to ask for help on xyz crop problem.
And these labels should never be used as the primary source of punishment. Any and all sanctions/penalties/dismissals must go through a new review process done by a different group.
Any scientist who gets a label in a particular year should be given a vote to review the review agency on several dimensions. These aggregate reviews should be published broadly but not trigger any automatic consequences.
Clear, accessible information is the basis for any self regulating human system. We need more of it in this field.
There's the Cochrane Collaboration. They don't tick off every item on your list but it's fairly close to what you're asking for. It's mentioned in the article as they do a lot of meta-studies. Unfortunately they only started trying to spot fraudulent RCTs in 2021. Also in recent times some people don't like them, because they did a big review of mask studies and found there was no reliable evidence that masks worked against COVID.
Cochrane (formerly known as the Cochrane Collaboration) is a British international charitable organisation formed to organise medical research findings to facilitate evidence-based choices about health interventions involving health professionals, patients and policy makers.[4][5] It includes 53 review groups that are based at research institutions worldwide. Cochrane has approximately 30,000 volunteer experts from around the world.[6]
The group conducts systematic reviews of health-care interventions and diagnostic tests and publishes them in the Cochrane Library.[7][4]
The success of the Nutrition Facts labeling does not get enough publicity.
Rather than outlawing certain ingredients, or creating some kind of health score which a product must be above, Nutrition Facts is a way for suppliers to attest to information about a product in way that is legally binding. If it isn't accurate the penalties are steep.
Consumers then have the information they need to vote with their wallets. Markets cannot function properly without symmetric information, and Nutrition Facts essentially creates a functioning market where one did not exist previously.
Any effort to regulate what's in food would probably be better spent expanding what must be in the Nutrition Facts label. I guess it's nice that we are finally getting around to banning artificial trans fats, but anyone who can read has been able to keep those out of their diet for years. The same can be said about the next bad ingredient, and the one after that.
> There should be Nutrition Facts but for scientific trials.
No, there should be prison time for scientists who conduct unethical trials or publish fake results.
The public (and policy makers) place such immense trust in those people and what they publish that nothing less is even remotely adequate.
When someone puts arsenic in food, they go to prison – labeling the food with "contains arsenic" doesn't cut it.
Do this and watch science magically fix itself.
Why would the statisticians be anonymous? I'm aware of at least a couple cases in which an independent set of statisticians were provided the data from a clinical trial specifically for a re-analysis. In one case, they showed some pretty concerning inconsistencies and the other confirmed no effect on the primary analysis, but suggested some sub-populations that might have shown an effect if a future study was properly powered. That follow-up clinical trial was just published showing pretty remarkable effect in the sub-population. I don't think there's reason to believe either independent analysis was anything other than independent.
There is already a mechanism for companies to submit questions to the FDA prior to clinical trial initiation. I'm not in these conversations, but I know the type of questions can be things like: would you accept this endpoint as a proxy for this indication, would you be satisfied with the effect size we expect, and are there other safety concerns you would expect us to evaluate other than those in our current plan. I assume EMA and other regulatory bodies have a similar process, but I'm not positive.
Disclaimer: I work in pharma, but pre-clinically. I am not involved in these clinical or regulatory issues.
Yes, I have said this as "The UX of [medical] study papers is terrible". Some people do not agree, they think that it should not be made easier to understand, that non-experts cannot really understand medical studies, so they should not be more approachable. I think that's dead wrong.
Im struggling to see the difference between this and the current FDA process, and think it is 90% the same.
Drugs have "prescribing information", referred to in industry as "labeling", which follows a consistent format containing safety, trial results, side effects, and mechanism of action.[1] I recommend people read them for drugs they take.
This should be an async non blocking evaluation. The statisticians who do it should be anonymous by default. There should be an appeals process for a scientist to explain why an unconventional new method is actually robust.
Third party analysis is the main difference here. In the current state, firms run the analysis for FDA review using standard practices, and must explain and get approval for any unconventional methods
>There should not be a single number published by this process, but rather a list of stats that speak to the overall quality of the trial on many dimensions (power, sources of bias, etc).
Labeling contains many relevant numbers. Trial sizes, how many per arm, what was measured, and and final results. Maybe there could be some squishy qualitative summary, but that seems more risky. I would rather know that 1 out of 20 patients died than it got a "2" on the safety scale.
>Only information that would not be the same on 99% of trials should be written on this label (no sec style everything is a risk word vomit disclosures).
Labeling contains drug specific information.
There should not be a pre-emptive application for a label - it can only be gotten after paper submission to reduce gaming.
Drug labeling requires pre-application and and a standard 12 month review period by the FDA prior approval
There should be an independent advisory org that scientists can literally call to ask for advice on structuring the trials. These calls must not be disclosed. Much like farmers can call the government to ask for help on xyz crop problem.
The FDA provides advice on structuring trials and acceptable design, size, power, endpoints. Firms do this by scheduling calls with FDA staticians and experts. [2]
And these labels should never be used as the primary source of punishment. Any and all sanctions/penalties/dismissals must go through a new review process done by a different group.
Maybe there is a difference here. I'm not sure what you mean by punishment? In the current system, The FDA can use the label as "punishment". The FDA may require addition "black box warnings" for drugs that are found to have serious side effects (e.g. high chance of death). They can also pull the label entirely, meaning the drug can not be sold.
Any scientist who gets a label in a particular year should be given a vote to review the review agency on several dimensions. These aggregate reviews should be published broadly but not trigger any automatic consequences.
This is basically how it works for medical device labeling in the EU. There are several "notified bodies" [3] which are private agencies to review the safety and efficacy. The firm then takes their mark of approval to the government agency.
https://www.accessdata.fda.gov/drugsatfda_docs/label/2014/20... https://www.fda.gov/media/72253/download https://climedo.de/en/blog/list-of-mdr-certified-notified-bo...
They say "medicine". I would say science in general, perhaps we could generalise even further to "any human activity is full of unethical people trying to exploit it". But 25%!? That suggest there is a big problem with how we "do science". Unfortunately I have no solution to the problem. Publishing everything (including raw data) for every research would probably help somewhat, but only teams repeating experiments/trials would ensure it.
I wonder if we suddenly took 10% of all money spent on science (let's say in medicine) and instead of novel research we used it to redo randomly chosen previous research. Would we loose or gain in terms of new cures? And if we gained, what if we spent 15%, or 25%? That's a great idea for a scientific study to find a point of diminishing returns on "research verification".
Would someone please write a research grant request for this?
Of course there are big problems with how we do science. Much of it is garbage. Imagine most software was written by junior engineers, without any code review or input from seniors. That is today's science.
Most scientific legwork is done by absolute beginners, i.e., graduate students. They often lack a support structure to focus on what they have learned so far. Most of the world is not Oxbridge, MIT, Stanford.
Where are the beginners' supervisors, you might ask? Chasing the latest trend to secure funding. Pondering how their line of research can be formulated as buzzword-du-jour markov chain. Ass kissing the dean to get department funding.
Having worked in research for 15 years, I am certain about two things: (1) The scientific method yields better results than doing things freehand. (2) Randomly axing 50% of academia would improve the situation.
That 10% investment would have a huge payoff too because it would shut down avenues that were only opened due to p-hacking long before they'd had a chance to seek further investment.
Another good reason why you should learn how to read studies:
https://peterattiamd.com/ns001/
https://biolayne.com/reps/how-to-read-research-a-biolayne-gu...
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7392212/
And as a bonus, read books by controversial figures who talk about these challenges through their own published studies and decided it isn't worth fighting.
What would be cool is to see "trust indicators" as part of a study's metadata as it is reviewed through time and continually shared by others. Could be a "study health score" or a checklist showing the study isn't biased heavily by sponsors, methods, or misleading language.
Especially given that medicine is becoming more personal with the advancements of AI and accessibility of tests, someone should be able to understand the health of a study at-a-glance reviewed by peer reviews rather than disclosed only by the authors.
A primary trust indicator of any research result is the reputation of the journal publishing the paper. Better journals demand more, not just in terms of the impact/insight of the results, but also the rigor of the experimental methodology used and how well the data was curated and the confounding variables identified and isolated.
And of course, the criteria for observational-based research differ a lot from mechanism-based, especially since the former can't control for nearly as many variables. The same goes for simulations or interpretive modeling, where experimentation is minimal.
Research studies differ quite a lot in how precisely the mechanism of action is identified, isolated, tested, and results interpreted. IMHO, blackening the trustworthiness of all of science is unhelpful, especially when some models and methods are surely more trustworthy and replicable than others. Better to identify and catalog specific sources of error (or imprecision) in order to remedy them than to just toss the baby.
If you read the article, the headline is wildly editorialized.
Whatever. Par for the course in 2023, right?
But this is Nature, a paragon of scientific literature, fueling the distrust of medicine.
There are great reasons to be skeptical of all trials and strengthen peer review and transparency, but this kind of headline is editorial malpractice, in my opinion.
No, not really. The headline says "plagued with", the subhead says "in some fields, at least one-quarter of clinical trials might be problematic or even entirely made up", and the article substantiates that. One quarter of all trials is more than adequate to plague the whole enterprise, and there isn't a problem with this headline.
Agreed that the headline makes it sound like a strong majority, when the article isn't nearly as strong on that.
Still, "Carlisle rejected every zombie trial, but by now, almost three years later, most have been published in other journals — sometimes with different data to those submitted with the manuscript he had seen. He is writing to journal editors to alert them, but expects that little will be done." is concerning. I'm almost afraid to know what the list of rejected papers covers.
We should be distrustful of medicine, of all science in general. Ignoring flawed methodologies or inconclusive results just means it's no longer science, it's ideology. I'm a physicist though so perhaps I could be overly jaded about science and peer review compared to most scientists
it's 2023, we have the means to cheaply record and store audio and video evidence for basically any medical experiment. we can record every patient reaction and opinion without relying on the reasearchers' hearsay. we also have the means to store and distribute all the binary/textual raw data gathered throughout the experiments.
maybe as an intermediate step we could make available all the recordings to the peer reviewers and only offer the raw experimental data bundled in the paper publicly? maybe in the future we can have 1TB studies without breaking a sweat? maybe all the money we give to publishers can be spent on servers to archive all the primary data so at least we aren't simply filling the pockets of MBAs?
This is a problem everywhere where the raw data isn't released (suitably anonymized).
In cognitive science, psychology, even computer science / ai / ml, business.
And the problem with rejected papers getting in somewhere else while being total garbage is pervasive. I've rejected a lot of papers because they were mathematically or statistically bogus only to see them get published elsewhere where reviewers were not so careful (a few times in Nature and Science).
We need an open science movement where you must release everything with your paper. The full pipeline to reconstruct every single result from the raw data. No hiding data. No hiding fmri scans. No "our code only runs on our machine". Etc.
I now know of a few more people who have completely lost both of their hearing after taking just one Wellburtin-class pill.
At the time, no mention was made in the pill’s warning pamphlet.
It is still difficult to secure a class-action suit in America.
Meanwhile, such quality of life would plummet into a silent world, even if one knew American Sign Language fluently beforehand, that tidbit can go against the victim in court.
There unfortunately are too many perverse incentives that encourage fraudulent studies. Everyone should take findings "established in the literature" with a grain of salt. We should also incorporate more intuition and first-principles reasoning, i.e. obesity is a harmful state for humans even if 100 RCTs proved otherwise.
Ironic that nature publishes this despite being guilty of constantly promoting it.
It’s hard enough to run a clinical trial guys. It literally costs $100 million at a minimum, yet the requirements that make it cost this much are not enough. We are basically going to regulate new medicine out of existence.
“I think journals should assume that all submitted papers are potentially flawed and editors should review individual patient data before publishing randomised controlled trials,” Carlisle wrote in his report.
This should be considered part of every journal's idea of due diligence and this shouldn't be a new idea. Shysters, con artists and snake oil salesmen have been around for a long time. The purpose of a Journal is to publish reliable information and weed out the garbage. How can you do that if you are not looking at the entire picture?
Raw data examined: Ok 56%, Flawed 18%, Zombie 26%
Raw data not available: Ok 97%, Flawed 2%, Zombie 1%
Perhaps it's not good to call an unknown as "Ok"? Maybe Carlisle should add his own paper to the mix?
Have a look at Derek Lowe's excellent blog [0] to view the depths of malfeasance, if not outright fraud, in which "studies" are created.
[0] https://www.science.org/content/blog-post/fakin-it-modern-wa... is just his latest in a long string of well-documented posts.
To be fair, science is now a “publish or be damned” business. So many papers are not worthy of publication and, frankly, are of common knowledge anyway.
Of course, this is driven by the money backing the research which, I’m sure, this is why clinical trials in the medical areas can be of a poor quality.
People want to keep their jobs.
There should be a paperswithdata for medicine like there is a https://paperswithcode.com/ for data science.
That and lawyer driven development.
I have multiple sclerosis, and I never know if a new medication came out because it's actually better or simply because they could get a new patent for a slightly modified molecule.
With the risk of starting a flame war, we recently had several well-publicised clinical trials that reported 95% efficacy of some certain modality. Yet, in reality, efficacy as defined in the trial turned out to be closer to 0%.
Instead of investigating what in the design and execution of the trial led to such a discrepancy, the problem was handled by denying there was a problem, changing the goalposts, reporting ad-hoc hypotheses as facts, silencing all critics, and forcing the public to take the modality anyway or lose jobs, school, and freedom of movement.
It was interesting to find out that test subjects who pass away during clinical trials are excluded, erased, from the trial records. Their participation and data is not included in the final results, as if they were never in the trial to begin with.
Many pharmaceutical companies have policies which prohibit patients from participating in studies for anti-depressants if they have suicidal ideations. In practice, this significantly reduces the participation of any patient with depression.
Even if they aren't faked, the ability to shut down a trial that isn't delivering the right preliminary data is a big problem for society. Why should we trust these drugs?
I will start my own medical 'system' and be my own doctor. To hell with the mainstream and it's corrupt/stupid adherents.
Nutrition for me it a big issue. It seems like and ideal area for scientific and medical study to give us light, but seems so hard to get truly objective info. Fat is the devial/good/bad/OK, sugar is the devil/bad/ok, etc. Should I avoid white bread like a hole in the head, or is it fine? Too many agendas and not enough truth.
More and more it seems that science is being clouded by moneyed interests and greed. If we can't trust science, what can we trust?
This is entropy in action. We're not going to move forward as long as malicious behavior is rewarded with money in large-scale systems. I suspect in coming decades more incurable chronic illness will become rampant and polypharmacy will become more common and necessary for larger percentages of the population. It is really sad to see.
How can you do science if around a quarter of the data is just straight up noise? Even when analysing large amounts of studies the results becone contaminated very easily.
And why is it not standard practice to provide anonymous data or even publish the data? What reason exists for that? So that only the researchers them selves can analyze it?
I'm not too knowledgeable when it comes to how scientific experiments/trials are done - are the people who collect the data, the people who interpret the data, and the people who fund/benefit from the data different parties? Or are they the same people?
Note that the article discusses research studies and not clinical trials for drug or device approval.
These research studies are important (look at how many were conducted on COVID-19 over the last few years) but are typically not held to a particularly high standard, as with most science. Which doesn’t excuse bad data or poor statistics (the latter supposedly supposed to be picked up in peer review).
It doesn’t help that front-line doctors are expected to publish.
I don't think it makes sense to publish the results of clinical trials and other scientific experiments until after they have been independently replicated.
Masters and Phd requirements should include repeating another research and checking whether they can recreate the results
I'd trust the NHS if the National Institute of Clinical Excellence (NICE) actually published their minutes online. Most employees only know what they have been taught, so whilst I agree with the headline that some trial data is dodgy, it doesnt just end there.
There are multiple pathways to factor in, there is redundancy built in, ie secondary pathways, there is the fact that not all chemicals go where intended (best highlighted by radioactive isotopes). When looking at the history of patented medicine, this really started between WW1 and WW2, before WW1, most GP's prescribed what was found in the body on a like for like basis, and in some cases prescribed organs in various forms, like desiccated thyroid gland for thyroid related problems, for pernicious anaemia, raw liver used to be prescribed to women and so on, but that has its problems like contamination and diseases.
So from WW1/WW2 onwards the rise of patented medicine took hold, but the main problem with patented medicine is the human body hasnt evolved to use these new chemical compounds in the same way as unpatented chemicals which have been around for thousands of years. And todays GP's dont really highlight the side effects of the patented medicine, and because they simply dont ask what you have been eating and drinking etc, they have this hubris which sucker punches your trust and sucks you in like a black hole, until before you know it you are on half a dozen different drugs, your quality of life is going down the pan and you've been left a zombie wondering where did Hitler go wrong modelling the German state of the time on the British Empire? You Americans complaining about the cost of healthcare, should count yourself lucky those insurance companies are looking out for your long term interests and theirs!
Fortunately, hospitals in other countries publishes studies and as english seems to be the main language used for science, I have to tip my hat to the Chinese who are roaring up the charts in terms of investigating and publishing relevant studies that will complement a quality of life one hopes to achieve, and we cant forget Wikipedia, Pubmed and Google for connecting users with pertinent studies.
Saying that I do sometimes wonder if something like ChatGPT has written some studies due to the poor quality of english used, but generally they stand out like a sore thumb.
Anyway, does any know why there is a connection between MDMA, blood clotting and Manganese?
In the age of QAnon and Alex Jones and science and vaccine deniers we can’t have things like clinical studies be corrupted. Ugh.
The over prescription of statins is a great example of what pharma sponsored clinical trials result in
No. no. This is not permitted. Trust the science or you're an antivaxer, Trumper, denier, racist, white supremacist.
Media and gov, or media at the behest of gov (as the twitter files prove, but it was obvious without them), censored as much as thet could, everything that went against what they wanted to push. It wasn't science but security/hygiene theater and it worked, because people did go along and did turn on their neighbours who opposed the measures.
But now we slowly get tidbits of things we can debate again ... Funny that, but no recognition that the entire lockdown, masks and vaccine mandate effort + economic destruction and theft (tax money to corps as "aid") was never a reasonable, logical or scientific response. It was an authoritarian and corrupt response of extreme Effectiveness and cynicism.
But here comes some guy to say "people have always quarantined in pandemics" or some other disingenuous claim that ignores the reality of what happened: healthy people denied basic human and constitutional rights.