Item — HN Reader

For those who think the author is in the wrong

1 - Do you think this story indicates a success or a failure of the IRB?

2 - Would you rather a better expedited review process AND more studies (remember the personal effort cost killed this study), neither, or do you see some 3rd option?

2 comments

1. It's a success of his IRB. Of the many requirements he described, both by his IRB and the auditors, he didn't make the effort to try to understand how any of them could be good for his subjects. Instead he focused on how they affected his study, and tried to find the bare minimum he had to do to get approved. If the result of it is he doesn't want to deal with IRBs anymore, it might be for the best.

2. I believe this study could have been done with much less personal effort. His hypothesis is the screening question leads to unwarranted diagnosis. He already had Dr. W. convinced that it's plausible, and willing to do full diagnosis to his patients. And he had other doctors that would only use the screening question. All that's required to refute the null hypothesis is compare the rate of "yes" vs "no" in the set of Dr. W's diagnoses, with the rate of the doctors that just use the screening question. If his hypothesis is true, the other doctors would get more positives. That's 1 bit of information he had to extract anonymously from the patients' existing records, which is a strict subset of the methodology he developed for the study.

If you don't test both kinds of diagnostics on the same patients, that doesn't sound the same at all?

As another example, if you have a ten-question survey, asking 1000 people one question is not the same as asking 100 people all ten questions. You don't learn nearly as much about the correlation between answers to different questions.

Assume the patients are assigned to the doctors in a way that's not affected by whether they have the condition or not (e.g. at random). If the screening question doesn't lead to over-diagnosis, then you'd expect the same ratio of positives/negatives from both sets.

You need more subjects to attain the same statistical power, but the data is much easier to obtain. 100 subjects already sounded like overkill anyway.

It doesn't strictly pertain to his hypothesis, but presumably you would like data on false positives and negatives as well as overall outcome percentages.

Granted. But to get that you need to jump many more hurdles (which are there for good reasons).

>>Do you think this story indicates a success or a failure of the IRB?

Failure of the IRB but mostly failure of the author to explore his options.

>>Would you rather a better expedited review process AND more studies (remember the personal effort cost killed this study), neither, or do you see some 3rd option?

As I've posted in this thread, this problem is solved. Private IRBs exist that are much more responsive.

What options do you think were inadequately explored to sign the consent forms? Finger paint?

It is worthwhile to consider that evidence of consent is mandatory, and signing in pencil is not considered a legally valid option. If you can't sign the standard form for safety reasons, there are all kinds of ways that have been traditionally used to document legal consent for people who can't sign documents for various reasons - for example, public notaries, signatures of witnesses or even simple video evidence of explaining whatever needed to explain and the patient consenting would be a possibility.

>>What options do you think were inadequately explored to sign the consent forms?

None.

Now, if you asked a question without assuming what you think I was saying, I'd tell you that Alexander's other options were to not use the IRB that he went to, and to explore the private marketplace of IRB organizations that provide much better service than the nightmare he dealt with - a very common practice in all fields.

> Alexander's other options were to not use the IRB that he went to

This is, frankly, ridiculous. The 50-70% overhead the university takes from every grant already pays for one IRB. If that one isn't working, the solution is not to tap in (almost inevitably tight) research funds to pay for a second for-profit one instead.

Given how hard it is to charge things like computers to grants, I'm actually amazed this is even allowed.

>> If that one isn't working, the solution is not to tap in (almost inevitably tight) research funds to pay for a second for-profit one instead.

I am curious: How much do you think private IRB engagement costs?

I looked into this once and it was a few thousand dollars.

"But Matt", you'll say, "Isn't that a drop in the bucket compared to an R01 (a few hundred thousand/year), let alone a big program project grant? Why not spend a bit more to start data collection sooner?"

This is a pretty tempting proposal. However, I think this analysis almost misses the point. Programs run by people who can land big grants usually involve someone who already knows how to navigate the IRB process and can get things approved relatively quickly.

However, small or pilot projects tend to get bogged down, since the proposers have much less experience with the process, as in the article--and my own experience). These is especially true for people like the author, who aren't "full-time" human subjects researchers. The article didn't mention the projects' budget, but I would bet that if it wasn't zero, it was close to it (especially after covering salaries, etc). Even $1000 is going to be prohibitively expensive for many of these projects.

More philosophically, I think overseeing research is part of a university's job (and, as I complained above), one that is already paid for via overhead. There's also something vaguely worrying about the incentives of a for-profit IRB.

>>I looked into this once and it was a few thousand dollars.

Our IRB cost is 10% of that, or less, depending on the number you are choosing for "few." Maybe if you used WIRB or another IRB that is specific for drug trials, yes, it can run that expensive. For run of the mill published papers, no, you should not be using services like that.

>>There's also something vaguely worrying about the incentives of a for-profit IRB.

This would imply any reviewer or journal actually checks IRB qualifications on submission, review, or referee. Devor et al in the JSCR/Ohio State case and all the reviewers coming out as a result of that and saying they don't check IRB shows that it is anything but. Again, this isn't true for drug trials, but for expedited review minimal risk studies, there is no reason to be using bureaucratic and slow moving IRB providers.

EDIT:

>>Even $1000 is going to be prohibitively expensive for many of these projects.

There are many IRB providers who do expedited review, minimal-risk, standard studies for less than half of this cost.

EDIT2:

>>I think overseeing research is part of a university's job (and, as I complained above), one that is already paid for via overhead.

Sure. I'm not going to argue against the idea that the government or the university should do their jobs that they were paid to do. But they regularly and demonstrably do not, in far more fields than IRB. Private markets exist to go around this bureaucracy. IRB is no different.

I'm not the parent but would like to know. How much does it cost?

And how does "I skipped our free IRB and paid this third-party IRB because they're laxer" really look to the department?

A few hundred dollars, generally. Depending on provider.

>>And how does "I skipped our free IRB and paid this third-party IRB because they're laxer" really look to the department?

In my experience the third-party IRB orgs are not more lax, they are just faster at turning things around. Alexander's article bemoans primarily the time it took to get all these decisions made, which private IRBs would be on top of. I got my rejections of consent forms within 24 hours and get my docs pre-qualified over the phone after submission and discussion with reps. Try that at your local university's IRB. Alexander also has complaints about the questions and methods, but they're mostly unfounded. The study title aspect is important, the signing in pencil is not really relevant (I get his complaint but most industries don't allow contracts signed in pencil), and the transfer of the study to a new investigator unannounced is a huge breach of procedure and he's lucky that all the IRB asked was for a submission of a new investigator. Usually it is much more arduous.

IRB simply controls the ethics and the informed consent documents and has the right to audit materials and process at any time to ensure you stay up on these things. Beyond that, it doesn't do much. When you submit your paper to a peer-reviewed journal (some who don't even need IRB approval, just an ethics statement, mind you), they don't call your IRB provider and reference check you.

Ohio State researchers published a fraudulent article about CrossFit years ago in the JSCR, which is considered a very prestigious exercise science journal.

http://retractionwatch.com/2017/06/02/journal-retracts-ohio-...

Devor stated they had IRB approval. In actuality, they did not. CrossFit sued NSCA (parent org of JSCR) and won lawyer's fees, and Ohio State settled with CrossFit for six figures for damages due to false publications.

IRB is a mere formality in today's landscape. You just need to get it out of the way. It's not like journals are cross-checking your IRB applications, they just want to see the approval that you put in there. And hell, if you're like Ohio State researchers (and probably many more), you can just lie about it.

Yeah, I'm surprised the funders (or someone in a business office, at least), don't baulk at paying for this.

@icelander, have you ever successfully charged these third-party IRB fees to a grant?

If one group of IRBs is prohibiting certain practices and another is allowing them, then that is obviously a major systematic flaw in IRBs as such. Either one IRB is letting shady practices through (and that's a big problem) or another class of IRBs is unreasonably restricting research, and that's a problem as well.

Shopping for a better IRB isn't a systematic solution, it's a workaround for a system of flawed IRBs.