Item — HN Reader

I recently got entangled with some "big data" and "machine learning" B.S. in the form of the U.S. health care system.

CMS, the federal agency that administers Medicare, introduced a hospital quality ratings system last year. It is supposed to combine a variety of objective metrics into an easy-to-understand grade for hospitals.

However, the techniques they used are really bad. For example, a programming error makes the model give different results depending on how the data are sorted. Some measures get negative weights, meaning a hospital should do worse to get a better rating.

I wrote more about the technical failures here: https://sites.google.com/site/bbayles/index/cms-star-ratings

Another criticism: http://jktgfoundation.org/data/An_Analysis_of_the_Medicare_H...

2 comments

This is very fascinating, so thank you for looking into this. I think you may have misunderstood the rating system. You said

>>>Let’s say I really have to produce a rating, though. What would I do? I would probably: - Find some experts and ask them to assign weights to my various measures on the basis of how much they contribute to quality

But isn't this is exactly what the latent variable model (really a PCA) is doing? The only difference is - rather than have experts pick 60 weights for each component, which would require 60 contentious decisions, the PCA does some form of dimensional reduction so the experts need only pick weights for 7 components, which "unravel" into 60 weights. This sounds reasonable to me - assuming of course, the PCA components each have meaningful interpretations, and measure the degree of "good".

I admire your charitable interpretation. I'd be fine with the rating system using automation to reduce the number of subjective decisions to be made - I thought the LVM approach was a good one when I first heard about it.

As the second article points out, even this is kind of crazy, and the implementors didn't seem to care at all about what the results were - for example, the imaging category is driven by a single measure related to abdominal CT scans.

>This sounds reasonable to me - assuming of course, the PCA components each have meaningful interpretations, and measure the degree of "good".

But they won't, unless it happens to project near one of your dimensions.

If the 7 components can only be understood as linear combinations of 60 factors, your experts still need to provide 60 scores.

The cynic in me is suggesting those errors are deliberate. Politics. Sorting data should not change the data. Why would a measure of a good thing negatively influence the rating? I know... Don't attribute to malice that which can be easily explained by incompetence, and all that. But when is government, politics has to be lurking.