▲ 563 points
back
380 comments
I RFTA and the majority of the complaints are about call center metrics and the pressure to ration care. These are real concerns about misuse of metrics, but not AI. The AI empathy thing was a 2024 pilot that was discontinued.
FWIW my wife works for Kaiser and finds a lot of value in the the medical LLM tools available to her. She tells me being able to do live translation, summarize notes, and quickly get comprehensive answers save her time and help her give better care. Her older patients also frequently come in bringing AI-powered alerts from their apple watches that detected cardiac events.
It's annoying that we use broad terms to describe a set of technologies that in some ways can be problematic and in another ways are very beneficial. We gotta evaluate each of these as they come rather than talk about blanket bans.
Things like this are (sadly) common (and age-old) problems with automation and computerization. (For a vivid account of this phenomenon, check out the novel _Close to the Machine_, by Ellen Ullman.)
As executives and analysts increasingly use the "AI" craze to push automation and computerization (and layoffs) generally, even aside from AI proper, it should not be surprising that the individuals and groups opposing those moves also use the same labels.
The lack of precision in language here sucks. It sucks for the discourse and it also sucks when it comes to focusing anger and productive energy on the core problems (obfuscation of human responsibility, erosion of human agency, declining institutional flexibility, deprofessionalization, etc.). But it doesn't begin with the critics of AI.
Being close relative for several med and care workers we have discussed it a lot and consensus is that it really depends. For example relying on LLM summaries sounds great until it doesn't. It doesn't matter whether you misunderstand LLM summary or LLM "misunderstands" you – there are real risks involved, and you wouldn't want them to weigh on your conscience if they were to materialize.
Relying on LLM to summarize things for you has one more issue. To outsiders, this seems like a tedious process, but is actually very important part of the thought process. Wording your thoughts and writing these down helps people to discover new aspects of the problem. It's how people learn.
At the moment consensus is that it must not be banned, but also not mandated in any way - people must take responsibility, and they must be able to decide for themselves where and when the LLM use is justified and where it is not.
> AI-powered alerts from their apple watches that detected cardiac events
Surely these are “good old-fashioned AI” (statistical learning) and not LLM, though.
I just want to be clear that the “medical LLM” tools are the new ones, and the Apple Watch alerts aren’t.
Cardiac events from Apple Watches is not “AI” though
Just for clarification, is your wife a doctor or a nurse?
I understand the frustration, but it's justifiable for the public to be concerned about AI in general because the novelty of the technology means that boundaries between beneficial and problematic usage is not yet stable and well defined. For existing technologies consensus on that has usually been reached, although it changes over time. In many cases we may not yet have enough information to decide.
Nor is this a trivial decision. AI has the potential to change society and economic relations as profoundly as the industrial revolution, or the invention of the printing press.
These boundaries are also contested, as interests which benefit from a particular application are different from those whose interests are harmed. Society needs to identify which usages have a net benefit.
It also needs to define which usages cause "absolute" harms which is will consider unacceptable regardless of benefits to some parties. Such as, potentially, reductions in personal autonomy, increased leverage or dominance by government or private interests.
Not only that, but data and models which were collected/ built for one purpose can easily be adapted for others.
This is also basically now happening all at once in many domains.
In short, you can expect there to be tension over these boundaries for some time. It's not realistic to expect that others will agree with your personal perception of which applications are "obviously" unproblematic.
>> It's annoying that we use broad terms to describe a set of technologies that in some ways can be problematic and in another ways are very beneficial. We gotta evaluate each of these as they come rather than talk about blanket bans.
I totally get it. I think few years, if some company said they record and transcribe every meeting/interview they take, it would be concerning. Now, its somewhat a norm for people to use these AI meeting tools which record everything you say and then go back to recording and exactly what people said. I'd call it surveillance than AI
“There is always a point at which the Amodei ceases to manipulate the media gestalt. A point at which the capital misallocation may well escalate, but beyond which the Amodei has become symptomatic of the capital markets themselves. Amodei as we ordinarily understand it is innately media-related. The Hangzhou hackers differ from other AI engineers precisely in their degree of self-consciousness, in their awareness of the extent to which media divorce the act of giving Amodei fuckin money from the original sociopolitical intent.”
everyone is very thirsty for AI hate, so its not unexpected. Today its a mixed bag of corpo hate, anxiety about the future, inequality and traditional class warfare, combined with the typical technical ignorance.
I would expect companies to blend shit metrics with AI systems, if not at Kaiser then at other places. People lack imagination and using AI to monitor your workforce has to be one of the possibly worst ways to use it. Alternatively some dickhead will "lean startup" their way into measuring "performance" in such a way with the "help" of AI that they will do something even worse.
If you think using a machine to evaluate how well a human is showing empathy is a good idea, you probably shouldn't have any position of power.
That is surprising. My primary care provider had a different response. Basically he said something in line of
"You wouldn't believe how much of a relief it has been. In your last visit, you saw me typing everything you were saying, right? I don't have to. I can listen to you and take very specific notes as necessary as opposed to focusing on both typing and listening to you at the same time. It has bought my stress levels down to here." (Indicated by his hand lowering)
Workplace metrics and surveillance... What could possibly go wrong?
That being said, more than one (female at that) doctor has told me in confidence and based upon their observations during residencies, etc. that if I'm ever admitted, be very careful how I modulate my interactions with nurses. They're not all Florence Nightingale and Mother Theresa and there exist those who will @#$% you up on various pretexts or are just plain sloppy and negligent.
Despite all the moralising fluff, it's just a job, not some saintly vocation. Some safety oversight is needed, just as it is for any other work function. Still, can bet that anything 'Corporate' has mandated will be Goodharted up the wazoo.
This problem with workplace AI interfering with nurses ability to manage healthcare obligations for their clients is not confined to Kaiser. UHC has also introduced the AI surveillance tools and are using it to do similar things.
The nurses at our local rural hospital are tagged and tracked wherever they go on the hospital campus. Time spent in one spot is part of their review.
I wondered why they zip in and out of the rooms, when just a few years ago they would spend fifteen to twenty minutes in each room. The patient load hadn't grown. The number of nurses has gone up, not down.
So I'm blaming the stupid metric on their evaluations for a worse standard of care.
The article mentions uses of AI but doesn't really give any examples of harm from AI. It does give specific examples where it sounds like Kaiser is optimizing calls to minimize cost rather than improve quality of care. It's also pretty easy to expect that the examples given (treating longer calls as a problem and penalizing nurses for giving more than 3 pieces of advice) reduce the quality of care.
the more I read about how AI is being used, the more I'm coming to the conclusion that despite me personally find it quite useful in my work, and even my personal life to some extent, it is a net negative to society as a whole, and if the choice is between zero AI or the AI that is being deployed, we'd be better out without it
what significant improvements to society or humanity have come about as direct result of AI, that wouldn't have been achieved without it? faster protein folding is the only one I can think of and that more a matter of "faster" than "impossible without AI"
I think at some point there's going to be some version of the Butlerian Jihad
They want you to blame "AI" like Claude jumped up and decided to start hassling nurses, not individuals in management
Every time “relentless push for efficiency destroying X” comes up I should repost this:
https://sohl-dickstein.github.io/2022/11/06/strong-Goodhart....
This is a well understood phenomenon.
It sucks because AI reduces call center costs in many toil tasks very effectively, regardless of whether you want to help or dissuade customers - it's just well-suited to the task and it's going to accelerate. It's a win for capital and the technology implementers, even if it's a loser for the call center employees and callers.
Some of the tech is pretty scary. One big vendor's solution [0] can provide not just AI agents but also use AI to snoop on calls in progress, evaluating sentiment from both sides [1], verifying phrases are said - pretty dystopian in theory. From experience, these things tend to go downhill based on the attitude at the top - is the mission to slash costs or take care of customers? A 1000 decisions follow from this one, and like Jira, it can be a useful tool or a prison-like hell.
[0] https://www.cisco.com/c/dam/en/us/products/collateral/contac...
[1] https://www.cisco.com/c/en/us/products/contact-center/webex-...
Thinks that are not allowed in the EU thanks to the AI Act.
My wife has been visiting an eye doctor several times. The doctor always use ChatGPT to double check (I'm giving him the benefit of the doubt; another way to frame this is the doctor didn't know shit and just used AI for answers).
I'm sure he doesn't think AI is making his job worse.
There is an unfortunate cycle that has been playing out this year in USA. Insurance companies are arming up with AI to deny billing codes charged by providers. Providers are arming up with AI to listen in on provider-patient sessions to prove the billing codes are legit (And teach providers to use keywords in session).
The loser, as always, is the patient's quality of care.
AI is making lots of jobs and workplaces worse.
> where they were found to have done everything right on a call — except staying on the line for more than 15 minutes
Do these idiots not know how intelligent assholes work? If you make this a policy, everyone who has a gripe will eventually learn to delay a call to 14 minutes and then make an absurd ask.
I can guarantee this is because the nurses don’t like the extra oversight and surveillance.
motivated reasoning: study in advance of nursing union contract negotiations
The sad thing is that there are many issues, but the study didn't really address them. One would hope for more effective critics.
The AI application I'd like to see would be to identify the elective hyper-utilizers (so we could try to demotivate them by addressing what's really bothering them). That would improve quality of care for them and quality of life for providers, and create time for outreach to the under-utilizers who need preventative care.
I'd like my providers to use Abridge for their notes, but because Abridge doesn't allow patients to opt out of using the data for training I decline every time.
Does "Manna" by Marshall Brain still sound futuristic?
> "Nurses fear that having long calls can lead to bad performance reviews"
> A company spokesperson said, "Kaiser Permanente does not use Average Handle Time to assess agent performance"
So uh, average time wasn't raised as a concern, calls beyond a certain threshold was. I wish this semantic discrepancy was better highlighted in the article.
Maybe healthcare shouldn’t be primarily for profit?
Kaiser was the first insurance company in america. If they cant make it work then we are lost.
And Kaiser already sucks big-time, so... that's not good.
Kaiser has priorities, healthcare being #4 or 5
I'd like to think it's not meant to make life better for a regular nurse, but rather to weed out the abusive ones? At least that's how I see it from my flawed, post-communist country perspective.
https://www.youtube.com/watch?v=DpQamTrynBY
[edit] preemptively, if you're going to claim "but this isn't art so irrelevant" then I claim bull fucking shit. It's the same problem no matter how you slice it whether it be engineering, support, art, or medicine. Get real. Look inward. Touch grass. If you think AI is a good thing for your profession--whatever that profession might be--you're probably a delusional psychotic. It's OK, you'll thank me later.
Obligatory dystopia reminder. It doesn't have to be this way https://marshallbrain.com/manna1
Population highly incentivized to denigrate new technology denigrates new technology. Film at 11.
There's a reason that among Americans, telekinesis and creationism are more mainstream positions than sloptimism.
About 14% of Americans think AI is moving us towards a better world. About 17% are creationists. About 26% believe in Telekinesis.
Just think -- with the increased efficiency, they will have more time for Tik Tok dances.
Unpopular opinion: People want to be lazy and hate things that force them to work harder.
They will vocally rationalize it.
I did it. "I'm more productive work from home." But then I do dishes, take an hour break, paid.
Foucault says that when people are observing them, power is placed over them.
If you are a worker you should hate this.
If you are a customer or owner, you should like this.
But I certainly won't be automatically believing people under surveillance who make claims it makes their quality worse.
> Nurses are instructed to stick to a script on phone calls and give no more than two to three pieces of advice, Capulong and other nurses said, which means they may sometimes need to decide whether to withhold advice or face a performance evaluation hearing.
> Another nurse speaking on condition of anonymity said “AI did not understand our job and would grade us wrong all the time.”
It's always worth remembering Goodhart's law https://en.wikipedia.org/wiki/Goodhart%27s_law - "When a measure becomes a target, it ceases to be a good measure."
In theory AI could usher in the first time in history where one can escape from this trap - because qualitative judgments can be made at scale, from an unbiased and universal baseline. In this situation, for instance, rather than collapsing call transcripts and reports into metrics, it could evaluate whether red flags are encountered in the context of a call, and allow for qualitative guidance on improvement, across a comparative corpus of situations that are themselves chosen qualitatively.
But very few managers are empowered to take this kind of approach; they're evaluated by their ability to report quantitative metrics, and thus they must implement regimes of quantitative metrics. And leadership instructs them to use AI to build that regime more quickly.
If you want to see an "AI native" organization, it's one where leadership actively fights this tendency, and sees managers as product designers who make the end-user experience a beloved and empathy-driven one, as opposed to a gear that turns accountability into a single number on a screen.
Kaiser has some of the best healthcare outcomes in the country/world due to their protocols and how good they are in ensuring adherence to them.
It’s going to be very improbable that these statements are true.
I would love to learn the cost of the AI versus how many new workers they could afford to hire and get more calls done. But I assume the end goal will be full replacement of human workers once the AI has had enough time to learn the job.
What these complaints always boil down to is autonomy and control.
The more centralized an organization, the more it relies on metrics to understand and exert control over its employees and customers.
People started hating tech right around the time metrics became popular. I don't think it's a coincidence. AI just accelerates the trend.
The problem is the misidentification of AI as the issue. As long as we don't understand the real issue, we won't solve it. AI is just a tool. It's being used in a way that denies human agency.
Our cultural values need to shift away from safetism that demands centralization. And shift toward valuing human agency. That starts with talking about the core issue.
While I'm sympathetic to the frictions of newly introduced AI and the fact that AI in healthcare, especially calls, can seem very uncaring, between the lines the article reads a bit like the typical union complaining about modern tech that reshapes their work, given the multiple mentions of protests, nurses union, etc.
Given how healthcare is one of these sectors that seems to relentlessly resist efficiency increases and is the prime example of Baumol's cost disease, I think any developed country with a costly healthcare system needs to do these AI experiments. The current versions will be shit, but the only way out is through if you still want to provide affordable care.
I honestly have no doubt that AI going forward will be able to do a good job at triaging via calls and also being empathetic about it. But of course it needs careful experimentation.