back
72 comments
With due respect to the Cardiogram developers (hi guys) as a doctor I really can't see a huge amount of value in this - my patients who present to emergency departments with paroxysmal AF are all anticoagulanted or on rate control and there has only been one instance in the last 3 years and many thousand patients when someone has presented to emergency and we have had to run through the full spectrum of echo -> anticoagulante -> cardiovert.

To the founders: what do you see as being he end game here? Are you just looking for validation (of the concept in itself, not the app- I use it on my watch)? Is the US market so different that this is particularly useful and cost effective for detection? Do you see this as eventually displaying early warning in the instance of an early warning?

Thanks and I don't mean to denegrate your efforts, but I do see lots of Consumer Med tech as solving a problem that really isn't creating value (i.e. Proliferation of devices,wearables and algorithms that proclaim the ability to help with X but are really marginally helpful at best) and I'm wondering if I'm missing something about the actual medical benefit, or whether what I feel like is true- that they aren't after being a medical device at all but instead are chasing the consumer dollars by making medical claims

Circulation published a nice review this week on the rationale for atrial fibrillation screening: circ.ahajournals.org/content/135/19/1851.full?ijkey=StzSPk8eljGaP2G&keytype=ref

The main reason is that 10% of strokes are associated with undiagnosed atrial fibrillation. The patients who present to the emergency room are a pretty biased sample--for one, they're experiencing symptoms. To prevent strokes, we need to have a way to catch AF in asymptomatic people.

Part of the challenge here is that episodes of atrial fibrillation can be infrequent--in CRYSTAL-AF, for example, it took 84 days from randomization to first episode--and existing monitoring devices like Holters or Zio patches are only worn 24 hours to 2 weeks. The great thing about Apple Watch and other consumer wearables is that they're worn for months or years. That means if we can prove the algorithm is accurate, we can get higher time-coverage than a traditional medical device, catch more AF early, and prevent those 10% of strokes described in the Circulation article.

Perhaps it would be more accurate to say "we might catch more AF early, and anticoagulating those people might reduce their risk of debilitating stroke more than we increase their risk of lethal gi bleed or head bleed"?

The issue is whether the af you find through this kind of screening is associated with the same risk of stroke as conventionally diagnosed af - if not, their risk reduction from anticoagulation might not justify the bleeding risk from anticoagulating them.

The linked article suggests it might be "unethical" to even do a trial of anticoagulation vs control in af detected by such screening. That seems like a dangerous position for them to take, particularly coming from a group that is largely funded by the drug companies who sell expensive anticoagulation medication.

I would encourage readers to jump to the "sources of funding" and "disclosures" section of the linked review article to see just how many of the authors receive money from the drug companies.

None of that is a criticism of the authors of the UCSF study, this kind of technology is certainly an area worth exploring. I would just be very wary of the push by drug companies and the doctors paid by them to 1) find asymptomatic conditions 2) call it 'disease' based on older studies of patients who were much sicker 3) 'treat' them with expensive medications for the rest of their lives without actually doing research to see if it would benefit or harm the patient.

Thanks for taking the time to reply, I can now better see how this could be of benefit
This is really cool. I have an ICM (Medtronic LINQ) for something I don't believe an iWatch could detect so this isn't an option for me, but I do know they are sometimes used for AF as well, and mine cost over $60,000 to implant (billed to insurance, though my part was still much more than an iWatch costs). I have to think doing the same with an iWatch massively increases the pool of people who are capable of affording continuous monitoring.
Hmm, this makes me think personal insurance will at some point in the not too distant future give hefty premium hikes to those who refuse to wear a 24-7 monitoring device; as vehicle insurance in the UK is doing with monitoring devices (cameras/accelerometer-type devices).
Hi Brandon-

I can't find good tables in the article or on your teams site. [edited]

I know you guys take your numbers seriously but I'd love to see anything allowed out pre-publication.

[edit]Thanks to poster below I see this is AUC. Thanks!

Actually this is a huge game changer from a neurology perspective:

1) for folks who show up in the hospital, with a stroke, often the cause is not clear. So they go home on aspirin but the latest studies seem to indicate that fully 30% of these folks with "cryptogenic stroke" end up being diagnosed with PAF after 3 months on a heart monitor.

2) this heart monitor is either a cumbersome external device they have to wear for 3 months, or the Medtronic LINQ implantable loop recorder, which is nice and under the skin, but costs a lot of money for a cardiologist to implant and monitor.

3) from a public health perspective, if the Apple Watch can automatically detect paroxysmal A-fib BEFORE a stroke (i.e. permanent paralysis, inability to speak, etc)...think of the massive societal benefits this could be...

As a Paramedic here in the US, I'd be happy to see more people alerted to unusual cardiac conditions via their Apple Watch early as possible rather than finding out weeks later after damage has been done. We don't convert afib in the field, but having an Apple Watch catch someone that's in SVT would be great. Reading down in the thread, I see that it's happened at least once.
Not the developer or founder, but a user of Apple Watch. I would like to see your questions answered as well. But as a hopeful consumer, I hope wearable devices / medtech startups direct more attention and funding to improving sensors and associated detection algorithms to such an extent that at some point in the future, it would actually be useful to real doctors and offer real medical benefits.
Wouldn't this help with pre-stroke detection? The article mentioned that 1 in 4 strokes are caused by AFib. I would imagine this would be helpful for in-patient Neuro-ICU and stroke hospitalists. On the patient side, shouldn't something like this - assuming the monitoring is in real-time - provide a preventative/detection benefits for patients with a high-risk for stroke?
This is exactly right—the benefit is to prevent strokes by detecting undiagnosed atrial fibrillation.
Even before that - putting in on an Apple Watch potentially enables screening from a public health perspective. I'd rather prevent the stroke rather than regretfully putting the patient on coumadin after they've been paralyzed...
>With due respect to the Cardiogram developers (hi guys) as a doctor I really can't see a huge amount of value in this - my patients who present to emergency departments with paroxysmal AF are all anticoagulanted or on rate control and there has only been one instance in the last 3 years and many thousand patients when someone has presented to emergency and we have had to run through the full spectrum of echo -> anticoagulante -> cardiovert.

Isn't this a version of survivorship bias? Who about to those who aren't that lucky to manage to overcome the attack and become patients...

AF is rarely fatal, although stroke is significantly disabling. Normally (well, I don't have any figure at hand and I'm not about to go and research it at the moment, but I'm using the term in the sense of >50% but <85%) of AF instances come with some sort of symptoms... feeling faint, pounding chest, shortness of breath... good questioning can be fairly diagnostic in the doctors clinic or emergency room. But yes, as the Cardiogram guys point out in their reply to me, and in the source article, a significant number of people have strokes secondary to AF.
(Cardiogram Co-Founder here)

Let me know if any of you have questions on the study, app, or deep learning algorithm. My colleague Avesh wrote a post with a little more technical detail here: https://blog.cardiogr.am/applying-artificial-intelligence-in...

How are you managing regulatory issues? The Apple watch is not a legal medical device, so my understanding is that it cannot make any diagnoses recommendations to the user.
I would assume any alerts they give would be something like "we have detected an anomaly in X metric we monitor, please consult with your doctor/health professional".
This is really exciting! Quick question for you.

Have you experimented at all with using the Apple Watch to measure blood pressure?

I have done some reading that suggests the optical sensor could measure blood pressure with some accuracy, but that Apple is hesitant to release it as a feature due to regulatory and accuracy concerns. It's my #1 wished for feature.

Is it possible you're talking about blood oxygen saturation? http://www.cultofmac.com/320322/apple-watch-sensors-are-capa...
When will we see this make it to production for all Cardiogram users?
Accuracy is a useless metric for something like this. If you have binary data filled with 97% zeros (ie most of the time it is not arrhythmia) you can use the sophisticated machine learning technique of:

  if(TRUE) return(0)
This will give you 97% accuracy.

EDIT:

I just read the headline earlier. Now after checking:

>"The study involved 6,158 participants recruited through the Cardiogram app on Apple Watch. Most of the participants in the UCSF Health eHeart study had normal EKG readings. However, 200 of them had been diagnosed with paroxysmal atrial fibrillation (an abnormal heartbeat). Engineers then trained a deep neural network to identify these abnormal heart rhythms from Apple Watch heart rate data."

So 1 - 200/6158 = 0.9675219. My method performs just as well as theirs if we round to the nearest percent. This is ridiculous.

From the commments on the article itself:

Cardiogram engineer here. 97% accuracy refers to a c-statistic (area under the ROC curve) of 0.9740. An example operating point would be 98% sensitivity with 90% specificity.

These important details are often lost in the news. You can some more details on our findings in our blog post:

https://blog.cardiogr.am/applying-artificial-intelligence-in...

I think the article is not precise with their wording, but the 97% is actually recall (i.e. detects 97% of the positives).
Then it's missed 100% of the arrythmatic cases. But I appreciate the sentiment of what you're trying to say.
We need to see the full, published study and its methods (particularly around recruitment and exclusion criteria) before we can judge it properly. Until then, the presented statistics about accuracy, sensitivity, and specificity potentially bear no relation to real world usage, if the cohort and data quality were tightly controlled, as you'd expect for an initial study involving the makers of the algorithm. A few other thoughts:

1. Even at 98% sensitivity and 90% specificity [0], which I don't think would hold up with real world usage in casual, healthy users, if AFib has a prevalence of roughly 2-3% [1] then by a quick back of the envelope calculation a positive test result is still 5× more likely to be a false positive than a true positive. With those odds, I don't think many cardiologists are going to answer the phone. You'd still need an EKG to diagnose AFib.

2. There is huge variance among people's real world use of wearable sensors, and also among the quality of the sensors. (Imagine people that wear the watch looser, sweat more, have different skin, move it around a lot, etc.) You'd likely need to do an open, third-party validation study of the accuracy of the sensors in the Apple Watch before you can expect doctors to use the data. My understanding is that the Apple Watch sensors are actually pretty good compared to other wearable sensors, but I don't know of any rigorous study of that compares them to an EKG.

3. Obviously, this is only for AFib. AFib is a sweet corner case in terms of extrapolating from heart rate to arrhythmia, because it's a rapid & irregular rhythm that probably contains some subpatterns in beats that are hard for humans to appreciate. As others—including Cardiogram themselves [2]—have pointed out previously, many serious arrhythmias are not possible to detect with only an optical heart rate sensor.

[0]: https://blog.cardiogr.am/applying-artificial-intelligence-in...

[1]: https://www.ncbi.nlm.nih.gov/pubmed/24966695

[2]: https://blog.cardiogr.am/what-do-normal-and-abnormal-heart-r...

Full journal publication is coming--as you likely know, the system doesn't always move as fast as we'd like.

> quick back of the envelope calculation a positive test result is still 5× more likely to be a false positive than a true positive.

For what it's worth, about 10% of people who come in to the cardiology clinic experiencing symptoms are diagnosed with an abnormal heart rhythm. So even a 20% positive predictive value would be an improvement over the status quo.

As mentioned below, you can use other risk factors (like CHA2DS2-Vasc, or even simply age) to raise the pre-test probability, and thereby control the false positive rate.

As a meta-point, I do think we let the perfect be the enemy of the good in medicine, and that potentially scares people away who could otherwise make positive contributions. For example, many of the most common screening methods in use today are simple, linear models with c-statistics below 0.8. You can build a far-from-perfect system, and still improve dramatically over how people receive healthcare today.

My overall message to machine learning practitioners sitting on the sidelines would be: please join our field. The status quo in medicine is much more primitive than we have been led to believe, and your skills can very literally save lives.

About 1) it would still be far better than many, many, medical tests.
> by a quick back of the envelope calculation a positive test result is still 5× more likely to be a false positive than a true positive. With those odds, I don't think many cardiologists are going to answer the phone. You'd still need an EKG to diagnose AFib.

This is a good point, and certainly nobody should go directly to a cardiologist based on these results. It seems that this would be a good system to recommend that people get an EKG done, though.

> probably contains some subpatterns in beats that are hard for humans to appreciate

Not really, no... As you said, AFib is one of a very small number of causes of irregularly irregular heart rates (and is by far the most common). AFib is pretty easy to spot, even just by feeling someone's pulse with your fingers.

For those who's going to mention Bayes' Theorem regarding medical tests, here's a link to save you a Google search

https://en.wikipedia.org/wiki/Bayes%27_theorem#Drug_testing

If I am reading this correctly, there were 6,158 patents with ~200 true positives, so approximately 3% of the population. 98.04% sensitivity (recall) and 90.2% specificity (%true negatives) leads to...

~4 false positives for each true positive.

That isn't bad, all things considered, but still a long way to go.

They aren't clear what they mean by "97% accuracy". Does that mean 97% of people with arrhythmia are correctly diagnosed, or 97% of people are correctly diagnosed or not-diagnosed? If it's the latter, it's not very helpful at all. The number of people in the general population with arrhythmia is significantly less than 3%, so if this Apple Watch test says you have arrhythmia it is far more likely to be a false positive than a true positive.
Here, 97% accuracy refers to a c-statistic (area under the ROC curve) of 0.9740. An example operating point would be 98% sensitivity with 90% specificity.

For reducing false positives, rather than starting with the general population, it'd be natural to start with a higher risk sub-group, e.g., people with a high CHA2DS2-Vasc score.

Earlier this week, Circulation published a review screening for atrial fibrillation: circ.ahajournals.org/content/135/19/1851.full?ijkey=StzSPk8eljGaP2G&keytype=ref

No, accuracy doesn't tell us anything about usefulness. If the test is cheap and it brings in people for a better test and thus saves lives it is useful. The only real danger is if people decide that since the watch says there is nothing wrong so they can ignore other symptoms.

It is something like chest pain: most of the time chest pain is not a symptom of a heart attack, but it is best indicator we have so you go to the emergency room when you have chest pain. Doctors there can evaluate your situation.

I've arrhythmia and I can see it on my Garmin watch by looking at the data, as well as .. my phone.

It doesn't help much though, because I don't know if its good or bad (well, actually I know but not because of the watch data). Doctors are still needed for this, and generally that includes a bunch of controlled tests and people listening to your heart while also gathering data (similarly to the watch albeit with a more precise apparatus)

I guess it can help to tell people they might wanna see a doctor if they haven't though.

Probably want to update the title, which still refers to the more general 'arrhythmia'. It sounds like Cardiogram's work is mostly focused on afib?
First validation study is on atrial fibrillation, although we've had users who have discovered other arrhythmias through the app. Here's on example of a person who discovered supraventricular tachycardia: https://blog.cardiogr.am/my-apple-watch-saved-my-life-a61256...
I'm seeing more and more people wearing Apple watches and more similar PR stories like this.

I saw one story that talked about a guy whose car flipped and he was unable to reach his phone but thanks to his watch he was able to call for help.

Despite all the caveats around this work, early detection is one of the reasons HR trackers are a great investment. You can't manage what you can't monitor
the one question I have is - how much better is the DNN vs. just simple rhythm analysis, i.e. periodicity, etc.
Always great to have percentages reported for a sample size <100.
? I don't get that logic... And neither do half of my friends. The other 50% actually don't get it, either. But I did only ask <100 people.
Am I the only one who thinks that 97% accuracy (1/30 chance of an error) is very bad in medical diagnosis?

At least I'd expect something like 99.9% accuracy (1/1000 chance of an error) when someone gives me my own heart diagnosis.

You'll find most most medical algorithms pretty disappointing. :)

For example, the algorithm in implantable cardioverter defibrillators generates unnecessary shocks in 1 in 6 patients. Its accuracy is getting worse over time: http://www.reuters.com/article/us-untimely-jolts-idUSTRE70O7...

This is really great news but I wish it was actually available. The Apple Watch right now is largely useless.

It captures all of these health metrics but then does absolutely nothing with it. It really is desperate for some actual killer health use cases.

Thanks! You can use the Cardiogram app today to understand your heart rate data: https://itunes.apple.com/us/app/cardiogram/id1000017994?ls=1...

We'll be incorporating these results into the app itself over time.

But as with anything in medicine... it's ready, aim, aim, aim, aim, aim... fire! :)