Item — HN Reader

I've been working on healthcare.gov for the last few months alongside a bunch of other Google, Facebook, and Y Combinator alums.

I'll always remember what Mikey told us in December, after the site was back up, could handle a non-trivial amount of traffic, and people who wanted health insurance could finally get it:

"1 in 1000 uninsured people die each year. It's not an exaggeration to say that due to the work we're doing here, 5,000-10,000 people will live to see the end of 2014. You should be proud of what you've done, but we should also all be grateful to have this opportunity."

We're all grateful to be here, but there's a hell of a lot more work to be done.

If any of you out there are an amazing software engineer or SRE, and want to help make our government work better, please shoot me an email: brandon@hcgov.us!

17 comments

> "1 in 1000 uninsured people die each year. It's not an exaggeration to say that due to the work we're doing here, 5,000-10,000 people will live to see the end of 2014."

This probably a significant exaggeration. It is based on a 2009 study[1] which examined correlation, not causation. It did not control for many factors that may be relevant (e.g. smoking). The study expressed this in much more careful words: "Lack of health insurance is associated with as many as 44,789 deaths..." (This number was then divided into 45M uninsured in 2009 to get 1 in 1000). Politifact did not rate this claim due to lack of information[2], but they previously rated "Half-true" a number half as big[3]. The latter essay cites work that did control for relevant indicators and found: "the risk of subsequent mortality is no different for uninsured respondents than for those covered by employer-sponsored group insurance."

I'm not sure where to place blame:

- On the authors of the correlation study, who should never have studied this question without looking at extensive control variables or without more specifically studying causation?

- On Alan Grayson and similar folks, who are smart enough to understand the difference but are happy to assert causation?

- On Abbott, who implies causation, pointedly rejecting caveats ("it's not an exaggeration") in order to motivate developers?

I don't want to blame brandonb, particularly. I very much support his recruiting effort. In fact, I would say that the government probably has a disproportionate number of people who can resist unwarranted self-justifications. But I don't think a statistic like this should be left unchallenged on HN.

---

[1] http://www.pnhp.org/excessdeaths/health-insurance-and-mortal...

[2] http://www.politifact.com/truth-o-meter/article/2013/sep/06/...

[3] http://www.politifact.com/truth-o-meter/statements/2009/aug/...

> It did not control for many factors that may be relevant (e.g. smoking).

From the abstract: "After additional adjustment for race/ethnicity, income, education, self- and physician-rated health status, body mass index, leisure exercise, smoking, and regular alcohol use"

> On the authors of the correlation study, who should never have studied this question without looking at extensive control variables or without more specifically studying causation?

How do you suggest studying causation in this setting? A randomized controlled trial where we deprive people of health insurance? Even that will likely not yield a true causal estimate, because randomization only helps for pre-randomization differences in the population, and behavior change from lacking health insurance will occur post randomization. The authors do extensively discuss their control variables, and important to remember is the fact that most papers only control for variables which ended up doing something, a subset of all variables that were tried. The NHANES data the study was pulled from includes a staggering number of covariates.

---

While not an ironclad study, I found the paper itself vastly more compelling than the politifact analysis of it, which boils down to "Well, observational studies might be wrong because reasons".

A randomized controlled trial where we deprive people of health insurance?

Yes.

Before instituting Obamacare/Romneycare/$POLICY, we should have run a pilot program based on random assignment with clear predefined success metrics. But that's politically dangerous - after all, what if the experiment shows that $POLICY doesn't work?

We did that, by accident, in Oregon (google Oregon Health Experiment). There were no statistically significant results beyond the placebo effect [1]. Strangely, none of our fact based politicians have proposed scrapping the medicaid expansion based on that.

[1] People with insurance perceived themselves to be healthier before actually consuming any medical care and became less depressed. But no statistically significant difference was observed in any of the objective metrics chosen before the study started.

Why would you blame the study authors for doing a correlation study? I know "CORRELATION IS NOT CAUSATION!!!" is the go-to takedown on the internet, but correlation studies actually can have significant value - otherwise peer-reviewed journals wouldn't publish them.

I'm also skeptical of Politifact's competence to adjudicate public health scholarship.

I agree that correlation studies can be valuable, but I see many correlation studies as (intentionally or not) exploiting a propensity (arguably a bug) in human reasoning that conflates it with causation. In this situation, we have thorough documentation of multiple people clearly making the error.

I'm actually not sure the study authors should be blamed.

But: given how politicized this question is, they could have reasonably anticipated the misuse of their results, and thus could have written their results in such a way as to avoid this. Or they could have publicly corrected non-experts who cited them for causation. Or: they could have controlled for factors that would make mortality and insurance-status independent. This last option is difficult & requires complex judgement calls (see [1] for a reasonable attempt), but even if you feel the other two aren't required of academics, this last one very much may be.

(Separately, I agree that Politifact should not be trusted automatically, but these seem like reasonable analyses, and I did not quickly find anything better.)

[1] http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2739025/

"But I don't think a statistic like this should be left unchallenged on HN."

Yeah, thanks for fighting for the cause! #sarcasm

Is HN supposed to be a place where we try to find a flaw on every statement and make sure it doesn't go unnoticed?

What you pointed out doesn't even diminish an ounce of what Brandon and his teammates are doing.

>It did not control for many factors that may be relevant (e.g. smoking).

Unless I'm misreading the polifact article you linked to in [2], it says that the 2009 study did control for smoking, and that they did a better job of controlling for such factors than previous studies.

> Still, their work stands out from previous efforts because it used more recent survey data and presented a more apples-to-apples analysis between the uninsured and insured populations. For example, it compared deaths rates for uninsured smokers with insured smokers, as well as other factors such as drinking, obesity, income and education.

Aside from the rhetoric, the political opinions and people firing at you now over the 5-10K thing, I do have an actual question of substance.

My assumption when things melted down so drastically, was that the key problem was integration with the various vendors. Yes we all could look at the Html itself and make assumptions about poor practices but given that you are on the inside - what was the biggest issue?

The biggest general problem is that people thought of shipping the site the same way they thought of shipping an aircraft carrier -- you write the code, hand it over, and you're done. It wasn't treated as a running service. So, for example, when the site went down, there wasn't a group of people responsible for bringing it back up.

That's what Mikey and the other Site Reliability Engineers fixed. They set up a war room with an engineer from each and every subcontractor, and the war room had three rules:

Rule 1: "The war room and the meetings are for solving problems. There are plenty of other venues where people devote their creative energies to shifting blame."

Rule 2: "The ones who should be doing the talking are the people who know the most about an issue, not the ones with the highest rank. If anyone finds themselves sitting passively while managers and executives talk over them with less accurate information, we have gone off the rails, and I would like to know about it."

Rule 3: "We need to stay focused on the most urgent issues, like things that will hurt us in the next 24-48 hours."

Once you have that process working, it's the same as optimizing software: you find the current bottleneck, fix it, find the next, etc. The Time article mentions two -- the lack of DB caching and the bad ID generator. There were dozens of things like that. And still are!

> The biggest general problem is that people thought of shipping the site the same way they thought of shipping an aircraft carrier -- you write the code, hand it over, and you're done.

So, the same problem every vendor encounters when doing a software project. I cannot tell you how many "lay people" can't simply grok this.

What is the biggest issue right now architecturally?

My god I wish my company would instigate Rule 2 !

Hospitals don't let uninsured people die and insuring people doesn't magically save their lives.

> Hospitals don't let uninsured people die and insuring people doesn't magically save their lives.

Not sure where you're getting this. A quick Google Scholar or PubMed search shows a consensus that mortality rate is significantly higher for uninsured than for insured. [1, 2, 3]

[1] e.g. http://jpubhealth.oxfordjournals.org/content/32/2/236.short -- On multivariate analysis, uninsured compared with insured patients had an increased mortality risk (odds ratio: 1.60, 95% CI: 1.45–1.76). The excess mortality in uninsured children in the US was 37.8%, or 16 787, of the 38 649 deaths over the 18 period of the study. Children who were hospitalized without insurance have significantly increased all-cause in-hospital mortality as compared with children who present with insurance.

[2] e.g. http://journals.lww.com/jtrauma/Abstract/2012/11000/Undiagno... -- Undiagnosed preexisting comorbidities play a crucial role in determining outcomes following trauma. Diagnosis of medical comorbidities may be a marker of access to health care and may be associated with treatment, which may explain the gap in mortality rates between insured and uninsured trauma patients.

[3] e.g. https://www.sciencedirect.com/science/article/pii/S000296101... -- A total of 1,203,243 patients were analyzed, with a mortality rate of 3.7%. The death rate was significantly higher in penetrating trauma patients versus blunt trauma patients (7.9% vs 3.0%; P < .001), and higher in the uninsured (5.3% vs 3.2%; P < .001). On multivariate analysis, uninsured patients had an increased odds of death than insured patients, in both penetrating and blunt trauma patients.

Hospitals do "let" uninsured people die because they are not obligated to provide the full extent of their care capabilities to people who cannot afford it. Come in with a hole in your abdomen, and for sure they'll patch you up even if you don't have insurance. Come in riddled with cancer? Don't expect the same as they would give somebody who could pay themselves or had insurance.

This said, it is still a shoddy use of statistics. "1 in 1000 uninsured people die each year" by itself tells us pretty much nothing. What is the rate of death for insured people?

Wikipedia tells me that 8.39 in 1000 people die in America every year, so if uninsured people are only dying at a rate of 1 in 1000 every year, it seems to me that either it is beneficial to be uninsured, or uninsured people are not representative of the population (perhaps because many of them are young and healthy?).

I suspect that what is going on is this person actually meant to say something along the lines of "1 in 1000 people die every year in ways that could have been prevented if they had insurance" A subtle but important difference. The actual mortality rate of uninsured people is most likely much higher than 1 in 1000, but the deaths of uninsured people in motorcycle accidents would not be counted in that "1 in 1000" figure.

Either way, it is shoddy.

Hospitals don't turn away uninsured at the ER.

However, due to lack of regular care, by the time you show up at the ER it might be too late.

> Hospitals don't let uninsured people die

They don't do as much to stop them from dying. (Other than public hospitals -- of which there are a limited number with limited capacity -- they are only obligated to stabilize them in the ER and then, if they have further medically-necessary care but are stable, they can discharge them and/or transfer them to a public hospital if one is available, whereas those that have resources, insurance or otherwise, to pay would be admitted.)

> insuring people doesn't magically save their lives.

Strictly true -- it doesn't do so magically. It does so, instead, but the relatively mundane mechanism of providing them the ability to pay for care other than emergency stabilization, which reduces the probability of them having such care denied or delayed.

An estimated 26,000 people die each year due to lack of insurance coverage. That's part of the reason the Affordable Care Act was passed. For more information, see this Reuters article from 2012:

http://www.reuters.com/article/2012/06/20/us-usa-healthcare-...

They don't let them die right there because of the Emergency Medical Treatment and Active Labor Act (EMTALA). But they patch them up and send them home as soon as they're stable. That absolutely leads to significantly worse outcomes which others have referenced.

http://en.wikipedia.org/wiki/EMTALA

People have to be inside hospitals for hospitals to help them. If you don't have insurance you're less likely to go in the first place.

You're talking about EMTALA, which is an act that prohibits turning away uninsured patients away who are in need of emergent care. Once the patient is stabilized and either discharged or admitted, they are subject to payments.

The patient with metastatic cancer who needs chemoradiation on an outpatient basis must pay for his/her care.

> Hospitals don't let uninsured people die

Irrelevant, even if true. Uninsured people often don't go to the hospital until it's too late, because they know the expense will ruin them and they're hoping it will get better on its own.

But if you know you can't afford it, maybe you'd avoid going to the hospital when you really should.

Update -- wow, a lot of you out there want to help with healthcare.gov, or the government more generally! That's awesome.

I'm trying to reply to each person, but it may take a couple of days. :)

If, for whatever reason, you haven't heard back from me by Monday, please email again!

(The discussion on the 1 in 1000 fact has also been interesting, and I've personally learned a few new facts about the original study and what is controlled for. I'd rather have rigorous analysis of impact for everything, including and especially "good causes," than to give us a free pass.)

Hi Brandon,

I emailed you on Thursday, haven't received a reply back. Is the Monday expect-a-reply-back time frame still good, or are you just buried under all of the replies? :-)

I don't want to spam you with inquiries if you're busy, just wanted to check in on this. Thanks.

(I'm working with Brandon)

To clarify, you don't need to be able to commit long term-- if you can take leave from what you're otherwise doing for a few months, that's enough.

If you want to work on something with real impact and help change the culture of government technology, email Brandon above! We're actively looking for a few good people.

So, out of genuine curiosity, why are you doing it? To help the country? It's exciting? For the money? Or some other reason completely?

What about your co-workers. What's everyone motivation to help fix this site?

man, i wish i had come across this offer a few months ago.

The least of my worries is government enforced purchasing of over-priced, broken healthcare.

How about sending to prison those that paid nearly 1BILLION for this site as well as the bankers and defense industry gluttons who stole trillions from us in the last ten years?

If I want to make my government modern - I'd put in place a system of accountability.

Thanks for the insights. Apparently Accenture has been contracted to help out (or take over) the HealthCare.gov website:

http://online.wsj.com/news/articles/SB1000142405270230381970...

I'm wondering - how is this perceived within the current team? Are you relieved that you will be getting reinforcements, worried that another big contractor could contribute to the complexity, or even disturbed that they could get the credit for fixing the site after all the hard work you put in?

I used to work at Accenture, so I'm curious as to how this will turn out.

> people who wanted health insurance could finally get it

Maybe I don't understand Obamacare well enough but I don't understand this statement (which I hear all the time on the commercials). Who couldn't get insurance before this website? I know the health care law did add guarantees around pre-existing conditions. So other than that, who couldn't call an insurance company directly (or use esurance or other portal) and sign up for insurance? This website is just a marketplace hooking you up with normal private insurance companies correct?

You're not allowed to "other than that" preexisting conditions. I'm one and the ACA means I got to quit my shitty telesales job and take a no-insurance-provided-at-a-small-company first software development job.

There are millions like me who can take risks for the first time, whether that's working for a small company or starting one. And I don't have to worry about the next 40K hospital bill burying me.

    Who couldn't get insurance before this website?

Poor people.

You can only get the Federal subsidies through the insurance exchanges which was supposed to primarily be online, but had small phone support staff too. The phone support staff was completely overwhelmed due to the website problems, which left thousands with no method to get insurance.

What if you'd like to help out but aren't an anointed™ one?

You certainly don't have to be anointed™ or have worked at any particular companies.

I'd say the two most urgent roles we're looking for now are: * Site reliability engineers * Software engineers, including backend and frontend

If you do something else, like design, product management, or data science, feel free to send me an email anyway — I'm certainly happy to check around and see what's needed!

just like with any other government contract - being anointed is the deciding factor.

what do y'all mean by anointed? I've been a gov contractor for many years. It doesn't take all that much to get into this space.

Basically, if you're a citizen and don't lie on applications and resumes and in interviews then there's really not any significant barrier to entry.

Haven't worked at Google, Facebook, whatever, but would love to help.

Thanks for doing this. I still think completely replacing it from October to ~December, and extending the deadlines, might have been a reasonable choice, and re-architecting to move all PII related stuff away from the "shopping" stage and only into the final request stage, but I guess it's better now. The "replace everything, end all the contracts, etc." option might not have been politically viable, and showing people the pre-subsidy prices with subsidy adjustments later in the process probably would have been a bad political move, too.

(I'm not in a federal marketplace state, though, and I ended up having my insurance canceled under ObamaCare and got a 364 day short term $750k policy instead. Still no idea what I'm doing in 2015.)

I'm curious how you're breaking down the work and if you only take full time devs or are bringing in hourly contractors for different pieces. Do you think you'd have better luck finding good devs if you broke it down like that?

So far, it's been on a rotation -- folks go on leave from their jobs for 1-3 months, move out to DC/Baltimore area, and work as hard as possible while they're here. :)

It's such a high-intensity project that I think the we'll want to keep it to full-time for the time being. Otherwise, it's hard to even keep up, let alone make progress.

(As a technical point, though, even though everybody is working on this for well over 40 hrs/week, we're technically hourly subcontractors. That's just how the government works.)

Besides the obvious issue that correlation does not imply causation what does the statement "1 in 1,000 uninsured people die each year" even mean given that the overall annual mortality rate in the US is 799.5 per 100,000[1].

Is this in addition to the background rate? If true that would be quite astounding. Or is it just rounding?

[1] http://www.cdc.gov/nchs/fastats/deaths.htm

"And, as the motley team of software engineers looked on in admiration, a single tear drop departed from Mikey's eye. As the drop hit the marble floor, it suddenly exploded into a majestic bald eagle, who screetched dutifully and flew out of the White House."

>"1 in 1000 uninsured people die each year. It's not an exaggeration to say that due to the work we're doing here, 5,000-10,000 people will live to see the end of 2014. You should be proud of what you've done, but we should also all be grateful to have this opportunity."

Using that logic, how many have died due to the botched rollout of the website?

Why isn't it open source like it was before?

The portion that was open-sourced was actually only the static portion of the site (the 'Learn' side). That was taken down before I got here, so I'm not sure on the complete backstory, but I think it confused a lot of people since they assumed it was the source for the whole site.

So you're saying that with the 6 million cancelled policies, the government has killed approximately 6000 people with this new mandate?

Does the extrapolated result include the number of people who are losing their insurance because of this program?