back

by brandonb·12y ago·view on hn ↗
I worked with Mikey on healthcare.gov and he's the real deal--wicked smart, but also humble, patient, respectful, grounded, and keenly insightful into how organizations work and how to fix broken ones.

The day healthcare.gov launched, six people total got through. Six. Nobody even knew when the site was down, except by checking CNN. Two months later, Mikey had rallied the 20-something contractors into punching through dozens of problems, and had recruited a team of (now-) 30-something engineers from Google, Facebook, YC startups, and other fine places to come in on rotations to keep the momentum going. As a result, 8 million people got health insurance.

If you haven't already seen it, I recommend this talk and Time article: https://www.youtube.com/watch?v=0albm_hhQzM&feature=youtu.be... http://blog-assets.newrelic.com/wp-content/uploads/80893.pdf

And those of you thinking about joining: do it please! You don't have to give up your life to the government—it's possible to help in short-term rotations—but this is the best shot we have in a long time to change the way our government works.

(If you're interested in joining, feel free to email me a resume and I'll make sure the right people see it: brandon.ballinger@gmail.com)

8 comments
If there's ever a story that deserves an amazing documentary (it literally is saving lives), it's the unfucking of healthcare.gov.

I've long wanted to find out more about the behind the scenes. Thanks for pointing to the videos.

edit

Off topic, I'm also struck by a statement Umesh makes during his talk in the video you linked to.

(paraphrasing): "there's this concept that people have different personas when they do different things, but I can assure you there is no difference in how they use sites and get information, no difference".

If anybody ever wanted to know why Google boggled up Google+ and fundamentally doesn't understand identity management, this statement gives me lots of insight into how that might have happened.

So tech companies screw up all the time also.

I say make it a documentary, but make it as layman as possible so that the rest of America can understand just how fucked up it was.

Use it as an opportunity to shame the government into action.

Don't overlook the fact that the system was actually built by private contractors. A small in-house team of developers (like they've got now) probably would have done a better job.
That is not sure. Overall management would not magically get better if the project would be done by small in-house team of developers. The same management that allowed contractor failure would allow in-house failure and the same management that hired incapable contractor would put together incapable in-house team.

Basically, the same bad decision makers would made the same bad decisions. You can blame the team for bad code, but you can not blame them for such a huge failure of the whole project.

  The same management that allowed contractor failure would 
  allow in-house failure and the same management that hired 
  incapable contractor would put together incapable in-house 
  team.
A lot of outsourcing contacts basically mean that if the contractors deliver late, the outsourcing company gets paid extra (more hours needed to finish it!) and if they deliver a buggy product, they get paid extra (more hours needed to fix it!). So scope creep, avoidable complexity and using bad tools are good for the outsourcing company.

For an in house team who do their own maintenance, the same things mean embarrassment, late nights, and possibly getting fired. So scope creep and avoidable complexity are bad for the in-house team.

Of course, if you don't have enough work to justify keeping the in house team around after the project is done, they might amount to the same thing.

According to accounts at the time, the original team on this project was working late nights long before the failed September deadline. Their attempts to inform government that they will not make it were cut with "failure is not an option". Are you saying that this situation would be impossible with in-house team? There were reports of contractor fighting government over scope, are you saying that in-house team would win that fight?

There was no one in government side with power to say "no" to features and with power to prioritize. Are you saying that such political power would magically appear in someone if they would use in-house team? There were reports of stressed governments people coming to contractor site in person all of sudden and forcing developers to show them not done yet hopefully cool features (nope not login, none of them cared about sign up and load problems). I am not making this up. Are you saying that the same geniuses would left in-house team alone?

There was no one officially responsible for integration of parts done by various companies and integration testing. Would the same management put aside time and resources for that work if the companies would be replaced by in-house teams? Well, maybe a little on this one, but I doubt they would put aside enough.

Plenty of outsourcing contracts have set price or deadlines and penalties for breaking them. Plenty of them are not paid on hourly fee basis. At least one of contracted companies has history of failures under their belt, so they are hardly innocent in the whole failure. That does not make government management more sound. It just makes them equally guilty at worst.

"For an in house team who do their own maintenance, the same things mean embarrassment, late nights, and possibly getting fired. So scope creep and avoidable complexity are bad for the in-house team."

There are few problems with this assertion:

* Scope creep is problem beyond the tech team. Scope creep happens to in-house teams of private companies as often as it happen to contractors. That one can be solved only someone with real political decision making power cooperates.

* In-house team does not get much embarrassment after failed deadline and firings usually does not happen after them. The thing is, if the management is competent, management knows the deadline will be failed in advance and have a plan for that. Deadlines are failed for various reasons overly optimistic estimates being the primary one.

* Most importantly, good management does not fire people after they failed important deadline due to laziness. Good management replaces lazy incapable people long before and does not assign mission critical tasks to unproven people. Management that ignores problems and then comes in all guns blazing after predictable failure is bound to build quite horrible in-house teams.

Managing contractors is somewhat different then managing in-house team, but both require sound management to succeed. You might be somewhat better at managing one vs another, but you are unlikely to be great in managing in-house team while being total failure in managing contractors.

My favourite quote on this subject comes from Tom DeMarco:

  Risk Management - Project Management for grown ups
^Not true at all

Not in the government. There is little to no pressure to get rid of underperforming employees because you can always just dig a little deeper in to tax payer pockets and get more people. And any time you happen to hit the jack pot and higher a good engineer they end up leaving after a year to two because it’s imposable to get anything done in the government. I have been working in the government as a software engineer as a contractor imbedded in a government agency for almost 3 years now. The stories of brain drain in the government due to the untenable working conditions are true. When I was first hired I was told that I got the job but was then strung along for 2 months before I got my start date. When I finally showed up on day one multiple people commented on how fast they were able to push me through the system... After starting I was told I could not have any internet accesses on my development machine and that I would have to use a laptop next to me that is connected to a government machine. That laptops internet connection was totally useless with every other page and virtually all downloads blocked. If I do manage to get a download to go through I have to burn it to a disk take it up 2 flights of stairs scan it for virus before transferring to my stand-alone dev machine. If I can't downloaded it I have to wait until lunch drive 25 mins home download the file, burn it, drive back, scan, copy back to machine. I fully understand that such security measures might be need for the most sensitive DoD projects but the stuff I work on is in no way sensitive. When I need to order something even small things the process can take months and months. We had to order some projector bulbs for a very specialized projector which was only produced by the company that made the projector. The procurement process took 10 months and we were not allowed to buy the bulbs from the manufacture we had to buy them from a 3rd party at a steep markup. I have stayed this long because I am being compansated well (a little better then I could do in the Valley after taking into account cost of living) but its not worth it. The system is broken and I am burned out.

Careful. HN likes to downvote anything that's not liberal. Real world be damned.
I read that as "no difference in how people use sites to get information", which is fundamentally different from posting information.

I don't want my work gmail and my personal gmail to have different interfaces. I do insist that the content be different.

From an interaction design point of view, people don't maintain different masks.

Yeah, I don't disagree, but I think it's easy if you're not careful (cough Google cough) to conflate that with people just have one identity. Which most people very much do not want to have online -- the simple example of which even my parents understand is your work persona vs. your leisure time persona.
> the unfucking of healthcare.gov

We have a working title of this documentary.

Do we have any evidence of lives saved by the ObamaCare website? Or conversely, people that died when it wasn't functioning correctly?
You want evidence that people who can get healthcare might live longer while people who can't get healthcare might die sooner?

I'm not sure anybody would fund a study that obvious.

I want evidence that people died because the website was unavailable, as the parent claimed.
People die. [1]

People with insurance are healthier. [2]

People who are ill have a chance to die from their illness. [3]

People who are ill can be denied health insurance from private companies [4]

The Affordable Care Act prohibits this practice. [5]

Plans not listed on the exchange are not required to follow the provisions of the ACA [6]

Therefore, there is a chance that someone was on an insurance plan that did not cover their illness or did not have insurance because of an illness who could have been successfully treated if the exchange was open even one day earlier than it was.

[1] https://en.wikipedia.org/wiki/Death

[2] http://www.nber.org/papers/w17190

[3] https://en.wikipedia.org/wiki/Death_by_natural_causes

[4] http://www.politifact.com/truth-o-meter/article/2009/aug/18/...

[5] http://www.gpo.gov/fdsys/granule/FR-2010-06-28/2010-15278/co...

[6] https://healthunsurance.bangordailynews.com/2013/11/08/indiv...

I'm the author of the parent. I didn't claim that.
There is none. People here are acting like healthcare didn't exist before Obamacare and like you can't just sign up with any company for the exact same prices.
I still want to know why we did not have this "All-Star American Tech Dream-Team" build the damn thing in the first place.

Talk about missed opportunity for the Administration to rally citizens around their landmark legislation. "Built By America's Best for America's Best" sort of thing. Instead, it was built by a no-bid process outsourced to a Canadian firm. The rest is history.

If your comment isn't simply rhetorical, there was a lot of coverage of the root cause: the federal procurement process is a mess, which is especially harmful for IT. A quick summary is Clay Johnson and Harper Reed's oped from last year: http://www.nytimes.com/2013/10/25/opinion/getting-to-the-bot...

For a longer read, you might enjoy a Clay's essay on fixing procurement that delves a bit deeper: http://dobtco.github.io/fixing-procurement-ebook/final/fixin...

It sort of fails to convey the scope of how enormous, byzantine, and Kafkaesque the whole system is, however. For a hint of that, you can try clicking through https://www.itdashboard.gov/ (remember, this didn't exist before the huge efforts of Open Government Initiative pushed in the Obama admin) - it's still completely opaque who runs, bids, owns, or works on any project.

There's some more detail about how the bidding process for Healthcare.gov went down: http://www.washingtonpost.com/blogs/wonkblog/wp/2013/10/16/m...

CGI Federal's bidding actually preceded the conception of Healthcare.gov! They were one of 16 companies certified for a $4B contract via a GWAC (that means if you weren't one of those contractors, you couldn't even be in the running). The way that GSA scheduling works pretty much guarantees that you won't get small effective teams building lean/effective tech.

CGI Federal of course was only part of the picture: http://www.npr.org/blogs/alltechconsidered/2013/10/25/240532...

There was some interesting tech analysis of how this actually played out in terms of the data flow: http://www.appdynamics.com/blog/apm/technical-deep-dive-what...

The Federal Information Technology Acquisition Reform Act (HR 1232) would help. It was introduced back in March 2013 and has slowly making it's way through, but basically doesn't have any traction and is IMO pretty endemic and a great example of the type of deep regulatory/legislative dysfunction at the Federal level.

https://www.govtrack.us/congress/bills/113/hr1232 https://beta.congress.gov/bill/113th-congress/house-bill/123...

Of course, all this information is readily available, so the indirect answer is that things are the way they are because more people don't care enough to fix things.

To be fair, three Canadians were part of the tech surge to fix healthcare.gov: http://business.financialpost.com/2014/06/21/the-president-n...

I don't think the nationality of the developers was a factor in healthcare.gov's disaster launch. The problems were related to systematic disorganization, and the heart of darkness was inside the government itself and its IT procurement system.

True - my comment was more along the lines of spending the project funds with a US based company instead of sending it outside. Like, have American companies build the American healthcare system. Especially for a no-bid contract...
It wasn't "no-bid", by any normal definition. It went through the standard federal technology procurement process.
Because things like Libya and IRS scandals pop up and the administration scrambles. We didn't elect Silicon Valley tech tycoons that know a few things very well. We elected politicians with a heck of a lot more distractions/concerns than you or I.
Of course, the truth is this system never needed to be built at all. There were already sites available to purchase insurance, and marketplaces like esurance. The government could have deployed a static link page to the insurance sites. It's ludicrous to claim 8m people have insurance because of this site, when they could have had it without the site.

What they could have focused on was a calculator for the subsidies and Medicaid qualifacation, with the subsidies being delivered as part of tax filing.

As it stands, we now have an overly complex process for buying an insurance like product, and very little actually targeted at our big problem forcing providers to compete on price for non emergency services.

It's just slightly more complicated than that. :)

The main thing healthcare.gov does, which esurance and others don't, is to tell the user which government programs they're eligible for and what their subsidy will be. The law makes that calculation is actually surprisingly complex, for example:

  * there are at least eight different state- and federal-level healthcare programs (advanced premium tax credit, medicare, medicaid, children's health program, children's rehabilitative service, Tricare, VHA, IHS)
  * which programs you're qualified for depend on your state
  * the subsidy is based on which plans are available in your specific area and their cost, your household size, the ages of each person in your household, whether each person smokes, and whether you want a dental plan
  * for calculating subsidies, different sources of income are treated differently. E.g., tribal income for Native Americans is subject to different tax treatment, earned income is treated differently than capital gains in some cases, etc.
  * insurance is usually bought for a household, and there are precedence rules among different state/federal programs, so if any member of a household qualifies for a non-APTC assistance program, it affects the subsidy received for the whole household
  * to get the tax credit, your immigration status needs to be verified with the Department of Homeland Security and your income needs to be verified with the IRS. This is the work done by the "Data Services Hub."
  * by law, open enrollment starts in November but taxes are due the following April, so you can't compute the subsidy as part of the tax filing.
So you can see how something that seems like simple e-commerce task--buying health insurance--would turn out to be much more complex than a static site. You could argue that the law should be simplified, and I agree, but that would require an act of Congress and the cooperation of all 50 states. :) So, realistically, the software has to be complex to capture the complexity of the law.

(You could also argue that this complicated eligibility calculation should be exposed as an API, which would unlock innovation in the private sector. We want to do that, but need more help. :) )

"You don't have to give up your life to the government—it's possible to help in short-term rotations—but this is the best shot we have in a long time to change the way our government works."

Can you connect the dots for me as far as why, if the government needs help, they can't pay market price for the help? Or is the drawback to working for the government something other than that?

If the government pays its employees at market prices for tech, it'd be another sign of the bloated, inefficient federal government and, uh, socialism or something.

If they pay a big federal contractor at twice that rate, who then pockets half and pays its people market rate, then that's private sector efficiency at work.

More generally - there is no market when there is only one purchaser. Right now, the market for "tech talent working in government" is separate from the market for "tech talent working in private-sector tech companies", and the former sector has only one customer, which can set salaries basically arbitrarily.

If you watch the YouTube video provided, it seems like the subtext of this appointment is that Obama wants to merge these markets, so that the federal government competes for tech talent out of the same pool as Silicon Valley, Seattle, Boston, NYC, etc. tech companies. There will probably be many subtle culture changes there - programmers may stop wearing suits in government contractors, market rates may adjust, etc. I can't predict exactly how things will change, but wouldn't count on the status quo remaining for long.

The government has two different kinds of workers. Actual employees they pay directly and contract employees they pay through a private company. Actual employees of the US government are generally paid a mediocre salary at best compared to what they could get in private industry. Government contract employees on the other hand are generally paid very very well. In excess of what a typical private sector non-government contract company would pay. It's common knowledge among government employees that you can double your salary by getting a job with a private government contracting company.

The essential problem of the US government is there is no accountability. Poor performing employees are never fired and poor performing contractors are never penalized. If a project is a massive failure there are never any consequences for the people or companies involved. Over time anyone with any sort of skill whatsoever tends to leave and what's left are the poor and mediocre employees.

I'd like to say creating the US digital service is a positive step but unless the basic problem of accountability is solved it too will be just as bad at IT as every existing organization in a few years time.

In the real world this problem of organizational calcification would be solved when the incompetent company goes out of business because they are over taken by competitors. In the government since there is no competition there is no natural accountability mechanism.

Part of the problem is that the contracting rules seemingly force contracts to be split among a bunch of contractors none of whom is fully responsible for the project, so when something goes wrong, it's a circular firing squad. My experience is that contractors do get fired, but the same people end up working for the contractors that are still there (and since they are often the only people who remember what the hell is going on, this is sometimes not a bad thing).
That's by design: it's built-in pass-the-blame with an extra helping of spreading the bacon around. Government contracting is little more than legal, officially sanctioned nepotism and graft. The whole point is to allow Congressmen and other connected government officials to enrich their friends and family. In turn, these officials get high-paying "jobs" at the same places they helped funnel lots of money to.

The people in these comments here on HN claiming that somehow that's "liberal government" at work don't know what they're talking about, at all, and I mean downvote-worthy, comically so. If we must label these antics with comparisons to American political ideologies, the government contracting system is closer to big-business, republican-style politics than anything else. Small-c-conservative and any-l-liberal are the last things I think of when I think of government contracting.

The law caps federal salaries at $155,500. Congress could change it, but given that they seem to have trouble passing a budget, that doesn't seem likely very soon. :)

That said, I think there are enough people that a) find $155k enough to meet their needs and b) believe in the mission that we'll be able to make a big impact.

(One thing that's important to remember—"the government" doesn't really act like a unified, coordinated entity. Different branches and agencies have different incentives and behave in different ways. It's often better to think of the government as a marketplace rather than an organization.)

$155K is a good to very good salary in Boston. DC proper isn't much more expensive. I'd take $155K in DC for two years.
155K is the cap for federal salaries, not the typical salary. The 155K number (which is not quite right) is for the highest ranking execs short of SES (CIO / CTO level for many organizations. http://www.opm.gov/policy-data-oversight/pay-leave/salaries-...
Plus, there isn't any stock/equity/etc. And less perks (for example, free food can easily be effectively raise your salary by a couple grand).
I realize. I doubt that group is getting pegged to the normal scale. =)
>'Or is the drawback to working for the government something other than that?'

I would say yes.

Getting paid is not the big challenge in Government work.

It's the lack of urgency, political infighting and general stigma.

The rotations described here and the leadership position / division described in the article both skirt those issues.

* Urgency: Healthcare.gov had it. It was a firestorm visible from a continent away. Most Government IT failures don't work that way, they're smoldering trash fires that generate a meeting to figure out who's going to fill a bucket with a follow-up meeting to decide who's actually going to carry it.

* Infighting: As described, the Digital Service 'will be focused on providing consultation' not 'a group that we parachute in to write code'. They're coming in from a position of authority - above the level of typical jockeying.

* Stigma: None. Finish your rotation and go back to Google. Finish your stint as a head of an elite Federal service and name your title.

Government often pays more than market price for most jobs. I think his issue was no one wanted to leave Google to go help the government full time, that is why they had the rotations, people were not giving up their interesting jobs to help fix the governments mess.
Working on a big project where people come in and out too often is a problem in its own right. The simple reality of code being written by too many people who are there only for short time tends to create special kind of mess. It also means that project will tend to repeat the same mistakes again and again.

This kind of integration projects tend to be discounted by some as boring and stupid and indeed there is a lot of boring and annoying work to do on them. The thing is, this kind of project also have non-trivial learning curve until you become effective and the more organizational mess there is to overcome the steeper it becomes.

There are more red flags then just salary cap on this one.

some rules specify you cannot charge the government more than you charged someone else. Not sure if that would apply here, may just be on large contracts for products and services.
Can I suggest that "service" might not be as sustainable as "SME" for breaking down the government / mega-corp lockstep.

I have not followed the USDS but the UK version has gone from nothing to world class in four years and is trying to fix the badly broken procurement process as much as setting good project guidelines - for example all IT purchases are supposed to go through two gateway frameworks - the theory is to pre-quality as many SMEs as possible - then a department can buy a service (saas or development) from that SME with a single purchase order / credit card.

It's not perfect, getting on the frameworks is crazy hard and there are still tenders like "run our back office services please" but there is at least light in the tunnel for fools like me.

I even spoke to an agent today who bemoaned these frameworks - having to work trough small companies was painful - so it is having an effect.

> The day healthcare.gov launched, six people total got through. Six. Nobody even knew when the site was down, except by checking CNN. Two months later, Mikey had rallied the 20-something contractors into punching through dozens of problems, and had recruited a team of (now-) 30-something engineers from Google, Facebook, YC startups, and other fine places to come in on rotations to keep the momentum going. As a result, 8 million people got health insurance.

Out of curiosity, how much of the site's problems at the start were due to high load and how much would have been there even if it had seen much lower traffic? In particular, how would it have fared with the original design and code if only a handful of states had refused to implement their own exchanges, instead of 27 states refusing?

Hard to know for sure, but most of the problems would have been exposed at even moderate amounts of load--there was no load testing or performance monitoring in place initially.
Java in production, especially with heavy "frameworks" use and "waterfall to subcontracted coders-for-paycheck" process (how else there could be a billion lines of code for a site) can't handle even a moderate load and runs out of memory on handling persistent connections (long sessions) and the fixes were mostly by adding tons of hardware (unlimited government money) and to redesign many components to share nothing (decoupling) and to work in asynchronous way with a short requests (service-oriented architecture + Erlang's process-based model) by Google, FB, etc. guys? That's the story?

btw, these are very legitimate questions. The original project was a epic failure which perfectly illustrated all the flaws of past-century practices (waterfall process with tons of specs created by idiots, outsourcing to the cheapest or favorite incompetent, blind faith in big names, etc.) and the fix, it seems, was to bring it close to the state-of-the-art tech which was evolved to use-at-home in [more-or-less] independent-thinking shops, like Google or FB.

From the talk:

> If you technology people can either build a working website that actually serves the needs for a billion dollars, or build one that doesn't work and totally fails for merely 100 million or 200 million or so. If you can do either of those things, then the government is coming out way ahead.

hahaha

Part of the problem is the hiring process. I'm very interested, but knowing where to start on something like this is difficult.

When I applied to the Presidential Innovation Fellowship a year and a half ago, and there was no communication from them for 3 months -- far longer than they said they'd respond on the site -- and no way for me to get in touch.

(Just out of curiosity, have you seen a lot of interest?)

at first it sounded like a team of problem solvers (fun), but ended up sounding like more techno bureaucracy (boring).

setting standards across federal agencies? single sign-on? may god have mercy on their souls.