Item — HN Reader

"There is no political will to do so."

That was true a couple of years ago, but healthcare.gov was a wakeup call, and everybody from the president on down takes reforming government IT seriously.

I was part of the healthcare.gov "tech surge" and I can tell you firsthand that the government is willing to change. In the last year, the healthcare.gov UI was streamlined from 76 to 16 screens for the average user. Large parts of the site now run on AWS and more is moving. The most problematic individual component, the login system, was rewritten from scratch with an API-compatible, drop-in replacement which now serves 100% of traffic. And this has been done within the same agency responsible for the original site. It is not a story of "smart hackers" parachuting in and wresting control away from government bureaucrats, but rather partnering with the agencies and creating reform from within.

USDS itself, although started by the White House, now has bipartisan support and a real budget to grow. Although you are right that there are powerful entrenched interests in the beltway, the political will to change is there, and now the challenge is to recruit people like Andrew to seize the opportunity.

4 comments

> I was part of the healthcare.gov "tech surge" and I can tell you firsthand that the government is willing to change.

So, you worked for a short while on an extremely high-profile project with the personal daily attention of the president. Yes, that does tend to smooth things over - but don't mistake that for a fundamental change in government priorities. There are 2.6 million federal employees, you don't re-jig politics and incentive structures that has been entrenched over decades overnight with a single successful project.

Sure, showing that it can be done is always a good first step, and I'm not saying that anyone actively wants to fail - but only just barely. More often than not it feels that a failure you can blame on someone else is better than a mediocre success (good luck running an election campaign on having reduced the number of screens in a system or migrating it to AWS).

That seems rather cynical, doesn't it?

Why don't you ask the people who work at USDS if they're having trouble getting interest from the agencies? That would be a way to test your hypothesis.

Actually, I was responding to the GP stating a hypothesis.

But my question would be this: Are your political sponsors (a number of rungs up) invested in your work? Are they willing to take flak for your team (ie. take personal career risk) when (yes, when) things go wrong and provide the cover to help you turn things around, or are they more likely to throw the cocky Google-kids under the bus and declare "experiment failed" the second things turn a bit sour, and return to business as usual?

Now, answer the same question again, but 18 months from now, on the hypothetical that the next president isn't going to have this much on the agenda. What happens when heading the USDS doesn't mean briefing the president in the oval, but rather briefing a junior staffer to an undersecretary of the treasury?

Getting positive answers to all those is what real change looks like.

From my perspective as a former federal government employee, it seems more realistic than cynical. I agree that your second question would be a way to test this perception against reality.

Tech reform begins with the NSA

I hope you are right, but from my experience with government IT I feel its highly unlikely that a top down approach by a bunch of smart people is going to break through all the horseshit IT fiefdoms in the US government. Billions of dollars are spent every year on government IT stuff, and all those vendors aren't just going to sit quietly while you try to replace their garbage, overpriced BS software that they are charging an arm and a leg for.

Maybe if the President has his personal eye on a project something could get done. But the President isn't going to put his personal eye on the USPTO back-end for example. Or the FAA exemptions submission page. Or any of a thousand other points of contact between the public and government.

> Large parts of the site now run on AWS and more is moving

I'm curious what, if any, security obstacles you faced with this, especially considering HIPAA and all that.

I'm guessing this is on GovCloud.

That environment has its own security accreditation

http://aws.amazon.com/govcloud-us/.

Short answer: no real security issues. AWS itself runs everything from credit card processors to hospital software, they have great documentation on how to design your system so that you comply with the major certifications like HIPAA, PCI-DSS, and FedRAMP: http://aws.amazon.com/compliance/ http://aws.amazon.com/compliance/fedramp-faqs/

Our security audit was apparently the first one in years to pass with no "high findings" (high-priority security issues that must be fixed before the system is approved to go live).

The main barrier we ran into was that although AWS was already FedRAMP-certified, it had not gone through CMS's own internal security review process, which requires hundreds of pages of additional documentation and an audit of the code, team, and penetration testing of the live system. At the end of that process, you get an Authority to Operate, or ATO, and that's what every site in the government legally must have in order to launch.

You can get a sense of how much paperwork is involved here: http://www.cms.gov/Research-Statistics-Data-and-Systems/CMS-...

Navigating this ATO process was the single largest roadblock in developing the various parts of healthcare.gov 2.0. It meant that what we expected to take two months ended up taking more like eight.

The good news is that once we had gotten an ATO for AWS [1], any project within CMS--not just healthcare.gov--could start using AWS. [2] And within weeks of the ATO, we started hearing about other groups building their new projects on AWS, including Accenture, who's now the primary healthcare.gov contractor.

And that also means that if a startup wants to work with CMS, rather than the typical set of DC contractors, now they can use AWS too and have one fewer barrier keeping them out of the system.

[1] as a small technical point, we split the ATO into "infrastructure" and "application" portions so that the infrastructure (AWS) portion could be reused.

[2] If you're wondering why it matters that the government can use AWS, it's because most data centers that cater to the government are really terrible. In the initial data center used for healthcare.gov, the way you provisioned a new server was to send a word document to a sales representative, listing the unix packages you want installed, and then a few weeks later they would give you a virtual machine which may or may not have what you asked for. You can't be agile at all in that type of environment, and you certainly can't do DevOps well because you can't script anything.

I'm seconding everything said here.

I recently finished a project whereby it took over a year, from kickoff to handoff to deploy two blade servers.

Government IT is often extraordinarily hamstrung in the types of solutions it can deploy due to mandatory requirements to use certain security software, or employ specific policies. [1] While those policies come with a framework for adapting them to the needs of an environment, most don't understand them, and so you're often stuck with a computing environment that any DevOps engineer would find to be practically broken. There's an enormous amount of manual steps and needless shuffling between systems to accomplish the most rudimentary of tasks.

This enormous inefficiency is accepted as being inevitable. Because simple tasks take so long, and making them more efficient would require enormous effort (such getting exceptions to policies approved), the issue is often solved by adding more people. Contractors make their money through staffing, so if you can show that you can get more done with more people, then it can be a good sell. Costs, however, go up. This is where your overruns come from.

(Regarding ATOs, that sounds like the "Type" vs. "Site" accreditations I'm familiar with.)

A large part of the current GOP (and to a lesser extent Dem) platform is built on the idea that government cannot under any circumstances (except defense, because magic) do anything successfully. How can there be successful reform when so many political careers are now based on this assumption? Success has become failure, I am not hopeful.