back

by brandonb·11y ago·view on hn ↗
Congrats to the team! Ben was one of the most brilliant students in my undergrad class and it's awesome to see him employ his talents in such a big way.

I worked on healthcare.gov last year and it's hard to overstate the potential impact of a tool like DCOS. At one point, we had 2000+ VMs, most manually configured, with no monitoring, and completely different configs between dev, test, and prod (not intentionally). Straightforward operations like migrating half of the database servers from one VLAN to the other took months, small mistakes like changing a database password could result in hours-long outages, and simply getting the data that Mesosphere displays automatically would sometimes take weeks and other times simply be impossible.

Of course, clean devops hygiene would have eliminated much of the pain in the first place, but not every organization has the expertise to do things right. In fact, most don't, and the solution for most organizations is good tooling that automates as much of the system as possible and provides good development discipline for the rest.

1 comments
Having 2000+ VMs without automation or monitoring in place is pretty unusual. Especially for a site like healthcare.gov which must have had all sorts of HIPAA and data security requirements.

But it's honestly not hard to hook all those machines up to a system like Chef/Puppet/Saltstack/Ansible and start automating common tasks within a few days.

Migrating databases between networks and rotating passwords would generally be outside of the scope of a tool like Mesosphere. Again, this is something that can be easily handled by existing automation tools. With most databases though, there's nothing straightforward about migrating data to nodes on different vlans or password rotations. If it's a one-time task, I recommend hiring a database consulting firm to do the migration or rotation.

I think that have good defaults and enforcing best-practices is a good idea. But I think that a lot of this can be achieved with existing tools. IMO, it makes more sense for organizations to automate and orchestrate Mesos deployments via existing/mature DevOps tools like Chef/Puppet/Ansible/Saltstack. Would also be exciting to see deployments working via NixOps/NixOS.