back
15 comments
I'm one of the minimega authors. A friend pointed out that we made it on the HN front page, so I thought I'd come in and offer to answer any questions people may have.
Hello! This looks like a very slick project, but before going down this particular rabbit hole, I'm wondering if you can tell me if I'm set up for success or frustration: I'm running a Debian based hand-rolled 3 node kvm cluster with sheepdog/corosync for my vm image store/cluster management and a fully-connected p2p infiniband network for replication. Would it be possible to use minimega for management of this environment? What features rely on a regular openstack environment?
To address the last question first, there's no dependency on openstack at all. If you want to run VMs across the three nodes, it should be fine, and in fact I think the "sheepdog:windows7" disk format might just work in minimega. You should be able to use the Infiniband network for inter-VM communication provided you've got IP-over-IB set up.

Now, to address the unspoken question: should you use minimega for this environment? It depends what you use it for. If you're hosting your mail server on the VM, I'd probably go with something else--we haven't focused on long-running unique/essential VMs, although stability is quite good (I've run experiments for weeks at a time). If you're playing around with network topologies, or developing distributed software, or any other situation where you're running lots of VMs that you may want to tear down and re-launch frequently, it's a great option.

Does this use libvirt? It doesn't look like it. Why not? libvirt already provides almost all the functionality you need, you just need to layer a distribution mechanism for configuration across multiple hosts on top.
Nope, no libvirt. One big reason for that is, unless I misremember, libvirt needs a daemon of its own. That adds yet another step to deploying minimega on a cluster; as it is, you just copy the minimega binary to every node and run it. If you're netbooting a cluster, adding libvirt is going to make that initrd fatter and make bootup slower. If you're running on a "borrowed" cluster, you may have to convince the owner to install libvirt for you. Basically, we try to avoid external dependencies when possible, to the point where we've even eliminated the two C libs we used to link in.

Additional reasons:

* We already had a lot of experience driving QEMU/KVM manually (before minimega, we'd write shell and python scripts to launch our big virtual environments)

* If we want to do something QEMU supports but libvirt does not, we have to work around that. Since libvirt just wraps QEMU anyway, we avoid the middle-man.

* There's XML in libvirt, and we don't like XML :)

But according to a sibling comment, you support containers too (like libvirt)... you must already have an abstraction layer between you and QEMU. libvirt exposes pretty much every QEMU feature in existence, anyway, and allows passing through raw command line arguments to QEMU if you need it.

How does minimega run, if not with a daemon? You can certainly copy a libvirt binary + deps to a bunch of hosts and run it, nothing stopping that...

My guess is that minimega just grew out of your bespoke internal tools, and you never seriously tried to get rid of those tools and replace them with a more standard tool, like libvirt. It's disappointing. We're never going to get a viable alternative to Openstack if people keep writing things from scratch instead of reusing existing tools.

My guess is that minimega just grew out of your bespoke internal tools, and you never seriously tried to get rid of those tools and replace them with a more standard tool, like libvirt. It's disappointing

I must have missed the toolchain-police memo that you can only work on one toolchain. This stuff works for them, has a few cool features, and sounds interesting - I will be trying it out. Nothing wrong with growing out of your internal tools, and open sourcing them. By your reckoning, we'd all still be messing around with 386BSD or MINIX

There's also nothing wrong with wanting tools to reuse existing platforms rather than reinvent the wheel. minimega looks neat, but I don't want to sacrifice all the other libvirt-based tooling that's out there.

If minimega's virtualization abstraction for a single host was somehow superior to libvirt's, then I would be happy to see them forging ahead. But it's not.

minimega looks neat, but I don't want to sacrifice all the other libvirt-based tooling that's out there

Great! So libvirt is a good fit for you, awesome. Maybe other people have other needs or wants, and now they have an alternative that might suit them better. Isn't Open Source neat?

TBH, your argument is starting to sound very much like "libvirt is my tool of choice and I'm pissed that not every available person is working on the betterment of my preferred tool, instead choosing to work on something that is of no interest to me"

Monoculture is bad. Why create clang rather than use gcc? Why create Ubuntu when Debian exists? By all means create new things atop existing infrastructure, but don't forget that there is value in having options.
Seems like a rough equivalent to LXD, but using VM instances instead of containers, and more actual cluster functionality versus just remote commands.
It does containers too; we're especially proud of how it uses common configuration when possible. For instance:

  # configure the disk image used to boot KVM VMs
  mm$ vm config disk /home/floren/debian.qcow2
  # give it 4 gigs of RAM instead of the default 2
  mm$ vm config mem 4096
  # configure the virtual network, in this case we call it "foo"
  mm$ vm config net foo
  # launch three KVM VMs
  mm$ vm launch kvm my-vm[1-3]

  # now configure the container filesystem
  mm$ vm config filesystem /home/floren/containerfs/
  # launch 10 containers; they will also get 4GB of RAM and be connected to the "foo" network
  mm$ vm launch container my-container[1-10]

  # start all the VMs that we've launched so far
  mm$ vm start all
Ahh, that's great, and helps explain what makes it different.

With containers in addition to VM, it now looks more like a stripped down proxmox that's much more scriptable.

Looks like it would be a good underpinning to implement your own VPS service with "resource pools", where end users get some amount of cores, memory, disk, but can slice them however they want.

Yeah, there's a lot of fun things to do with it. Namespaces are especially cool. First you set up your minimega cluster (the nodes can auto-discover each other), then you run e.g. `namespace foo`. This creates a namespace containing all other nodes in the cluster (your machine is considered the defacto 'head' node but you could also add it to the namespace if desired). Now if you launch VMs, they'll actually get distributed among the nodes of the cluster based on available memory & current load. This makes it super easy to go from prototyping on your desktop to running the whole thing across a cluster--add a 'namespace' call at the top of your script and it'll Just Work.
Sounds cool.

I am currently managing hundreds of KVM VMs with libvirt, what would convince me to change to Minimega? Is this production ready?