▲ 318 points
back
63 comments
This seems to still be very much an AWS/Amazon project with no clear path to becoming its own independent thing. For example, you want vulnerability scanning on the OS? Well you can use an Amazon product for that, otherwise *shrug* [1]. So I guess as long as you plan to run Bottlerocket in AWS, you're fine.
I wish the Bottlerocket team would do 1 of 2 things. Either own up that this is just an AWS project, or start to solve for things like this and actually be a product that "runs in the cloud or in your datacenter" as they suggest on their website.
To be fair, I think "VM" on the OS for Flatcar / BottleRocket / CoreOS is not a requirement in the same way as on RHEL etc.
Do you want to know if you are patched? Are you running the latest version? If so, you have all the available patches.
I appreciate this can cause difficulties in some regulated domains because there's a "vm" box that needs to be ticked on the compliance worksheet.
Most of the reason we need VM on a "traditional" OS is to handle the fact that they have a very broad configuration space and their software composition can be - and often is - pretty arbitrary (incorporating stuff from a ton of sources / vendors and those versions can move independently).
But that's not how you're supposed to use a container OS.
If you do "extra work" to discover vulnerabilities in "latest", you are not really doing the job of a system owner (whose job is to apply patches from upstream in a timely fashion), you are doing the work of a security researcher.
If you’re interested in something not AWS check out Talos https://www.talos.dev/
It’s been around longer than Bottlerocket
It's not like something is stopping one from doing a vuln scan, right? Like, there's something that SSM's in (or uses the admin container) and then runs the scan. Couldn't you just do the same thing?
Genuine questions, I don't know if this is the case or not.
That's a good point. And it sounds like it would work to me as well. I don't know the answer either.
I guess my point is the project should be providing a clear path that doesn't involve AWS instead of just stopping short.
I just wrote a post on this. We have an eBPF + SBOM based security tool and it runs great due to hooking the kernel directly via Kube DaemonSet: https://edgebit.io/blog/base-os-vulnerabilities/
tl;dr: Amazon prioritizes patching really well, fixing real issues first
Why would one need vulnerability scanning on an immutable OS image? It can be done before deploying the image to the host machines.
Indeed, but it's just an example. Imagine it said "For example, you want Feature X on the OS? Well you can use an Amazon product for that, otherwise shrug" instead if it makes it easier.
Bottlerocket does not off FIPS mode like most other enterprise *nix distributions.
Just to save anybody the trouble who needs FIPS approved encryption for host OSes that you use at work for various compliance programs. This makes Bottlerocket a non-starter for us. A very active issue has been open for over 2 years on this and the dev teams don't seem to be convinced that this is important. We even communicated with the dev team through our dedicated AWS reps and they have no interest in adding this.
Here is the open 2+ year thread on this: https://github.com/bottlerocket-os/bottlerocket/issues/1667
Disclosure: I work for Amazon. I’m also the principal engineer for Bottlerocket.
FIPS support continues to be the top customer ask by a wide margin. Unfortunately the timing here is not kind for a new distro with no previous FIPS offering. New FIPS 140-2 certifications are no longer available, and new FIPS 140-3 certifications have to make it through a lengthy queue as the entire industry switches over.
If this were something the dev team could just power through, I assure you it would have happened by now. I apologize for giving the impression that it’s not important. It is, but that doesn’t help the timeline in this case.
In my experience with FIPS certification usually requires some changes that undermine security.
If you need it, then you need it, but having the certification is a mildly bad sign in my opinion.
I’m not the only one with this opinion. For instance, the Microsoft Windows team seems to agree:
https://techcommunity.microsoft.com/t5/microsoft-security-ba...
In the GitHub issue, there is a mention of replacing rustls and Go's crypto library with OpenSSL. That seems like a serious security downgrade.
Having been around a bunch of former-government people and bumping into FIPS myself a few times (like yubikeys) and reading about it, that's also been my sense, but it's nice to see a formal writeup with examples.
Thanks for the link.
"FIPS is the answer to the question, 'How can we force all cryptographic software to be approved by a government committee?'" [1]
Here's your reminder that even if FIPS itself isn't evil, it moves in evil circles with evil people. [2]
[1] https://twitter.com/matthew_d_green/status/41279364233232384...
[2] https://threadreaderapp.com/thread/1433451378391883782.html
This looks very interesting but as other commenters pointed out, the path to running it yourself seems to be obscured. Even the GitHub page is listed only on the main page.
I found the VMware instructions at https://github.com/bottlerocket-os/bottlerocket/blob/develop...
Very similar to CoreOS'[1] directive
And Flatcar Linux, derived from CoreOS https://www.flatcar.org/
Forgive me for the dumb question, what are the benefits of CoreOS over the alternatives ex : Alpine?
How does ostree compare with the A/B partition scheme used by Bottlerocket for updates?
Neither "Get Started" nor the FAQ tell me how to run this.
1. Sign up to AWS with your credit card.
2. Easily spin up as many instances as you need!
Great project, but it's been around since 2020: https://aws.amazon.com/about-aws/whats-new/2020/08/announcin...
Is anyone successfully using this outside of AWS?
Just one data point, but I wasn't able to get it to work on hetzner cloud
This seems really useful for stuff like AMD SEV-SNP, where we want a measurement of the (kernel + initrd + arguments) to guarantee certain behavior from the machine. Ideally, we could use this as the container hypervisor, and have it produce attestations that bind to the hashes of the running containers. This relies on not having container escapes; not sure what the state of the art on that is right now.
I'd much rather go with CoreOS. Is there anything they've open sourced widely used outside of AWS?
Website says that the OS does not have a shell. I cannot imagine a useful docker container without at least one shell script inside. So, if there is no shell, doesn't it mean that Bottlerocket is generally unusable except niche scenarios?
The docker containers can have shell scripts inside. The host machine doesn't have a shell. You can bring a docker container with a shell, and run it privileged, to have a shell on the host machine.
Containers which contain shell scripts also contain the shell itself. It is not typical for the host machine's shell binary to be made available to containers running on the host.
The idea with Bottlerocket is that the host itself does not have a direct shell nor a way to access it via SSH or any other method. Instead this responsibility is delegated to the admin container which is where you would actually connect to via SSM/SSH. From here if you needed a root shell you would use the `sheltie` utility to do so.
It's not uncommon to have docker containers without a shell for security reasons. For example distroless.
Here's a diagram on the subsystems. Bottlerocket has API that can be called from a shell in a container.
shells
|
containers
|
Bottlerocket
|
OS kernelYes, It's for worker nodes in orchestration services (k8s, ecs).
Verified boot?
It means there is a full trusted boot chain from the TPM to loading the immutable root filesystem: https://github.com/bottlerocket-os/bottlerocket/blob/develop...
Regular Linux distributions don't have this, even if Secure Boot is enabled: https://0pointer.net/blog/brave-new-trusted-boot-world.html
I still don't understand why people are so keen to shoot themselves in the foot and make everything sandboxed containers with virtual filesystems and networks.
Just use the damn OS and hardware directly. SSH into the host whenever you need to see how things are performing.
Kubernetes only works so long as you don't really care about resources being used well.
Reproducibility, automation, scaling.
Because Kubernetes is both abstraction and de facto standardized platform across infrastructure providers. All deployments to customers who are large institutions start with provisioning, OS alignment (there are huge differences between RHEL, SLES, Debian or Amazon, then customisations like hardenings are put on top), networking, storage, access rights. You don't want to deal with that from scratch each time. It costs both time and money. Direct dealing with hardware is long gone (Hyper-V and VMware), and now it's time to cut out the upper layers. Also, Kubernetes allows better resource utilisation and scaling.
> SSH into the host whenever you need to see how things are performing
These SSH people is how EC2 instances get hacked. Please stop and use telemetry.
There's also the SSM agent for people that feel the need to touch their instance.
how does this compare to nix?
Nix has no security guarantees, nor sandboxing primitives. So not really comparable.
I think Nix intention is more general purpose OS and tooling. Bottlerocket is about being just enough of an OS to run containers, and that's it.
You could use Nix to build (and manage/update) an OS similar to this.