Item — HN Reader

>> Imagine if you were running a business, and deployed CentOS 8 based on the 10 year lifespan promise. You're totally screwed now, and Red Hat knows it.

The hypothetical you posed is the actual situation, I am now learning, I have apparently forced on my team. We've ramped up labor 3x revenue preparing product launch in 90 - 180 days. We created an image containing centos 8 , Java , postgres and tomcat a year ago and that what is deployed to beta clients and what we've been testing.

What's ironic is that I sort of went out on a limb with my team by forcing us to go with Linux over Windows and the way I allayed concerns was to ask them to just "wait and see" in hopes that the performance differential would make it a moot point.

edit: after a little thought it seems that moving to RHEL might cost us the least amount of money and downtime. Still sucks and not what we need to be working on right now.

6 comments

What I don't get is this:

You build software, you ship it on an OS so it works, cool, this makes sense (I assume you need hardware and VM support, not just VM).

Why would you accept additional risk on the OS if you can easily reduce the risk, and ultimate cost, by going with an OS that has vendor support written into the actual contract? RHEL is 11-13 years total, Windows is... I'm guessing my grandkids will be using some form of Windows 10. CentOS is, and always was a community "best effort", with some serious delays occasionally (not often, but it happened).

A RHEL server license starts at $349, I have to assume that's at least an order of magnitude (or two or three) less than the cost of your software based on the technologies involved (sounds enterprise-solutiony). In other words a rounding error overall.

As a sysadmin in academia, this is not so straightforward. Since number of servers/VMs are in ballpark of over one hundred, RedHat license costs will be over 30,000$/year. This is not insignificant amount of money and not easy to get the money suddenly.

$30k is a significant amount, especially in academic environment, I don't deny it. But it's not much for > 100 servers. In my previous job, I had licenses which were about 15k€ for one server (thanks Oracle...) and the Windows Server licenses were also pretty expensive. $300-500 per server/year is not much more than a Photoshop license for one designer. And if we talk big business (1000's of servers and even more), you will get very special pricing anyway.

Not just price, but ability to function without constant access to the satellite portal. It's darn near impossible to upgrade a RHEL box from a work at home VPN environment.

> edit: after a little thought it seems that moving to RHEL might cost us the least amount of money and downtime. Still sucks and not what we need to be working on right now.

And that is exactly what IBM is counting on. Vendor lock in.

We are switching to Debian-based distros because, frankly, we don't trust IBM not to knife us in the back even on RHEL for more money. Of course we have the advantage of being able to take the time to convert.

This is not about IBM. It may not be a pleasant change for everyone, but it's a change that has strictly technical motives.

CentOS was acqui-hired because Red Hat's upstream for layered products (at the time mostly RDO/OpenStack and oVirt/RHEV) could not use Fedora because it was too far from RHEL a year of two after RHEL was released, could not use RHEL because upstream contributors would have to pay, and could not use CentOS because its releases had too large delays. The solution was to make CentOS releases happen timely by paying people to make them.

These days a RHEL downstream is not enough for the layered products. Some of them require the kind of bleeding edge feature that is backported every six months to the RHEL kernel, and corresponding userspace changes (BPF, virtualization, etc.) and cannot afford waiting for the CentOS release because development must be done in parallel with RHEL. So the solution was to move CentOS from happening after RHEL to before RHEL which is what CentOS Stream is.

I can confidently say that the reasons are technical because other CentOS downstream have the same needs (e.g. Facebook's) and they also want to send patches to CentOS for bugfixes or features themselves, instead of waiting for Red Hat to find out about that bug, or decide they need the same feature. Plus there's no reason for rebuilds to disappear. The SRPMs will still be released by Red Hat.

That in no way explains why they don't continue to have both. They indeed will have both for another year. There was no requirement the Stream product even use the CentOS name.

CentOS was a community project whose leadership and control was taken over (acqui-hired as you say) by Red Hat and then it's core use case for the majority of people actually using it was discontinued. That is a statement of facts that happened as I understand them, not some spin on my part.

If Red Hat had not stepped in, perhaps some of CentOS problems (trouble getting releases out on time) would have been worse, or perhaps some other companies would have stepped in. We don't know, but we do know that CentOS has not been changed to be something different than it was before. It used to be a free re-spin of RHEL. Going forward it's something entirely different.

Red Hat always had the option to stop funding/providing resources to CentOS and name their new thing something else, but they didn't, and now they've effectively co-opted CentOS to be something different than it was originally intended to be.

> That in no way explains why they don't continue to have both

Because they don't need it anymore. CentOS Linux or other rebuilds can still exist (just not using the name; I disagree with that but I can understand Red Hat doesn't want its name attached to something that might have large delays in security fixes in the future) if somebody funds it or volunteers to do it, just like CentOS still supports Xen but RHEL does not.

Also for what is worth there have been lots of engineering changes to RHEL in the past couple of years that make nightlies (and CentOS Stream) much more stable than they used to be, especially with respect to regressions. Running CentOS Stream is not going to be like Fedora Rawhide or Debian sid.

>>> it's a change that has strictly technical motives.

I understand the business reasons for doing so. I don't agree with anyone branding this as done for purely technical reasons. Having CentOS Stream may be needed for technical reasons. Stopping CentOS 8 is in no way a technical decision. They are unrelated in any technical sense.

If Red Hat just doesn't want to put resources towards CentOS as it traditionally existed anymore, that's their option, but they deserve any flak they get for taking over an open source project just to extinguish it, since CentOS is in no way really needs to be linked to their Stream product. They could just as easily called it RHEL Stream and said it's free, and it would be a less confusing and more direct funnel of people that want RHEL stability into RHEL subscriptions. Using the CentOS name is just a mind-share grab and screwing over an open source community. They control it so can do it, but that doesn't mean I'm not going to call them out for doing so.

> Because they don't need it anymore.

Isn’t that the crux of the problem? CentOS used to be about “us” (the users), not “them.”

> That in no way explains why they don't continue to have both.

They're a business ... are you honestly and in good-faith actually confused about this outcome?

Oh come now, this is Red Hat, a company based on Open Source software. We're not wrong for wanting to hold it to a higher standard. They have benefited from the OSS community just as they have contributed to it.

Red Hat didn't make CentOS they acquired it. This is very similar to a large corporation acquiring a smaller competitor, promising to continue supporting it, creating a new, different product using the brand, then dropping the original product.

I'm not confused at all about that. I was responding to the point that said "it's a change that has strictly technical motives." It's a business decision that's based on profit and positioning and name mind-share, so let's not hide that.

I'm wondering, what ties you so strongly to a single OS? Nay, to a single Linux distribution?

I have a hard time imagining, and the only scenarios that come to mind are those where things have gone awfully haywire.

I'm sure I'm missing something. Enlighten me?

Software doesn’t run on its own. They rely on bunch of other software that is typically provided by distributions. So unless there’s some magic piece of technology that can take CentOS RPMs and make it work flawlessly on any Linux distro, the entire software industry is suddenly going to have to spend significant amounts of time on repackaging work.

There is alien[1], but I'm not sure if it is flawless, because of slightly different ways of doing things across distros like /etc/default vs /etc/sysconfig for example.

Depending on how good is the source, its usefulness and dependencies, packaging it to Debian is pretty straight forward. The dh_* helpers[2] does the job automatically most of the time. There is also tools for helping with specific languages, like dh-make-golang, dh-python, etc...

[1] https://packages.debian.org/buster/alien

[2] https://manpages.debian.org/testing/debhelper/debhelper.7.en...

From the looks of it, alien is just a tool to convert between different package formats and is not a replacement for proper packaging work. Very much like how you can’t take a deb package from different releases of the same distro and expect things to work properly, you can’t just take RPMs from CentOS to a whole different distro and expect things to work.

It’s not only the differences in the packaging format that you have to take care of. There’s also version differences, path differences, dependency handling, and many other stuff to take care of. These are the kind of tasks which can’t be automated away and require non-trivial amount of work.

For organizations that maintain tens, hundreds, or thousands of CentOS packages that spans multiple teams, moving to other distros would be time-consuming and costly. It would certainly pay off if it was driven by technical reasons, but for organizations that are forced to switch by this announcement, this is just pure overhead.

>Java , postgres and tomcat

That's his stack...

>So unless there’s some magic piece of technology that can take CentOS RPMs and make it work flawlessly on any Linux distro,

It's not magic, just alien.

> >Java , postgres and tomcat

> That's his stack...

What’s your point and when did I ever mention Java?

> It's not magic, just alien.

Dumping the contents of an RPM archive on random distros is never going to work except for the most simple of cases.

>Dumping the contents of an RPM archive on random distros is never going to work except for the most simple of cases.

That says allot about you packing hygiene, and you don't dump it..that you can do with tar alone. You convert it to a deb.

Took me a minute to figure out where that came from, it was in the GGP comment.

But surely if you depend on those packages, you can maybe track (or just collect) their versions, then get the same versions in another package manager (possibly via pinning).

Though my experience with package manager is that dependency management is hell (rife with potential conflicts), so I do see the problem a bit better.

Still, it shouldn't take that long to fix? Like a few days of sprint to setup Nix or something like that?

I think it depends on the number of packages that you maintain. If it’s only a handful of packages, the entire process of repackaging, testing, and deployment might be doable in a matter of days. If it’s tens or hundreds of packages that depend on each other, I think that’s a whole different story. If those packages are maintained by different teams, it could take more than an year to complete the transition.

As for pinning packages, that’s only practical if you’re using Nix. As much as I prefer Nix over RPMs, not all of us have the pleasure of using Nix at work. It’s kind of a bummer because Nix packages are so much easier to work with and maintain compared to the competition.

An existing support contract?

If you run a for-profit operation, and downtime is costly, you (or your VP of eng) want a way to pay for immediate assistance from the OS's maker / distributor, when (not if) things go wrong.

> If you run a for-profit operation, and downtime is costly, you (or your VP of eng) want a way to pay for immediate assistance from the OS's maker / distributor, when (not if) things go wrong.

There is a difference between asking "how do I ensure there is one throat to choke when things go wrong?", and "how do I minimize the potential for things to go wrong?"

RHEL is a decent solution if you are trying to answer the former question (or both), but many budget conscious orgs focused on the latter and chose CentOS. Red Hat has now pulled the rug out from under them by trimming eight years off the EOL previously committed to with little notice.

EOL in this case means "no security updates" so even if your org was prepared, technically, to deal with a zero-day for example by rolling out an update in a timely manner without relying on paying a vendor for hand-holding, that option has now been eliminated.

Essentially, you now only get the stability and problem-minimization if you also pay the vendor for support. Otherwise you're stuck with a (relatively) unstable rolling release that will keep your internal teams very busy with a constant stream of minor issues, or potentially trying to roll-your-own updates or backports after the EOL for anything serious.

It's difficult to see this as anything other than a naked, money-grabbing betrayal of users.

I’d recommend using Oracle Linux 8 if you still need 1 to 1 compatibility. At my company I’m recommending Amazon Linux 2.

Not sure why I got downvoted. I’m trying to offer the poster a viable alternative to CentOS 8 with a minimum of changes needed. I get that a lot of people don’t like Oracle (and for good reasons) but really they are the best alternative (and free) EL8 disto now.

Consider downgrading to CentOS 7. It should work for 4 more years.

Can we take RedHat's word for it, though? CentOS 8 was supposed to be supported until 2029 before this bombshell announcement.

https://web.archive.org/web/20201101131417/https://wiki.cent...

The dates on that page were always dependent on Red Hat. In retrospect, obviously we should have made that clearer, and we will endeavor to do so in the future.

Running modern versions of software on CentOS 7 can be a giant pain due to the old version of glibc that’s baked in.

I was supporting CentOS 6 until about a couple weeks ago.

Then surely you understand what a pain it is.

An easy way around this problem is to run the software in a docker container with a newer glibc.

Docker "works" but has huge performance costs (disk IO) on CentOS7.

Java and Postgres will work.

I go and drink some Corretto :)

Out of curiosity why did you decide to deploy your application as a VM image? Is there a reason you didn't go with a Helm chart or other container native deployment?

You're asking why someone who uses CentOS hasn't gone with Helm charts?

That's like polar opposites of stability and bleeding edge.

I was curious what drove the decision to deploy that way. I was under the impression most new applications being developed today would choose a more modern deployment method. There’s a lot to maintain in a VM image like that, containers just seem easier to me. Helm chart or otherwise

The amount of maintainance with virtual machines images or container images are mostly the same nowadays.

I will argue that there is less maintenance when handling virtual machines images, because it uses less bandwidth and need less tooling around it comparing with container image based infrastructure.

But in general both are nothing more than the golden image concept.

>a Helm chart

As a sysadmin i have never heard of this and es tun 800 VMs.

It's a bundle of jinja2 templates for YAML files defining Kubernetes API resources. There's an accompanying application, Helm, which applies values to the templates and handles the interaction with the Kubernetes API.

Helm uses Golangs built-in template language which is definitely not jinja2 compatible.