▲ 383 points
back
225 comments
Free account, indefinitely, to whoever needs it to store this data … including the storage vendor.
Just email…
Parent is maintainer of rsync.net, in case that wasn't obvious. (There's no contact information in your profile.)
> indefinitely
Which, in practical terms, means "while I or my investors have money". These guys have 70 years worth of data, can you guarantee to keep it for another 70? The real answer to these kinds of problems is managing the data yourself
I highly suggest (if you haven't already done so) you contact Nine PBS, as they (Nine PBS) won't be reading Hacker News.
rsync is good stuff, I've been a happy customer of https://www.rsync.net/products/borg.html for years.
Out of general interest: whats your bus factor?
Epic.
fucking legend
It’s entirely possible that Iron Mountain will be happy to turnover the data but needs a court judgement to do so without incurring more legal exposure. The article says the system belongs to OSS. If this is something like a colo arrangement then probably that is the only reasonable position to take. They can’t just go crack open a customer’s server and pull data out of it for whoever asserts a claim on it.
Iron Mountain is typically very stringent on their retention/disposal policies.
Storytime, since I'm no longer under NDAs for the company, over a decade ago somewhere that I worked, a disgruntled employee who decided to quit, made a point to within a week of quitting, take a whole bunch of important physical documents they had (because, FBOW, our email and other data retention was so bad, it was safer to have them as prints) and threw them all in our fancy Iron Mountain 'disposal' bin.
Apparently due to the agreement with them, short of a court order, only specific employees/titles in the company were allowed to be the persons to retrieve them.
Keeping that situation in mind, it's clear that this dance for better or worse is part of their posturing as a retention center with very strict rules to ensure compliance while also avoiding snooping.
They have a judgment against OSS ordering the data to be returned, and affirming ownership.
It still sucks for all involved, and it's a shame there's not a better solution here. It's gotta be so frustrating for the folks at PBS to keep getting hit like this.
Ok, it sucks, losing data always sucks, but what ever happened to the 3-2-1 backup rule? even if your off site backup provider goes tits up, your on site backup is still there.
If you don't want on site, a second off site provider, 50TB on backblaze would be an extra ~$350 per month (prob less if they were to commit to capacity instead of using their PAYG service)
> but what ever happened to the 3-2-1 backup rule?
I wasn't 100% sure, but the data backs this up[0]; the "3-2-1 backup rule" is a relatively modern concept, at least by that name. I also remember things like "have one backup in one place, another in another place" since before this mnemonic, but I feel like it's unfair to judge a 70+ years archive on something that just recently actually became a widespread mantra.
- [0] Interestingly, Ngram (https://books.google.com/ngrams/graph?content=%5B3-2-1+backu...) says ~2010 while Trends (https://trends.google.com/explore?q=3-2-1%2520backup%2520rul...) says slow build up from ~2020, guess "everyday" language would sit somewhere around those two different truths.
It's worth noting that PBS affiliates are non-profits that have fairly constrained budgets, especially now that the Corporation for Public Broadcasting has been dissolved due to DOGE.
Non-profit IT has always seemed, to me at least, to be a land of compromises. There's rarely enough money, knowledge, or time to do things "the right way", so you have to do the best you can. If I could only choose one backup to have, it'd be remote on a provider like Iron Mountain. It would seem as if they are storing things on Iron Mountain through an intermediary company, who may or may not have actually paid the bill.
So, y'know, fraud's a possibility here.
I’m not sure this is the only copy of the data.
Sounds like they paid a company to digitise all their old tapes, and store the results. They possibly still have all the tapes (on site) and are just trying to avoid paying the costs to redigitise them.
But, it’s also quite likely they destroyed all original the tapes after digitisation, and this was the only copy.
From the article: It's not clear if the channel has backups of its data, but based on its determination, we suspect not. This is strange, though, especially for a major organization, as one can easily purchase a six-bay NAS on Amazon and populate it with the best HDDs. This also goes against the 3-2-1 backup rule, which says you should keep three copies of your files stored in two different mediums, with one of them running offsite. Hopefully Nine PBS will implement a better backup system once it regains access to its data.
The backups (and servers) seem to still exist, but KETC's vendor is estranged with its own hosting / siting provider, Iron Mountain, which is a going concern.
IM seem to be holding KETC's data hostage, which is Not Cool.
Not PBS but an affiliate broadcaster and the data is not lost as in destroyed, but legally encumbered.
Full headline: PBS broadcaster loses access to 50TB of data comprising 70 years of TV history after contracted cloud storage vendor goes defunct — public TV channel sues Iron Mountain data center, which hosts archival materials, to ensure preservation
The channel’s attorney also reached out to Iron Mountain to inform the company of the pending case in Colorado; Iron Mountain finally admitted to holding the data. Iron Mountain initially agreed to turn over Nine PBS’ data, but it eventually backtracked and claimed that OSS owned the data it stored. Because of this, Nine PBS now has no choice but to sue Iron Mountain, too, to ensure the data is not deleted.
>storing over 50TB of data
I thought this would be like the backups I've heard broadcasters have of every broadcast they've ever had (I've heard of people being able to request 20 year old news clips before).
But just 50TB of data? Duplicating that would have been cheap and trivial.
This might be the website: https://www.osstorage.com/
I wonder how they got the contract? Seems to be 4 people 'employed' at this company. Of which, one is retired, one is "Linkedin Member", one is a controller and the last one is a social media person: https://www.linkedin.com/in/isabelle-anderson-7aa250b5/
So, just the person to deposit checks and a social media person to get paid. Who exactly is left to work on storage infrastructure?
The news article is really not clear about whether this was, relative to the company "OSS":
a) OSS is a colocation customer with its own hardware colocating it inside an Iron Mountain datacenter. In which OSS owned the bare metal and paid iron mountain for rack space and power.
or
b) OSS is a dedicated hardware customer of Iron Mountain running a service on bare metal owned by Iron Mountain, and has gone defunct, leaving behind a bunch of servers/storage arrays that would in normal circumstances get wiped/reprovisioned.
From the point of view of a customer of OSS (PBS), that's two extremely different things.
If it's scenario A, I don't see how PBS has any claim against Iron Mountain. Your typical datacenter colo host for bare metal hardware owned by a customer has no involvement whatsoever in the condition or operation of the data, operating system, filesystems, RAID arrays, ZFS, etc of how the customer has set up their environment. Nor any ability to do anything with it. A colocation host that hasn't been paid for its rack space and power will typically have clauses in its colo contracts allowing for seizure and sale of abandoned hardware after a certain period of time.
It's funny to me that entire companies exist that have a fraction of storage that I personally have in a datacenter on flash let alone in total. 50 TiB is so little. It's like $7k annually on R2 and under a thousand bucks on S3 Glacier Deep Archive. I suppose it's just whoever knew the cousin of whoever at whatever position of authority but it's funny how just using deep glacier means he would have such insane margins over paying Iron Mountain.
There's an update about a hearing yesterday that has more details about the server setup and what the concerns on both sides were: https://current.org/2026/08/judge-sets-framework-for-nine-pb...
With the stuff about "corruption" and losing the data if the server is shut down, I wonder if part of the concern is about losing in-memory decryption keys.
This underscores the importance of the Internet Archive and people like Marion Stokes (https://en.wikipedia.org/wiki/Marion_Stokes).
They should've kept the archives in their intern's car, like the State of Ohio did
https://www.newson6.com/story/5e3687772f69d76f6209d656/tape-...
Loses "access to" the data. The data is still there, they're just waiting for a court order.
> This wouldn't have been an issue if the organization bought a $1,000 NAS and populated it with a few hard drives.
Okay hang on now.. that strategy comes with its own risks. Risks that I imagine PBS was trying to avoid by using a third party that specializes in archival data storage. This is a silly takeaway.
Edit: I was too quick to comment. I think they are suggesting that the NAS should have been used in addition to the third party storage, which is actually a great takeaway.
I get why having an offsite backup is handy. But for less than $10k you can easily get a 4 bay NAS with 96TB storage for a local backup.
This smacks of saving pennies on IT to spend dollars on lawyers.
The situation is something we all will have to deal with as time goes on - this is an archive. If I remember rightly Facebook used to archive by just putting the old hard disks in a room. At the volume of data we are heading towards most successful companies will be in the hard position of finding some way of keeping data for the lifetime of a human being - this is hard
Such data should never be held by a third party unless it is one of several duplicates.
I keep photos on Google and MS but they are short term storage only and never the only copy.
When I worked for Syncplicity, we offered an on-site version for "cloud" storage specifically for this reason. (Ok, not specifically for this reason, it was because customers demanded it and an on-site version of our product was a great differentiator in our niche.)
Calling out Nine PBS for not accounting for the fact that their vendor could effectively just decide to commit crimes to get out of their contract and their vendor’s vendor could just decide to compound the problem in the face of a court order is wild. Yes, we’ve all learned to anticipate everyone else on the planet being an absolute shitheel; no, Nine PBS did not do anything wrong by assuming the professionals they were paying to render a professional service to them would do so. We’re all paranoid engineers, but this is absolutely no goddamn way to run a society.
I don't trust cloud services; I bought a 12T USB hard drive for my backups.
Isn't 50TB just 6 years of videos, in even the best case (1GB/h)?
Where have I heard this story before?
The magic of public-private partnerships strikes again.
Like most of the time, this will probably be held for some kind of extortionist legal ransom. We just love handing our stuff over to bad-faith third-parties in the name of some kind of imagined freedom. Just can't get enough of it. Public infrastructure, national archives, city services, we just seem to be completely deer-in-the-headlights when they pull this stuff on the daily...
So here we go again. We blindly trust in private parties always magically acting in good faith when there's billions on the table incentivizing them not to.
We've been doing it like this for 40 years and it hasn't worked, but I'm sure our ideological commitments will bend reality eventually.
This is your money btw.
> since 2019
At that point, why wouldn't you pick a major cloud vendor? This feels like more of a governance failure.
Intermediaries. When it's not the DNS it's the anomalous status of intermediaries.
Drives are cheap, data is expensive.
I work in several labs where I have to give this advice, which some labs are smart enough to make an ersatz backup that we store somewhere (PI's house) as a cheap offsite. Even a partial recovery is great in the face of fire/flood/explosion, or the most common error: IT fucks it up and then says "oh well".
Hope someone has some copies and they can rebuild, what a nightmare.
All this over 50 TB of data. Could have created an in-house replicated NAS with like $2500 worth of hardware.
From the article:
>This wouldn't have been an issue if the organization bought a $1,000 NAS and populated it with a few hard drives.
Look, I know that a certain percentage of Tom's Hardware articles are just sales pitches for a NAS, but this is asinine. Iron Mountain deals largely in tape storage. 50TB of tapes fits in one of their small archive boxes. If they're using the latest tech (LTO-9), it fits on 3 tapes.
This is exactly how you are supposed to store archival media for decades, not on a NAS. A NAS is for storing the files you need to actively access.
even r/DataHoarder won't be able to help that one
Sounds like we need another documentary.
Title should be updated to reflect the fact that this is a single PBS station.
The affected entity is a single PBS station, which As Is the Present Custom fails to provide call letters or geographic location.
"Nine PBS" seems to be <https://www.ninepbs.org/>, A/K/A KETC, St. Louis, MO.
<https://en.wikipedia.org/wiki/KETC>
The upstream cited article which should be subbed in for this submission does clearly identify the station in the first line of its report: <https://current.org/2026/08/nine-pbs-sues-iron-mountain-over...>.
(I've submitted this to HN's mods via email. The original submission was to a very poor TomsHardware link.)
OT: The tendency of local broadcasters to fail to identify either on websites OR QUITE OFTEN ON BROADCASTS THEMSELVES even vaguely where they operate is ... a long-standing annoyance of mine. Sure, local audiences may know, but those passing through, or catching the signal at a distance, likely won't.
I've spent more than a few nights at higher elevations (campgrounds, highways) listening to clear channel stations reporting on weather and traffic "in the local area", sometimes for hours, without ever hearing what "the local area" is. That for signals travelling hundreds of km, and not infrequently multiple thousands of km.
(Don't get me started on entire countries where stations are identified simply by number, e.g., "Channel 4" in the UK.)