Tell NYT, Atlantic, USA Today to keep Wayback Machine

Don't know if it helps your musings at all, but there's a good chance that if a high-profile crawler like archive.org disrespected their robots.txt, that archive.org would be faced with lawsuits (or some other form of pressure). This is not merely the most moral move; rather it is the only sensible move.

The only reason "others are rewarded with profit" in cases like these are because pinkie-promise-style obligations don't affect players too small or shadowy to bother litigating.

Correct. Example snippet from the nytimes.com robots.txt:

    User-agent: archive.org_bot
    Disallow: /

No, archive.org does NOT respect robots.txt. You need to reach out to them directly and ask your site not be included: https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...

It's because they want to restrict AI companies from stealing content, but they can't do it if internet archive proxies it all for them.

All of the LLMs would be massively less useful if it wasn't for scraping the latest news.

It's the same idiocy that DRM created.

Be a pirate, because a pirate is free...

And find a litigation pool so the Archive can ensure LLMs crawling it contribute back.

That actually sounds like a cool idea! If only they would allow at least this.

Yes, it's called an escrow

This seems like a nice compromise. The news orgs get to keep the initial flurry of page views while the free information/universal library role of the Internet is maintained. But still those magazines will want to control their back catalogues. They currently sell access to libraries/universities. And as many on HN suggest, some of those news orgs would like to change/update stories without a publicly available "revision history".

Not surprised. They're working from the wrong model for the wrong age with the wrong incentives. They're still acting like they live in a world where data and information is scarce; and they are the one true source of truth.

It's flipped right now. There's no single source of ground truth, but data and information are abundant. Yes, that abundance that includes false data and lies, but it is still abundance.

The work The New York Times and The Atlantic do at their best days, i.e. their investigative journalism team adds to this world, but they try to hide / cloister that work away even though the journalists themselves want to make it accessible.

In an ideal world, every child would learn how to read english via the NYT and The Atlantic, they'd grow up with these sources of record, learn from them, and watch the world through them. But the current model doesn't allow for that.

I think a patronage mixed with wikimedia-style foundation might be a better fit. Readers who love the institution and its mission are invited to pay as much as they want with scaling benefits (let's say you love the NYT so much that you want to give $10k/mo for their work, you should get commensurate access / get to ask questions). And these contributions flow into the endowment, which is invested and the outputs of that are distributed as a part of their operating budget.

I don't think classical journalism can survive an information abundant world without a patronage-based approach.

> can’t be pulled more than N times a day, in case it becomes relevant in the future

In case it "becomes relevant." Wouldn't that benefit you either way? It makes you wonder if they have a dashboard of unfortunate digital statistics on display somewhere and worship of these numbers have replaced the underlying spirit of journalism.

>about the paywall workarounds that are frequent on HN—I was genuinely shocked to learn they hadn’t heard about it.

Is the Internet Archive regularly used as a paywall workaround? Generally it's archive.is, which has no connection to the IA.

I would be glad if these “news” sites weren’t posted to HN at all. If the article is true and worth discussing, it will be reported by a more reputable organization (e.g. Reuters) or it’s a primary source that should be posted directly (sometimes the source is posted then a news article covering it is posted later, I don’t know why both aren’t merged).

Too often they’ve been caught selectively reporting details and quotes, or reporting facts from an unreliable source that turned out to be outright false. In the latter case they quietly retract the article, so most readers continue believing the lie (maybe that’s why they don’t want to be archived).

Even posting a small blog is better, while it can also be biased and untrustworthy, if it has original thought, supports an individual, and doesn’t have ads. Although the amount of obvious LLM blogs submitted here is another issue.

Chomsky one time was talking about, gosh, his eyes are too old to be reading this microformat thing with a magnifier at the library in order to research archived newspapers like The (New York) Times. (This was sometime in the 90’s.)

> What is the advantage to those organizations to have their work preserved?

It becomes a research resource. It also creates a high-friction interface for potential subscribers.

I wound up subscribing to Le Monde Diplo because of a HN comment referencing a paywalled article. I didn't want to sign up just for one article. So I bypassed using one of the circumvention sites (I think outline was popular then). The article was compelling enough that I signed up for the paper, and remain subscribed to this day.

Can't the archive publish the SSL signatures of all the requests or something?

You can cryptographically verify a timestamp though by piggybacking on bitcoin like opentimestamps do.

Many unrelated archives would be good enough

Would you have paid NYT to view the article if there weren't an archived copy? I doubt it.

> It's not like anyone is committing these rags to microfiche anymore.

I recommend you actually go and read those fiches. The press was not historically high quality. Mass media has had the same problems for decades.

What it used to have was genuine independent competition.

And by what standards have you determined that these outlets are circling the drain?

The work of independent journalists is more important than ever before.

> the clearest signal of their irrelevance

NYT had $2.82B in revenue in 2025.

I'm a paying NYT subscriber for years. NYT has a ton of ads, even for subscribers. They don't offer an ad free version despite it being totally viable at a few more bucks a month based on their finances. Their ads are super disruptive to reading and their privacy policy appears to indicate they buy and sell your data.

I dunno. That seems like a pretty big fuck you to a paying customer already when all they have to do is provide a sub for a few more bucks a month. But I guess I'm a child of the Internet.

How about we go back to the era of humanity where modern marketing didn't exist?

How much faster would consumer software be if adware was made illegal? How much faster would our devices be if we didn't have half the code base supporting malware?

Acting like an ad enabled internet was the only option is extremely foolish, especially when the ad enabled internet was fully chosen and pushed onto the public by very specific people (thanks Newt Gingrich!).

I'm fine with ads as long as they are integrated with the page. What I hate is the typical Google Adsense garbage where the same ad is plastered in 4 different places on the page, with a video ad playing in the corner, and if you're lucky, a popup ad as well.