▲ 425 points
back
107 comments
Excellent! ArchiveTeam have always been impressive this way. Some years ago, I was working at a video platform that had just announced it would be shutting down fairly soon. I forget how, but one way or another I got connected with someone at ArchiveTeam who expressed their interest in archiving it all before it was too late. Believing this to be a good idea, I gave them a couple of tips about where some of our device-sniffing server endpoints were likely to give them a little trouble, and temporarily "donated" a couple EC2 instances to them to put towards their archiving tasks.
Since the servers were mine, I could see what was happening, and I was very impressed. Within I want to say two minutes, the instances had been fully provisioned and were actively archiving videos as fast as was possible, fully saturating the connection, with each instance knowing to only grab videos the other instances had not already gotten. Basically they have always struck me as not only having a solid mission, but also being ultra-efficient in how they carry it out.
Title is imprecise, it's Archiveteam.org, not Archive.org. The Internet Archive is providing free hosting, but the archival work was done by Archiveteam members.
What exactly is archiveteam's contribution? I don't fully understand.
Edit: Like they kinda seem like an unnecessary middle-man between the archive and archivee, but maybe I'm missing something.
What ArchiveTeam mainly does is provide hand-made scripts to aggressively archive specific websites that are about to die, with a prioritization for things the community deems most endangered and most important. They provide a bot you can run to grab these scripts automatically and run them on your own hardware, to join the volunteer effort.
This is in contrast to the Wayback Machine's builtin crawler, which is just a broad spectrum internet crawler without any specific rules, prioritizations, or supplementary link lists.
For example, one ArchiveTeam project had the goal to save as many obscure Wikis as possible, using the MediaWiki export feature rather than just grabbing page contents directly. This came in handy for thousands of wikis that were affected by Miraheze's disk failure and happened to have backups created by this project. Thanks to the domain-specific technique, the backups were high-fidelity enough that many users could immediately restart their wiki on another provider as if nothing happened.
They also try to "graze the rate limit" when a website announces a shutdown date and there isn't enough time to capture everything. They actively monitor for error responses and adjust the archiving rate accordingly, to get as much as possible as fast as possible, hopefully without crashing the backend or inadvertently archiving a bunch of useless error messages.
> Like they kinda seem like an unnecessary middle-man between the archive and archivee
They are the middlemen that collects the data to be archived.
In this example the archivee (goo.gl/Alphabet) is simply shutting the service down and has no interest in archiving it. Archive.org is willing to host the data, but only if somebody brings it to them. Archiveteam writes and organises crawlers to collect the data and send it to Archive.org
Archive Team is carrying books in a bucket brigade out of the burning library. Archive.org is giving them a place to put the books they saved.
Related. Others?
Enlisting in the Fight Against Link Rot - https://news.ycombinator.com/item?id=44877021 - Aug 2025 (107 comments)
Google shifts goo.gl policy: Inactive links deactivated, active links preserved - https://news.ycombinator.com/item?id=44759918 - Aug 2025 (190 comments)
Google's shortened goo.gl links will stop working next month - https://news.ycombinator.com/item?id=44683481 - July 2025 (222 comments)
Google URL Shortener links will no longer be available - https://news.ycombinator.com/item?id=40998549 - July 2024 (49 comments)
Ask HN: Google is sunsetting goo.gl on 3/30. What will be your URL shortener? - https://news.ycombinator.com/item?id=19385433 - March 2019 (14 comments)
Tell HN: Goo.gl (Google link Shortener) is shutting down - https://news.ycombinator.com/item?id=16902752 - April 2018 (45 comments)
Google is shutting down its goo.gl URL shortening service - https://news.ycombinator.com/item?id=16722817 - March 2018 (56 comments)
Transitioning Google URL Shortener to Firebase Dynamic Links - https://news.ycombinator.com/item?id=16719272 - March 2018 (53 comments)
Can we build a blockchain/P2P-based web crawler that can create snapshots of the entire web with high integrity (peer verification)? The already-crawled pages would be exchanged through bulk transfer between peers. This would mean there is an "official" source of all web data. LLM people can use snapshots of this. This would hopefully reduce the amount of ill-behaved crawlers, so we will see less draconian anti-bot measures over time on websites, in turn making it easier to crawl. Does something like this exist? It would be so awesome. It would also allow people to run a search engine at home.
> This would mean there is an "official" source of all web data. LLM people can use snapshots of this
that already exists, its called CommonCrawl:
Why would I spend time and resources to feed a machine which wastes more resources to hallucinate fiction from data it ingested?
For digital preservation? We may discuss. For an LLM? Haha, no.
No, thank you.
Recent update from Google: https://blog.google/technology/developers/googl-link-shorten...
Yeah, I'll take that "update" like the extremely unreliable info from an extremely unreliable company that it is.
I have a question about this.
Per google, shortened links “won't work after August 25 and we recommend transitioning to another URL shortener if you haven’t already.”
Am I missing something, or doesn’t this basically obviate the entire gesture of keeping some links active? If your shortened link is embedded in a document somewhere and can’t be updated, google is about to break it, no?
I don't really understand this. Is it really that costly to keep the entire database if they're going to keep part of it?
This leaves me wondering what the point is? What could it possibly cost to keep redirecting existing shortlinks that they consider unused/low activity already anyway?
(In addition to the higher activity ones parent link says they'll now continue to redirect.)
I don't understand the page, it shows a list of data sets (I think?) up to 91 TiB in size
The list of short links and their target URLs can't be 91 TiB in size can it? Does anyone know how this works?
The data is saved as a WARC file, which contains the entire HTTP request and response (compressed, of course). So it's much bigger than just a short -> long URL mapping.
I did some ridiculous napkin math. A random URL I pulled from a Google search was 705 bytes. A googl link is 22 bytes but if you only store the ID, it'd be 6 bytes. Some URLs are going to be shorter, some longer, but just ballparking it all, that lands us in the neighborhood of hundreds of billions of URLs, up to trillions of URLs.
The 91 TiB includes not just the URL mappings but the actual content of all destination pages, which ArchiveTeam captures to ensure the links remain functional even if original destinations disappear.
3.75 billion URLs, according to this[1] the average URL is 76.97 characters would be ~268.8 GiB without the goo.gl id/metadata. So I also wonder whats up with that.
https://web.archive.org/web/20250125064617/http://www.superm...
They might be storing in WARC format, which records all the request and response headers and maybe even TLS certificates and things.
Is there anyone archiving all of reddit? Or twitter? I mean even if their terms have changed to not allow it.
> reddit> twitter
There used to be one such project (Pushshift), before the Reddit API change. You can download all the data and see all the info on the-eye, another datahoarder/preservationist group:
Not that I know of, and you haven't even been able to archive tweets on the Wayback machine for YEARS.
Academictorrents has monthly dumps of all reddit submissions and comments even after the API restrictions.
ArchiveTeam was doing that, but their stuff no longer works due to changes at Reddit. The wiki page about it links to some other groups doing Reddit archiving.
ArcticShift is a project with that goal. It picks up where PushShift left off when the API changes killed that project.
Ask OpenAI maybe?
Does "all" mean all the URLs publicly known, or did they exhaustively iterate the entire URL namespace?
They iterated the entire URL namespace by having volunteers run a client so they didn't get IP banned.
The goo.gl URLs that are publicly known are already in the Internet Archive and Common Crawl crawls.
Glad I contributed to this in some small way.
Same, it's nice to see my username on the leaderboards.
Even though all I did was setup the docker container one day and forget about it
I wonder how many of them lead to private YouTube videos, Google documents, etc.
I was going to be cheeky and say "well, now you can download them and search" but it seems it's "Access-restricted-item: true" for some reason, above and beyond being 10G a pop <https://archive.org/details/archiveteam_googl_20250228144231...>
Happy go have contributed a hundred thousand links by running their docker container!
Google said they would keep hosting any recently-clicked link; does this mean that all the links are now recently-clicked?
“Recently clicked” wasn’t the criterium, it was “showed activity in late 2024”. So nothing that anybody has done this year – including this archiving – will affect which links Google keep alive.
Why? Did they ask anyone if it was okay? Anything sensitive at those links? Anything at those links people didn't want or need anymore? Maybe people thought those links were dead? Did Google provide a way to cancel those links first?
It's like when the GPT links were archived and publicly available that contained sensitive information.
It's a link, what privacy can one expect?
Especially with short links there's always the possibility of entering ~6 characters and getting a hit. So I believe expecting any secrecy from urls is silly.
That's like posting your passwords on Twitter because "Why would anyone find my account"
Sometimes to preserve history, you just have to go do what you gotta do.
After all, these are just short links. They link to other things on the Internet. Which is inherently public anyways.
You cannot expect privacy via a simple URL. These short URLs are short, hence programmatically scraping all the URLs.
The GPT Links situation is nothing like this imo. Both however do come down to the stupid human aspect.
If you want something to remain private, don't post it on the public internet.
Hell yeah!!! Fantastic work, everyone!
Gamefaqs remains unarchived.
Be the change you want to see in the world. Contributing .warc files to Archive.org isn't a gated club. My understanding of calling down the Warrior team is when something is time sensitive and needs to pseudo-ddos the site to get the bytes right now. Unless you know something about the demise of Gamefaqs, you have the rest of your life to archive a page at a time
Ok how do I access them, or is that not the point?
The point is that content previously referred to elsewhere on the Internet (for example, on Stack Overflow) via goo.gl doesn't have to suffer unrecoverable link rot.
They are being added to web.archive.org, so you would access them through that.