▲ 357 points
back
392 comments
> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.
It's massively less annoying than a captcha, which is both a longer delay (typically, at present) and a massive cognitive distraction/roadblock.
The anubis author has stated they recognize it's an arms race, but PoW scales. Captchas and other signals are already at the end of the road; any additional difficulty increases false bot-positives, which are already unacceptably high.
For websites running dynamic languages, a binary (anubis is in go) sentry that operates before[1] the website is forced to expend any resources, is usually a large improvement over a site-hosted captcha. I would rather, and I think most humans would agree, have to wait a few seconds, maybe even closer to a minute in the future, to get a website access token good for a day or a week, than be forced to solve a captcha.
The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.
[1] this is true regardless of whether anubis is in reverse proxy mode or auth mode.
Proof of work does not scale. It trades something fungible and incredibly cheap (CPU) for something incredibly expensive (user-visible latency). There is no set of parameters where the cost is going to be a meaningful deterrent to any kind of abuse (even something as low-yield as scraping) without adding crippling amounts of latency to real users.
> The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.
There is no dilemma. They get a token, they maybe do some automated multi-armed bandit per-site to figure out how to maximize the extraction rate they get from a single token, and then they use an IP for that many requests / that amount of time before ditching it.
Anubis is by far the least annoying throttler I encounter. Entirely agreed, just crank it up when you get a flood, I much prefer waiting a couple seconds to interacting with custom UI for tens of seconds.
I'm so glad to see that (essentially) HashCash is coming back. Now we just need it for email, like it was originally designed for...
Or you can go full Reddit and just block anything that seems even remotely suspicious.
Your sibling, roommate, neighbor that uses your internet, previous IP owner, posts too much? You get blocked too.
Using VPN? Blocked.
Your iPhone is too old, blocked.
Your screen brightness too low? Believe or not, blocked.
PoW barely affects the "residential proxies" aka. malware botfarms. The IPs are free for them and siphoning additional system resources for PoW doesn't matter at all for them. PoW only affects the large centralised scraping by the AI providers, which are not operating behind "residential proxies".
> The anubis author has stated they recognize it's an arms race, but PoW scales.
The scraper wars are largely between script kiddies and people with both deep intimate networking and DOM knowledge. Yes greyhairs, I’m looking at you.
The problem is, you can’t PoW every page load and resource request because the user experience will suck and people will run away. And that window - the gap between what people will tolerate vs draconian enforcement - is exactly what the scrapers exploit.
And looking at the PoW options out there - I’ve seen at least one PoW WAF (honestly can’t remember if azure or amazon) have their PoW boil down to repeated trigonometric functions, ie very optimisable.
It’s a neat concept, but the answer and future to my eyes look bleak.
I don't think PoW scales, because if the bot authors get serious they'll start using native implementations that are much more efficient than the web ones real users are running. In theory maybe Anubis could start using WebGPU to help close that gap, but then anyone without WebGPU support is out of luck.
Then again, a large portion of the problem seems to be bots making way too many requests and in general not being optimized in the first place, and this does help filter those out.
At least anubis works for me. (I run umatrix)
Unfortunately whatever HN is using routinely blocks my login with "Sorry."
some websites just always give me 403.
Well, we don't use a captcha either. If it were a choice between a captcha and a proof of work system, we'd have to reevaluate things. Luckily, for now, we're able to get away with a much lighter touch.
> but PoW scales
Not if the honest party is doing it in a browser: The same computer can so any POW so much faster in C than any amount jf JS and WASM that it will never ever ever be a contest.
> becoming much more obvious and easy to block, or they have to use massive amounts of compute.
If you believe this, please contact me: I think compute is free[1] and can probably help you out.
I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs.
I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.
The article at the end talks about how is very easy for arbitrary apps from app stores can install a residential proxy on your phone.
10 years ago, apps had to explicitly state if they needed network access. And then the powers that be decided that really all apps need network access no matter what. And both ios and android make it hard to deny apps network access.
But really, this finally explains the hordes of really basic boring games that just advertise other boring games. Idle games and the like that really just want you to keep your phone unlocked and open. Millions of downloads on the app stores for entirely offline content (and ads) and no way to block the network access.
The issue with scrapping is the intensity and volume of bots.
I think that nobody would care if I use wget or curl for few pages, e.g. because I would like to read a site as offline or archive it.
Btw average age of any page is 10 years. Deletion or structural change after acquisition is common, Signal vs Noise site recent wipe out could serve as an example why we need to archive sites.
One article mentioned in the OP was discussed here:
Disrupting the largest residential proxy network - https://news.ycombinator.com/item?id=46802748 - Jan 2026 (221 comments)
The comments are not showing up for me now, but when they were still showing for anonymous users, there was a link to https://commoncrawl.org. I've been sort of worried about letting agents hit websites, I wonder if a fetch_url agent tool could be made to look in common crawl first before hitting the web for it?
I wonder how much of this is traffic caused by peoples agents using web tools causing searches and fetches rather than general trawls of the internet.
Residential Proxies are the most emblematic technology of our era- a group of people looked at something that used to be considered a crime (botnets) and realized that if they just did it openly, no one would ever punish them.
Can BitTorrent’s architecture contribute anything useful here?
I admit this is a naive question. I have no idea how applicable bt is to web requests. This problem just seems to have a similar “too many people want this resource” shape.
Ever since bots became a problem on the internet 10-20 years ago, it has seemed like the common-sense solution is some kind of micropayment. Pay $0.01 to view the page. When money is on the line, scrapers are likely to be more well-behaved, even if they do pay. The problem is, and has always been, the friction of payment. How do you pay $0.01? The credit card processors will tack on a $6 surcharge. We need a trusted third-party that turn money into "internet article credits" that you can spend in small increments, like a video game. But I suspect that thousands of people have already though of this system, and tried it, but ran into some roadblock. I'm guessing there's some egregious regulation that makes micropayments impossible.
mmm, in many cases these residential proxies are media boxes, and they consent as much as anyone else consents to what amazon, or google or facebook does; it's buried somewhere in the recesses of the TOS.
The question is more about why the US and others can't properly enforce the bullshit all this amounts to.
What a pity. Mostly I just want personal archives of things so that I can search them much faster than commercial solutions and the like.
As this article points out, it's tremendously unclear who is using residential proxies.
The big AI models claim they're not using them. I'm not inclined to "just believe them", but no incriminating evidence has leaked, and—as pointed out in the article—many of the bots that are running on these residential proxy botnets are coded in incredibly stupid and inefficient ways.
How confident are people who research this stuff that the RP botnets are actually being used for AI training?
From the article:
> More recently, media-streaming devices have been identified as a major carrier of malicious scraping software. Sometimes the devices are compromised at the source; other times, they are just poorly secured and easily compromised after the fact.
I run an OPNsense firewall at home and the OpenWRT router at a hackerspace. Are there ways of auditing that devices aren't compromised? Tracking which devices still send lots of data when no one else is using the network?
>We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.
The first argument that it introduces delays to users is solid, but I would advise reconsidering on the second one that a PoW workaround will be found. The moment it does you'll be able to tell because Bitcoin will crash to 0.
Will bots use infected computers to do compute to work around it? Maybe, but it requires a CPU in addition to a network reputation, 2 mechanisms are stronger than one.
> widespread scraping of web sites in search of training data for large language models and related projects
This is a good thing, thanks to this we have powerful open source LLMs.
> This activity overwhelms sites with traffic.
When LLMs get good enough, we won't need those sites anymore :)
[not satire, this is what I think, without self-censorship]
https://fzakaria.com/2026/07/09/who-does-anubis-actually-sto...
I wrote about this recently as well.
I was involved in both sides of this battle over ten years ago. Things haven't changed all that much.
It's important to note that neither side has moral legitimacy. Not everyone who carries a rifle is a enemy. Not everyone wearing body armor is a saint.
I have given up on the idea that "human vs bot" matters at all when it comes to anything other than voting (which should only be done in person with paper and pen, by the way.)
You could make an argument that "likes" are a form of voting, but you shouldn't. We need to abandon the idea of supposedly democratized algorithms and focus instead on actual democracy.
I think this Anubis project is a terrible solution to the problem posed by aggressive web scrapers. Using a web browser with reasonable privacy settings has become a big loss in quality of life already, but the first time I encountered Anubis I got completely locked out of most web servers that deployed it. The situation has improved a little, but I hate that maintainers of great web services have rationalized themselves into believing that creating massive barriers to access their sites is a fair trade-off. Unsurprisingly, I have nothing but negative associations with their mascot.
The FSF has the right idea about all this:
> Some web developers have started integrating a program called Anubis to decrease the amount of requests that automated systems send and therefore help the website avoid being DDoSed. The problem is that Anubis makes the website send out a free JavaScript program that acts like malware. A website using Anubis will respond to a request for a webpage with a free JavaScript program and not the page that was requested. If you run the JavaScript program sent through Anubis, it will do some useless computations on random numbers and keep one CPU entirely busy. It could take less than a second or over a minute. When it is done, it sends the computation results back to the website. The website will verify that the useless computation was done by looking at the results and only then give access to the originally requested page.
> At the FSF, we do not support this scheme because it conflicts with the principles of software freedom. The Anubis JavaScript program's calculations are the same kind of calculations done by crypto-currency mining programs. A program which does calculations that a user does not want done is a form of malware. Proprietary software is often malware, and people often run it not because they want to, but because they have been pressured into it. If we made our website use Anubis, we would be pressuring users into running malware. Even though it is free software, it is part of a scheme that is far too similar to proprietary software to be acceptable. We want users to control their own computing and to have autonomy, independence, and freedom.
https://www.fsf.org/blogs/sysadmin/our-small-team-vs-million...
I don't run one of these sites that has these issues so I'm really not aware of this problem. How can it be that sites are getting overwhelmed with scrapers that are just looking for training data? You only need to scrape it once to train a model, so shouldn't there be less traffic from this than there is from search engines?
On the other hand if the article is wrong and the traffic is coming from other ai uses (like an agent visiting pages on behalf of a user) then that would make sense.
"The startup, which markets itself as the second-largest data collection firm after Alphabet Inc.s Google, has grown significantly on demand from data-hungry AI companies."
https://www.bloomberg.com/news/articles/2026-07-10/web-scrap...
Seems like Google is going after the competition
i think the best way to keep your site working for legit users is serving static cached pages to suspected bots. decide based off something like cloudflare bot score.
crawlers get redirected to content that might be stale but its still useful for them and costs you almost nothing to serve. put a warning banner on it so if a user (or smart agent) accidentally ends up on the bot version they can click on it and get anubis checked for the real site.
of course this only works for people like me who want their work to be used for training ai.
>There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time.
I don't get it. Don't we keep blacklists of this stuff? And if they hammer thousands of requests per site per second and never reuse an IP, they'd run out of addresses in a few weeks.
Then they'd switch to IPv6, and... well, are we using IPv6 for anything important?
Like we need it for IoT, but do you want random IoT devices talking to your web server? (IPv4 handled mobile phones just fine not that long ago, right?)
> There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time.
Maybe there's no point for the scanned server to block the address, but couldn't collective / shared block lists help with sites that may get scanned by the same address after the initial one?
The main problem becomes managing lists of millions of individual addresses. My (only semi-reliable these days, due to lack of time for maintenance) little project has nearly 2.3 million addresses recorded - although only 590k are from 2026, and only 38 were probes on ports 80 and 443. So maybe more manageable than I thought (but my servers don't host anything beyond personal interest to me, and access is filtered via cloudflare, which is it's own "internet control issue").
> In general, these companies range from those that aspire toward some appearance of legitimacy, advertising "GDPR compliance" for example, to others that are just overtly sleazy.
Overall, my gut feel on residential proxies is that they're an untrustworthy scourge. I'd be interested in any arguments for residential proxies by people who don't (intend to) profit from using it facilitating them.
In regards to Bright Data, one of the companies that attempts to appear legitimate, at minimum these domains should be blocked:
brdtnet.com
luminatinet.com
bright-sdk.com
luminati.io
As listed in this article, on HN's front page 34 days ago: https://news.ycombinator.com/item?id=48422993 (https://blog.includesecurity.com/2026/06/the-smart-tv-in-you...)
Sorry, I understand scraping is a problem, but talking about open Internet while simultaneously complaining you can no longer discriminate datacenter IPs like you used to is hypocrisy.
I use a datacenter-based IPv6 address because my local ISPs don't offer v6 connectivity and the Internet is already broken for me. And generally the entire idea of a "residential" IP address smells.
Has no one noticed their miniflux instance failing to fetch feeds because of this?
How much does routing traffic though residential proxies cost?
I’m skeptical that the problem they are trying to solve is truly unreasonable bandwidth demands.
Sometimes it feels like what people want is to only serve websites and content to good normal users but not evil bad “scrapers” (because maybe maybe your content will be monetized in some nebulous way) but … you put your content up publicly on the web! That should be part of reasonable use!
EDIT: Lwn.net is perhaps not a fair target of my ire.
“There is also a desire to not impede the operation of legitimate search engines, the Internet Archive, and other such groups. Some sites may add explicit allowlists to, for example, give the dominant search engine access to the site. Such measures have the effect of further entrenching a monopoly that already serves us poorly and should be avoided. We have, thus far, succeeded in that.”
Is reasonable! Many others are not
Google itself is a huge database.Who makes these rules depends on who's leading the market.
i wonder if residential ISPs can play a bigger role here
I find the notion that you would use residential proxies to scrape LWN somewhat laughable, I'm reading this article using a VPN.
residential proxy bandwidth isn't that cheap, I could see it be used on a reddit (though i would probably just mass register accounts to bypass their block instead).
This is a predictable consequence of age verification laws and social media bans. Formerly VPNs were a nice to have but now they are a necessity in many countries to navigate the modern internet.
The cheapest way to get a VPN (and if you're a horny and broke teenager perhaps the only way) is to trade your clean but censored IP address for an uncensored IP address in another country. You accept the bot traffic in return, or externalize it to your parents or the owner of the internet connection.
Disclaimer: I own and operate proxybase.xyz [0]
Hi HN,
I wanted to jump in and share a few thoughts. Not all the residential networks mentioned on this page are bad actors.
Transparency & Auditing: Our clients are completely open-source, and we run strict internal audits before every single release [1][2]. You don't need to be a security wizard to verify this, either. You can easily audit the code yourself—just clone the repo, feed it into an AI, and ask the right questions.
Ethical Sourcing: Consent is everything. At Proxybase, we always get explicit consent from our providers before adding them to the pool. This is exactly how ethical sourcing should be done. Historically, this industry has been incredibly shady think malware bundled into iOS/Android apps or second-tier smart TVs secretly installing background scrapers. Fortunately, the sector is finally becoming more ethically aware.
Fair Payouts: A lot of networks hold onto provider funds for months, staking them to earn passive income while making users wait. Between sky-high payout thresholds and endless waiting periods, it’s a broken system. At Proxybase, we have a $1 minimum payout sent directly to your wallet using US stablecoins.
If you have a better idea on how we can make this industry better, just lmk. I'm reading/writing on HN everyday.
Running a small public JSON API, the traffic breakdown is eye-opening. Roughly half is trust/uptime "scanners" and generic monitors; a solid chunk is well-behaved crawlers that declare themselves with real UAs and honor robots; and then there's a long tail of vuln-scanners blindly probing for /.env, /.git, wp-login and the like. The genuinely evasive residential-proxy scraping is a minority by volume but by far the hardest to separate from real users — it's the one bucket where UA and IP both look residential, so you can't tell bot from human without behavioral signals. What's shifted in the last year: the "polite" bots got politer, while the abusive layer moved almost entirely onto residential proxies. IP reputation alone is basically dead as a filter now.