Many of those user-agents listed are often faked. Look up which ASN owns their IP. If I block most VPS providers most of the faked bots vanish. There are still some running from residential and phones using hijacked code (readers that are not really just readers but really multipurpose proxies). On that note, do not trust the linked source code but rather decompile the live code your phone is running and have AI analyze it.
back
4 comments
I did just this. Using a $2k a year database from a smaller provider that isn't maxmind, claude and I built a pretty slick ASN based categorization system. I can categorize an ASN as a residential IP, a service provider, a legit crawler/scraper, etc. For anything that is suspicious, I dynamically use turnstile to gate access to our service. Turns out there's no ISP for any VPN, they just contract with a shitload of mom and pop shady colocation services across the world.
We collect signals that help determine good vs bad networks. For example, large amounts of requests to .php endpoints, large amounts of empty accounts from the same /24 subnet, etc etc. All these signals let us automatically determine risk, and then put up a challenge. Authenticated users never see the challenge even if they are on a risky network (VPN 99.9% of the time), unless the network has been identified as 100% malicious, then it gets a full block.
Here's a small snapshot of the dashboard:
https://cos.ridewithgps.com/screenshots/6a7c54d0-12Aug26-358...
This was probably a total of 3-4 days of work, spread out over a couple months of iterative claude led hacking. I didn't know exactly what to build, but had some of the key architectural ideas in my head. Opus+Faable made easy work of it all, and ended up guiding some really slick improvements for performance.
I would say this has dropped about 20% of all traffic to our service, though it turns out turnstile is a massive target for bots, so replacing that with something custom is next on the list.
Contracting with their colocation facilities is exactly how that's supposed to work. If you don't actually operate a wide area network then you aren't supposed to be registered in these databases and have IP blocks. The exception is people who do anycast, but VPN companies don't.
You know all these guys just switch to residential proxies if they detect a site is blocking data centers, right? Because that's a very common thing to do.
Not sure what you mean by your first comment - there is no technical reason that I know of that prevents a VPN provider from having their own ASN and address space.
As for the latter comment....not sure what your implication is. Yes, bot/spam mitigation is whackamole, but there are consequences for not playing the game of whackamole. Luckily residential proxies are few and far between so far, but they will grow in popularity. When they do, and I can't get by with the occasional individual residential IP ban, we'll come up with other methods to handle.
Luckily the signal is strong with vulnerability scanning, which makes it pretty easy to automate. The only reason to put up whole ASN mitigation (captcha/turnstile, outright bans) is just efficiency. Nothing stopping individual IP banning. The scrapers are the tricky ones, since they more easily hide in legit traffic. However legit traffic has patterns that scrapers do not emulate (at least for a service like ours with millions of pieces of user generated content that's easily walkable), so you can still pull out the signal. It's just a little trickier.
Definitely a continual arms race though.
Majority of scraper traffic right now is from residential proxies
That’s not the pattern I am seeing, but I might be outside the norm. The majority of bot action (scraper, spam) comes from vpn providers by a long shot, ignoring (Chinese ASNs, Indian mobile ISPs etc). I see very little consumer isp action except cheap international providers, which are still swamped by vpn traffic.
Same for the origin IP address. The fiber leaving your country is tapped, and those people can inject packets with any origin IP that they want. Your ISP has no way to check if their peer actually received a certain packet from a certain country or not.
From a technical perspective, all this "china/russia" attribution is built on a quite shaky foundation. As a sysadmin you'd never know if it would be the British crown attacking your European company instead.
Not minimizing nation state cyber crime here, but the packet goes through many hands with different incentives.
Problem here is there are not single fibers attaching (most) countries, but a bunch of them. If you control both the ingress and egress for some particular users it's possible, but if you don't then your probing packing may end up back in China with a lot of evidence of backscatter.
I'd be surprised if there is a single route from EU to non-EU countries which does not pass through British control.
Does Britain own all fiber links between Switzerland and France?
Unfortunately I can't check how traffic flows from France to Switzerland because I'm not in France.
My traffic from Germany passes through a British-owned hop on its way to Switzerland. My German ISP is British as well so either way it wouldn't make a difference, they basically have all traffic twice.
Is there an easy way to block any requests originating from VPS etc instead of residential/commercial IP from legitimate users ? I know cloudflare does a few things but I really want to figure out a way to block any request say at nginx or caddy (reverse proxy) from reaching origin servers if they are not from an IP that is not a VPS etc.
I run an open source honeypot that collects these botnet scans and produces blocklists.
Blocklist download and configuration: https://knock-knock.net/blocklist
Honeypot dashboard, where you can see attempted attacks in realtime: http://knock-knock.net
Cool site. I was curious and dropped your 100k list into a reverse DNS lookup site [1]. They may still have some of the records cached. I recognized quite a few of the scanner nodes and some other usual suspects.
Yeah, that’s pretty interesting! You can also see a live view of the ASN/ISP leaderboard by going to https://knock-knock.net and choosing ISP from the carousel. That’s ordered by bot transaction count rather than IP count though.
It never ceases to amaze me that these ISPs don’t bother to shut down the botnets. They could do so very easily. For example, they could identify the IP address of every bot that hit this honeypot with their ASN with one API call: https://api.knock-knock.net/check-asn?asn=<asn number>. (See https://knock-knock.net/api). They just don’t care!
The ISP's do not have a financial incentive to shut them down. To them that's a paying customer. The feds will go after the big botnets if they are touching financial networks or siphoning enough money from people because there is usually a few big bank accounts and virtual currency exchange accounts they can seize once big enough to look good in the media. That's why it's on us and a few big CDN's to block some of them.
What if I have a slimy TV box or nasty on my phone, living on my network?
You take IP down, you kill the cancer but you also end up killing the patient.
This looks cool, where can I find the source?
It's on github with an MIT license: https://github.com/djkurlander/knock-knock. Have fun!
Thank you!!
Thank you for sharing. I will take a look.
There are several methods. [1] The most aggressive method-02 and method-03 on my document will block VPS and some data-centers but that also means it will block some legit users that are on a VPN. Most VPNs transit a data-center. If experimenting with these methods use a test server that you do not care about and set up a dummy site and ask people in your circle of friends to test it. I have to step away for a bit but if you have questions I will try to answer.
[1] - https://nochan.net/b/Internet-Crap/20260606-How-To-Block-Som...
Your site does not currently seem to be reachable / responsive when I try to reach it from a US Comcast IPv4 address - you are not advertising IPv6.
Edit / Update: It was Apple's Private browsing mode that causes it not to respond. I can now see it when this is disabled.
I've noticed they strip away a header [1] in private browsing mode but I don't know why they do it since it does not disclose anything about the person. I think that may be the same thing that causes some people grief on Cloudflare as well.
Per your link...
> block http 1.1, real users only use 2.0
Chrome on android and Firefox on linux both appear to use 1.1 still...
By default they use 2.0 [1] unless someone or an addon disables it or unless the person is on a really old version. OperaMini however will use 1.1. No idea if anyone here uses OperaMini.
There are some reader apps that act as a proxy that only support http/1.1. Be careful, some of those are not just readers and do not trust what they claim to be the source code. Some of them are created by cute and fuzzy bunnies.
There are a number of botters on HN, some that control residential and phone browser-hijacked systems. One was sending me playful messages the other day. I enjoyed the bot block-jousting with them.
Apparently nginx didn't enable it on my new server... why it's not enabled by default, I don't know... I retract my idiocy :P
No idiocy, sometimes defaults change or don't get updated by a deployment script. Different distributions may have a slightly different default configuration file depending on how involved the artifact maintainer is with the project. Or put another way, I've done far sillier things.
> /commercial IP from legitimate users
No, because legitimate users do not just use residential and "commercial" IPs. Like me, right now
You are the 0.001%
Much more than 0.001% of people care about their privacy or (the larger portion) do not have unfiltered access to the internet.
I second this. When I have tested blocking VPS/data-centers to my silly blog there were about a dozen people on HN [1] that could not view my site out of the roughly ~17,000 (not counting bots) that could. It's not a big number but those are real people and they count.
I am going to move full blocking to a test node that people can play with but I have to finish working with Claude to revise someones repo is is no longer maintained because one does not simply put an anonymous chan board on the great wide open internets without some critical thinking.
Yes but it’s not cheap. Maxmind and ipinfo etc sell a tier that tells you this information, then you can 403 based on it.
But the price is nuts like $40,000 a year.
Open source works just fine for this - take a look at geoacumen-country
Isn’t that only countries though? They were talking about detecting VPS
I work for IPinfo. We offer IPinfo Lite for free. With a little bit of time in identifying the ASNs, you can implement a decent way to block a good number of bots fairly easily using the free data alone.
Good to know, thank you. Would you do this by fully blocking particular ASNs? Or something more granular?
You can block entire ASNs. If you are frustrated with bots, blocking Tencent's entire IP address space would have very few downsides.
If you have fail2ban or NGINX logs, you can use our CLI to summarize those IPs and identify the ASNs you want to block. But before you block entire ASNs, make sure they are not classified as "ISP" type. For that, visit our website's ASN page first.
I have quite a few community posts around this approach. https://community.ipinfo.io/
If you have raw logs, you can send them to me as well, and I can review them and provide some guidance.
Focsec.com IP database for offline use (datacenters, VPNs, proxies, bots) runs around $1k/month for internal-only use.
You don't need to spend anywhere near $40k a year to get that info... You don't even need to spend $1
What's your strategy?
See my comment in the parent - there are cheaper options, don't use maxmind!
No, because the bad guys use residential proxies if you block DCs. They just prefer not to because it's more expensive. Meanwhile that silly HN project that scrapes and reformats your site is now dead before it began. Who are you actually defending and from what?
Yeah, that's exactly what these visits are: faked user agents that fail IP verification or Web Bot Auth. What's interesting is the surge across so many websites in the last week.
There are many possibilities but one of them could be some new vuln was released and they are looking for it. That would require looking at the URL's they are requesting. Botters run their own purpose built campaigns. Do you also have a summary of URL's requested by unique counts?
Looks like many of the paths relate to AI coding tools. There are some examples below the chart
You keep repeating this about a small minority of the tools that were posted.
I've had a similar bump in scanners in the past week, more than half of it is coming from MS and Google owned IPs and all of them are spoofing AI agents.