back

by Bender·9d ago·view on hn ↗
Seems about right. I rotated my logs this morning. Most humans go to access.log and most bots go to botpoop.log. This is the line count:

     2 access.log [1]
    40 botpoop.log [2]
2 is really 1 since a human will grab the CSS file. Most bots do not bother with the style-sheet so that's a 40:1 bots to humans. I could cut that down by blocking data-centers but then I inadvertently block a lot of VPN's which I really don't need to do for a static compressed blog served from ram. The bots just get a TCP Reset but it's still fun to log and study them. The most interesting one I've seen recently is ReadYou which may be a reader but it appears to be much more, possibly acting as a cell phone distributed bot collecting data for a centralized site.

[1] - https://nochan.net/logs/access.log

[2] - https://nochan.net/logs/botpoop.log

2 comments
Is there some existing mechanism already that counts how often an ip only scrapes the page and not the css and then block those origin IPs if it occurs “too often”?

Unfortunately, the best practice is to make css cacheable so you need to keep long histories.

I thought about that but to your point CSS is cachable. In fact I made mine immutable. No I just visually spot patterns and use that to study other facets of the agent, other client headers or lack thereof, supported protocol, accepted encoding and so on.
Maybe it is enough to include some css/js which is served without cache and is loaded after all user visible css/js is loaded. Make it small enough to not cause too much bandwidth for the server and legitimate clients. Then anyone who doesn’t hit that CSS file gets banned.
Possibly. I've noticed however that if people default to reader view they don't always pull the CSS. Seems to vary by client software. I'm weird enough to just stick with visual inspection and add characteristics when botters get too clever. Playing with the botters is almost as fun as Megabonk.
More realistic numbers now that HN and others have been visiting all day. Line count:

    104 access.log
    366 botpoop.log
Unique IP's:

     54 access.log
    181 botpoop.log
One of the botters Lily is showing off their mobile/residential bots to me. It is refreshing to see at least one person can follow instructions.