Item — HN Reader

Respecting robots.txt is not a legal obligation. If you don’t want the traffic, ban the offending IP block or ASN, or put the origin behind Cloudflare for their crawler blocker.

You could go down the rabbit hole and try to find someone at Amazon to tell their crawler to chill. Don’t expect success but would make a fun blog post.

https://www.cloudflare.com/learning/ai/how-to-block-ai-crawl...

1 comments

My concern is more in relation to trying to use an endpoint from non-public source code, it seems negligent to randomly try endpoints like this

Whether it’s negligent isn’t terribly relevant. Is it illegal? It is not, they aren’t bypassing access controls. No different than using Shodan, scanning public IPs, crawling open directories, etc.

It would be different if they were attempting to brute force credentials to access an endpoint, but they aren’t.

> they aren’t bypassing access controls.

Weev went to jail for accessing public api's, https://en.wikipedia.org/wiki/Weev#AT&T_data_breach

> The flaw was part of a publicly-accessible URL, which allowed the group to collect the e-mails without having to break into AT&T's system.

It was argued that he didn't circumvent, but it didn't stop them from putting him in jail initially.

His conviction was vacated and Amazon has deep pockets and diffusion of internal liability. The illicit state drug charges did not help his case.