Item — HN Reader

The only solution is regulation. If all content created by anyone has a copyright, how does an implicit opt-in (which is what happens if you don't create a robots.txt file for your website) for scraping make any sense? Moreover, even if you have a robots.txt, AI (or whatever) bots often don't respect it (or use workarounds - they outsource scraping of such "restricted" sites to unethical third-parties to get the data; Meta has even resorted to piracy, openly!). So clearly, the logic and the "honour system" has failed.

Cloudflare, Google Captcha, HCaptcha etc. are all shitty technical solutions because, as we are all discovering, it comes at the cost of our privacy (i.e. our personal data may monetise these services) and / or our computing resource and time. If current copyright laws aren't sufficient to prevent this, we have to acknowledge the system is broken. The answer could be enhancing it with some kind of Digital Millennium Copyright Act (DMCA) -like laws, but in favour of the creators against BigTech or rogue actors.

- Web-scraping and copyright law - https://www.neudata.co/blog/web-scraping-and-copyright-law

- Why DMCA Claims Against Web Scrapers Face Long Odds - https://capstonedc.com/insights/why-dmca-claims-against-web-...

5 comments

I don't think regulation will stop web scraping, not least of which because it can be done from locations outside the jurisdiction of the regulations.

> we have to acknowledge the system is broken

The system is broken. It probably takes, what, 10 seconds or less to use a residential or foreign proxy, 6+ months to internationally track and prosecute a single offender? So like a million times more effort going the regulatory route.

Just as criminal laws don't end all crimes, copyright laws and anti-scraping regulation won't end all scraping. But it will greatly reduce it and limit it to rogue actors. Two examples I can cite here are the laws against email spams and laws against unsolicited marketing calls - they had a definite impact in reducing both (even in India, from where I am, where implementation of laws are often lax).

I basically agree that the idea should be to reduce, not eliminate, bots.

However, a big difference with crimes involving the internet is that they can be launched from anywhere. In the real world, I can't steal from someone unless I'm physically present in the same country as my victim. On the internet, the US could outlaw scraping and Russia would keep doing it.

We already have ways to deal with badly behaving countries.

One thing I want to know is how porn spam (that is, spam containing pornographic image attachments) vanished so suddenly in the early 2000s. It used to be a huge segment of spam, and then one day it all just vanished.

So.. what made it go away, and how could learning about that help us make all the rest of it go away too?

Visa, Mastercard, and PayPal stopped processing payments for adult sites that used deceptive marketing. Hosting providers terminated accounts that sent image-based spam because the legal risk was too high. The same approach would work for other spam types. Target the payment processors, the hosting providers and make spam financially unsustainable.

Exactly. Bot activity is a problem of volume, not all-or-nothing. Solving 95% of it would be a win.

Or you could let information be free, at least the stuff that’s on the public net.

As for issues like bots overloading websites or using too many resources scaling laws will take care of it quickly, it’s not like you can’t serve thousands of RPS from a Raspberry Pi these days.

> The only solution is regulation.

The thing why Cloudflare got invented isn't AI scrapers. These are just the latest development... the original reason why Cloudflare got created and why it experienced such a meteoric growth is DDoS and botnets.

Yes. We need regulation in the AI space. But it will be useless as long as bad actors aren't held accountable - and a lot of the bad actors aren't in our jurisdictions. You got hacked devices all over the world in giant botnets, controlled by Russia, Chinese, Iranian and North Korean actors. You got Chinese AI scraper bots as China is heavily investing into training their own models. You got Indian, Filipino and Myanmar-based scammers.

And frankly I have no idea how to get all of that under control. As much as I'd like to see sanctions against both domestic and foreign enablers of abuse (which includes residential ISPs) - it's going to be one giant ass whack-a-mole game.

I just wish that DDoS wasn't such a huge racketeering game.

Who profits the most from it?

The services charging you to defend against it, that's who.

Or the regulated agents standard that cloudflare is conveniently going to steward alongside Google...

I mean, you could just turn on WebGL or use an approved, secure, agent to access the web. If you have nothing to hide, then you have nothing to fear.