I’d let them do their thing, why not?! They want the internet? This is the real internet. It looks like he doesn’t really care that much that they’re retrieving millions of pages, so let them do their thing…
back
4 comments
> It looks like he doesn’t really care that much that they’re retrieving millions of pages
It impacts the performance for the other legitimate users of that web farm ;)
Some scrapers respect robots.txt. OpenAI doesn't. SP is just informing the world at large of this fact.
The CTO isn't even aware of where the data is coming from (allegedly).
(admittedly)
The (allegedly) implies they do know, but to avoid possible litigation they feign ignorance. The CTO of ClosedAI is probably not a complete idiot.
> The CTO of ClosedAI is probably not a complete idiot.
Either way, ignorance is not an excuse despite how people generally react to it.
Thats what the whole thing is about. He is complaining that they don't respect robots.txt