I didn't understand what Cloudflare is saying, so perhaps somebody else could clarify - Google already crawls sites to index them for its search. So why would Google need a separate AI crawler? Cloudflare (according to the article) claims they already "block" Gemini - does that mean Google does operate a separate AI crawler? Why though, when they already have a huge index? What does an AI crawler do different from a search engine (indexing) crawler?
back
6 comments
Google has made it very difficult to completely block their AI crawling by also using the standard googlebot search crawlers to feed data into their AI overviews and other AI features within Google search. Google says there is a workaround but it also blocks your site from fully indexing in Google search. This also also all covered in the article though.
> What does an AI crawler do different from a search engine (indexing) crawler?
Many people don't want the extra bot traffic hitting their site that comes from AI, especially when AI chat & AI overviews in Google provide such a small amount of traffic in return and that traffic pretty much always has horrendous conversion rates (personally seen across multiple industries).
It doesn't seem like the extra traffic is the issue. People don't want Google's AI from reading and summarizing their data and thus preventing clickthroughs. Why would I click on your site if google did all the work of giving me the answer ahead of time?
> It doesn't seem like the extra traffic is the issue.
it really can be, Anubis AI crawler detection was create mainly because of "way to many AI bot requests" to quote
> This program is designed to help protect the small internet from the endless storm of requests that flood in from AI companies.
Both are an issue. People don't want AI overviews cannibalizing their website traffic. People also don't want AI bots spamming their website with outrageous numbers of requests everyday.
In the specific case of Google would there be any additional traffic that isn't just the normal googlebot? I can't imagine they would bother crawling twice for every site on the internet.
There are about a dozen Google crawlers that can hit your website for different reasons:
https://developers.google.com/search/docs/crawling-indexing/...
Google-Extended is what is associated with AI crawling, but GoogleBot also crawls to produce AI overviews in addition to indexing your website in Google search.
While the number of crawlers and their overlapping responsibilities makes it difficult to know which ones you can safely block, I should also say that pure AI company bots behave 1000x worse than Google crawlers when it comes to not flooding your site with scraping requests.
which again comes back to
this is a problem which needs regulatory action, not one which should be solved by a quasi monopoly forcing it onto anyone but another quasi monopoly which can use their monopoly power to avoid it
require
- respecting robots.txt and similar
- require purpose binding/separation (of the crawler agent, but also the retrieved data) similar to what GDPR does
- require public agent purpose documentation and stable agent identities
- disallow obfuscation of who is crawling what
- do enforce it
and sure making something illegal doesn't prevent anyone from being technically able to do it
but now at lest large companies like Google have to decide weather they want to commit a crime, and the more they obfuscate that they are doing it the more there is prove it was done with a lot of bad faith, i.e. the higher judges can push punitive damages
combine it with internet gateways like CF trying to provide technical enforcement and you might have a good solution
but one quasi monopoly trying to force another to "comply" with their money making scheme (even if it's in the interest of the end user) smells a lot like you can have a winnable case against CF wrt. unfair market practices, monopoly power abuse etc...
I find it wild that you focus on CF being a monopoly here when they are providing tools that help publishers not have all of their content stolen and repurposed. AI companies have been notorious over the last few years for not respecting any directives and spamming sites with requests to scrape all of their data.
There is also nothing stopping other CDN/DNS providers spinning up a similar marketplace to what CF is looking to do now.
> this is a problem which needs regulatory action
I thought we were broadly opposed to regulatory action for a number of reasons, including anti-socialism ideology, dislike of "red tape", and belief that free markets can solve problems.
Google does not use a separate crawler, but they do use the robots.txt semantics to control what's allowed in the AI "index" separately from the search index. From the docs:
> Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity.
https://developers.google.com/search/docs/crawling-indexing/...
The assumption is that "Google" is a good player, not a malicious one. Same idea as robots.txt. It's only there to configure the "good guys" behavior not to prevent malicious actors.
So I'm assuming Cloudflare is basically asking Google to split its crawler's UA and distinguish between search and AI overview and respect something akin ot robots.txt
cloudflare wants to sell a "AI bots have to pay to access your site" service, so they need to argue it out with google one way or another
the reality of the tech is irrelevant
Google does crawl with a different crawler. Makes no sense other than clearly two different teams with different goals.
it could be the "grounding with google search" feature when using Gemini models