i) Javascript: Very few crawlers can read your javascript code. Look at the source html of Google. Is pure Javascript!
ii) Accept crawling based on IPs.
iii) Use captchas.
iv) Use cookies and Measure the crawling speed.
v) Think that if someone wants to copy your content, they can just look at the cache of some search engine, they don't need to crawl you.
back
2 comments
Javascript isn't going to stop me from crawling a site if I want the content. Browser automation is simply too good.
This is basically cloaking (showing your content one way to regular readers and showing it a different way to search engines,) which could get you slapped by Google. Not worth this risk IMO. However, maybe there is a way to do this so you don't have to cloak? Not sure, I have never looked into it.
Just support Google's AJAX crawling standards and disallow the escaped_fragment requests from crawlers that are not on your whitelist.
... Isn't that exactly what the New York Times does with its required sign-on? You get prompted to sign in unless the referrer URL is google.
I didn't say cloaking, you can show the same content in javascript and html. Show html for search engines.