>Many paywalled sites will let googlebot index their content but block other crawlers.
Doesn't that infringe upon Google's own rules? I always thought Google didn't like it when sites served its crawler content that's different from what users get when they follow Google's link.