back

by wglb·17y ago·view on hn ↗
So long as it is presented to your web browser, you can also read it with a spider (wget, etc). However, your question suggests that what you do with the data might fall under some sort of copyright if you are going to republish it. I have a book "Spidering Hacks" that discusses this, but I am unable to locate my copy.

One additional point that the book makes is that one should keep in mind is your robot showing up in the logs of places you search. You don't want to overload sites, or get on the wrong section of the site admin's radar.