Item — HN Reader

Here are some awesome libraries I've used for HTML scraping:

1. Python - BeautifulSoup

2. Ruby - Nokogiri (use in conjunction with Watir if you're scraping a very client-heavy website).

3. C# - HtmlAgilityPack in conjunction with ScrapySharp (there's a nuget package for both) - I highly recommend ScrapySharp because it allows you to query elements using a very familiar Css selector type similar to how you query dom elements in jQuery. :)

Scraping online content is so simple these days, if a website doesn't offer an API you still have alternatives ;)

3 comments

> Scraping online content is so simple these days, if a website doesn't offer an API you still have alternatives ;)

Having written code to both leverage a site's (private) API and to scrape that same site (when the private API stopped existing), I would much, much rather use an API. Yes, the scraper works, but the scraper's code is much messier, and JSON keys, for example, provide some documentation in their own right. Looking at the scraper months later, there's a lot more headscratching. Scraping will work, but it also will leave you searching for many little pieces of information that would be exposed by an API but aren't by a static site.

Still, I will agree that scraping is much easier; I've used jsdom (Node.js) extensively, and, for my use cases, it feels like working in a browser (full DOM, scripts, etc.).

Another point in favor of APIs is that scrapers are very brittle, and likely to break with changes to content presentation. Also, scrapers have a lot more overhead as you need to both receive and parse the markup data.

Also recommend Mechanize for Ruby (uses nokogiri under the covers).

http://mechanize.rubyforge.org/

http://phantomjs.org/ or "its easier to deal with cousin" http://casperjs.org/ for very client heavy sites.

FWIW, you can get away with HTML only scrapers most of the time, you just need to look harder to find all the data. Totally recommend using "View page source" as that would always give you the original HTML vs the possibly altered DOM (after JS has run on the page) that you might see with Dev Tools/Firebug.

Mechanize is far and away the best and easiest way to scrape with Ruby until anything is rendered in javascript, which is explicitly not supported.

I tend to use Mechanized until I can't, then switch to Watir. Over time, I've found myself just strait up picking up Watir as it runs your browser directly and supports javascript rendering as a result.

How is performance with Watir? With casperjs a page takes me on an avg. 5-10 secs. to process.

Not great. About the same...

I recommend Selenium before I'd recommend PhantomJS in situations where Mechanize/Nokogiri don't cut the mustard,

I've found Selenium scripts much easier to comprehend, modify, and maintain over time than the PhantomJS scripts.

Check out casperjs, it should make life easier. Phantomjs by itself is extremely cumbersome in my experience.

CsQuery is another interesting library for C# along those same lines: https://github.com/jamietre/CsQuery