back

by gregsadetsky·9y ago·view on hn ↗
To scrape, start here

http://metmuseum.org/api/collection/collectionlisting?offset...

and increase 'offset' by 100. The JSON output contains image URLs. The total number of results is 441048, so finding another endpoint that doesn't enforce a limit of 100 on the 'perPage' argument would be great.

---

EDIT: thanks to spitfare, I updated the perPage argument to 100. The site doesn't allow larger values, but that's definitely a great start! (about 4k requests to get everything)

4 comments
They published an index on Github. It would likely be a more responsible way to access this versus hammering their API.

https://github.com/metmuseum/openaccess

Unfortunately, as stated in that repo's readme: `Images are not included and are not part of the dataset.`

The repo doesn't include the images, which is understandable; however, the CSV file doesn't even include links to the images.

Excluding the images themselves from the dataset is one thing, but there's an open issue to include a link to the images: https://github.com/metmuseum/openaccess/issues/2
edit: it doesn't download the highest available resolution...

This Perl script should get most of the images (but it would probably be better to save them in the original sub-directories):

    use strict;
    use warnings;
    use LWP::Simple;
    use JSON qw(from_json);
    
    my $url    = "http://metmuseum.org/api/collection/collectionlisting?offset=";
    my $args   = "&pageSize=0&perPage=100&sortBy=Relevance&sortOrder=asc";
    my $offset = 0;
    
    while ($offset < 5000) {
      my $decoded = from_json(get($url.$offset.$args));
      $offset = $offset + 100;
      my @results = @{ $decoded->{'results'} };
      foreach my $i ( @results ) {
        my $filename = $i->{"largeImage"};
        my $title = $i->{"title"};
        print "Status: ".getstore("http://images.metmuseum.org/CRDImages/".$filename,$title.".jpg")." ".$filename. "\n";
      }
    }
Why doesn't it download the highest resolution out of interest? Thanks for the script...
not sure why, I haven't looked into yet... probably the url needs to be fixed
If you finish scraping, can you still make a .torrent?
Are there any docs for this API? I wish I could filter by artist...