back

by yamrzou·5y ago·view on hn ↗
This uses Chrome DevTools Protocol in a pretty clever way. I used it to archive a highly interactive website and it worked like a charm.

The README states: "It runs connected to a browser, and so is able to access the full-scope of resources (with, currently, the exception of video, audio and websockets, for now)"

I wonder what kind of limitations makes it hard to intercept those resources like the rest of the content.

3 comments
Video and audio is probably just a matter of not having gotten around to it yet. WebSockets are another matter. I’m not sure what one would do with a two-way channel in a general sense. It’s often not an idempotent operation.
This guy is right I just haven't gotten around to the audio and video it's a little bit more work I think. websockets is not a normal request response thing that I think is easy to cache so it's a little bit more complex and I haven't really begun to think about it in a way that I feel will be a good approach... Or any approach that I think could work.
Video stream is rarely accessed in one request. There may be prelight request to read format structure and of course any seek operation fragments it even more (Accept-Range header). That makes it hard to assemble reslurce for replay later.
Video files are massive, so it may just be the case that archiving videos takes so long they didn't want to support it.
That's also a consideration. And I haven't thought of a way to approach it where we get the nice sort of size of the archives that we have currently, and when adding video it would just balloon and I think that could be surprising so I haven't thought of a good approach to gel all that.
Not to mention things like e.g. livestreams ala Twitch, all it takes is one long stream at high resolution to take up your entire disk and then you can't archive anything else.