back
3 comments
We launched Music Video Dispenser on HN roughly two months ago (previous discussion: https://news.ycombinator.com/item?id=13308820 ). At the time we rendered videos on the client using the brand new MediaRecorder web api. Now we've moved rendering to the cloud using MediaRecorder in a headless Firefox instance instead so we can guarantee perfect 60fps full hd videos without frame drops!
There is nothing there to explain how it works. Does anybody have some information?
We use the Web Audio API, HTML5 Canvas (with PIXI.js) and MediaRecorder to render and play-back generative music videos on the client side. However, it turns out that MediaRecorder can only work in real time, so unless you have a reasonably beefy computer, your video render might suffer from frame drops and random hangups. In an effort to eliminate these frame drops we moved the same js rendering code that we use on the frontend into a headlessly run Firefox instance wrapped in some additional node.js logic for controlling logic flow. That required splitting the rendering process into an offline pre-processing step for FFT analysis on the audio stream, a separate rendering job for rendering the frames themselves, then using ffmpeg to combine video frames, audio data and remuxing timestamps to ensure proper playback speeds.