Haha, is this a coincidence or did you take inspiration from my 2016 blog post?

"Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS"

http://blog.waleson.com/2016/01/parsing-10tb-of-metadata-26m...

We used a lot of the same tech: python, golang, S3, 32 core machines. Anyway, nice read and good use of Hetzner.