I don’t have benchmarks for anything like that runtime. 100 files of 100K is the biggest. For raptor the biggest issue is picking the ‘—rate-mbps’ which controls how fast packets are sent. If defaults to a pretty low value so if you have a Gbps connection it will need to be specified to go fast.
> Do you have benchmarks for the total round trip time as well, rom sending to full reconstruction?
One use case I can talk a bit about involved transforming a large (100s of millions of records) hash database to individual S3 objects. Each was around 2kb. It took eons, even with Lambda executions sharing the work at scale.
The full end-to-end time can be observed using ‘—confirm FILE’ which will make raptor wait until the multipart upload is completed (and confirmation notification received). The baseline number is around .5s but it varies. Larger blocks with repair packets in use take longest.
> People can tar + ztsd infinitely many files and upload that archive normally and avoid the small-file problem altogether.
Sure, is that works for the use case. But sometimes small files i S3 are the desired outcome. For example, to serve as web pages or a static JSON API.
> Your pipeline involves quite a lot of moving parts.
It seems simple to me. It’s just a couple queues and a couple Lambda functions (one very simple). The FIFO queue is important and does double duty - ensuring only one completion event gets through, and keeping them in order. I didn’t add much error recovery. Resending files is often the fastest path. There are some ways to get there though. If you use the ‘—no-prefix’ flag only incomplete blocks will be processed on repeated uploads — not a solution, but something that could be built on.
> Pardon my dumb question. The final multi-part upload in S3 still uses the same number of HTTPs connections as if the file were sent from clients, right?
Yes, you have this right. Since we don’t control the S3 API all we can do is show how much better it could be if S3 had an API like this. It could save an insane amount of electricity.