back

by mlhpdx·3mo ago·view on hn ↗
The screenshot on the repo only shows a comparison to the AWS CLI. Here is a comparison for uploading a single 1KB file with the AWS CLI, a minimal AWS API client and raptor:

  ~ $ time aws s3 cp data/1kb/file_1.bin s3://${BUCKET_NAME}/test-upload/ --quiet
  
  real    0m0.720s
  user    0m0.593s
  sys     0m0.070s
  
  $ time ./s3upload data/1kb/file_1.bin s3://${BUCKET_NAME}/test-upload/ --quiet
  Uploaded data/1kb/file_1.bin
  
  real    0m0.327s
  user    0m0.136s
  sys     0m0.062s
  
  ~ $  time ./raptor data/1kb/file_1.bin $RAPTOR_ENDPOINT --server-key $RAPTOR_SERVER_KEY --client-key $RAPTOR_CLIENT_KEY --silent
  
  real    0m0.303s
  user    0m0.013s
  sys     0m0.014s

AWS S3 CLI or API, it's the transport and protocol overhead that are burning the extra CPU time (power). At least, that's my conclusion.

Code for the s3upload tool is here: https://gist.github.com/mlhpdx/aaf99a468c06747dbea12f9639102...

The raptor tool has another nice advantages not possible with the AWS API:

Optional Acknowledgement. The `--confirm` option allow skipping waits for ACKs, which takes the run time down to just what it takes to put the outbound packets on the network. That makes uploads very fast, and the reliability is tunable with the `--overhead` option. On networks with significant packet loss this can make urgent uploads much, much faster.

The `--rate-mbps` controls the transmission rate. Sometimes slower is better, and having this option means not needing to setup bandwidth control using `cgroup` or similar.