Wow. It looks like a 16hr drive to the Oregon center. Given an 8hr break in between shifts it works out to almost 10 Tb/sec transfer rate.
back
2 comments
Unless they’re shipping those disks as a ready-to-go container DC[1], you need to account for the time to unload and install them. Actually, you need to account for it on both ends.
They are though!
> The Snowmobile comes with a removable connector rack that needs to be mounted on one of your data center racks where it can be connected directly to your high-speed network backbone. The connector racks provides multiple 40Gb/s interfaces that can transfer up to 1 Tb/s in aggregate.[1]
so there is a bit of setup and teardown, but they're not plugging disks in one-by-oen
That's talking about customer premises though, not AWS side.
What's the difference? They'd probably do the same thing at both ends. Snowmobile is essentially a giant external drive hurtling down the highway, and the removable rack (which I'd assume is just switches) is its "USB" interface. Actually, it'd probably be more efficient on the AWS side, because they likely already have said interconnects in their facility and don't need to unload it.
If I were designing it I'd expose a private S3 interface, or maybe NFS, to the customer from the truck, store the incoming bits on-disk in the form that S3 expects, then just copy raw blocks from disk and merge metadata databases on the AWS side so you're not doing two passes through the S3 API. I can't imagine they're doing something much different than that. Not every customer would be in a position to handle iSCSI or FC, for example. You could unload the disks at the destination, too, but that seems less efficient and more disruptive to their lifetime.
I can't envision S3 ever wanting their secret sauce to be out there in the world available for everyone to potentially compromise and figure out. Merging the metadata together would be a complex task that would require significant engineering effort on all sides, especially given none of the components were ever designed with such a task in mind, and everything is highly tuned towards the actual types of workload they typically see. Pushing things through the front end API would be the quickest and easiest route for all parties concerned. That's what it's there for, and scaling will be a known quantity. Regardless of how impressive the Snowmobile is, and the quantity of data it stores, the bandwidth it would consume putting in to S3 would be a drop in the ocean compared to their usual workload.
Any evidence for any of those claims? Putting anti-tampering failsafes and GPS tracking on the Snowmobile doesn’t seem like such a stretch, when you consider they don’t rent this out to just anyone on the street.
And 100 PB is still a lot of data. Facebook in its IPO filing said their entire media library was at the time 100 PB. Backblaze’s total backup size was 150 PB in 2015, just 2 years ago. I’m sure S3 is at exabyte scale now, but even so, 100 PB is not a drop in anyone’s bucket.
> Any evidence for any of those claims? Putting anti-tampering failsafes and GPS tracking on the Snowmobile doesn’t seem like such a stretch, when you consider they don’t rent this out to just anyone on the street.
> And 100 PB is still a lot of data. Facebook in its IPO filing said their entire media library was at the time 100 PB. Backblaze’s total backup size was 150 PB in 2015, just 2 years ago. I’m sure S3 is at exabyte scale now, but even so, 100 PB is not a drop in anyone’s bucket.
I'm ex-AWS, and used to work on a team closely involved with both S3 and Import/Export (the team that produced snowmobile, though that expansion launched after I left) I'm well aware of the kinds of capacity scale S3 operates at, as well as some other storage teams. Remember that backblaze, for all its scale, operates out of a small number of datacentres, where AWS has operations right across the world.
Being ex-AWS also why I'm familiar with the way AWS Security thinks, and I can easily picture how they'd hit the roof if S3 was "ported" on to the device.
That's before we even begin to tackle the concept of just how hard it would be to port S3 in to a small scale platform as the Snowmobile. Don't forget, you're not just talking about all the things that make up S3, but all the things that make up Amazon infrastructure as a whole. One of the reasons Amazon is able to push out so many services is because they've got a mature and well established ecosystem behind the scenes that is designed to operate at scale.
You'd be talking year(s) of effort to port S3 into a snowmobile, at best, vs. a matter of a few months to port and fully test a layer on top of the device. When it all boils down to it, the S3 API is pretty simple. Why on earth would you choose the most technically complicated way to approach the problem?
It’s cool they actually do it like that, but transferring that amount of data locally still takes some non-trivial time.
> When your Snowmobile is on site, AWS personnel will work with your team to connect a removable, high-speed network switch from Snowmobile to your local network and you can begin your high-speed data transfer from any number of sources within your data center to the Snowmobile.
At the maximal throughput of 1 Tbps, 100 PB of data would still take almost 10 days to copy.
It may very well be “mmapped” onto S3/Glacier API on the AWS side, but a copy is still necessary on the customer’s side.
I believe it is s3 at re:Invent. Meaning it contains the s3 API in the trailer and you can use it just like you would s3 in one of the regions, just a local endpoint of it.
Would have to re-watch https://www.youtube.com/watch?v=z8-DeSuKf9I to be sure.
It's a ready-to-go DC: https://aws.amazon.com/snowmobile/ -- basically you plug in the fiber and power and you're good to go.
Not entirely because you forgot to count the data transfer on and off the drives used for transport.