Other than backwards-compatibility, I can imagine simplicity being a reason. For instance, sequential pushing makes it easier to calculate the sha256 hash of the layer as it's being uploaded, without having to do it after-the-fact when the uploaded chunks are assembled.
back
3 comments
The fact that layers are hashed with SHA256 is IMO a mistake. Layers are large, and using SHA256 means that you can’t incrementally verify the layer as you download it, which means that extreme care would be needed to start unpacking a layer while downloading it. And SHA256 is fast but not that fast, whereas if you really feel like downloading in parallel, a hash tree can be verified in parallel.
A hash tree would have been nicer, and parallel uploads would have been an extra bonus.
sha256 has been around a long time and is highly compatible.
blake3 support has been proposed both in the OCI spec and in the runtimes, which at least for runtimes I expect to happen soon.
I tend to think gzip is the bigger problem, though.
> sha256 has been around a long time and is highly compatible.
Sure, and one can construct a perfectly nice tree hash from SHA256. (AWS Glacier did this, but their construction should not be emulated.)
But then every single client needs to support this.
sha256 support is already ubiquitous.
Every single client already had to implement enough of the OCI distribution spec to be able to parse and download OCI images. Implementing a more appropriate hash, which could be done using SHA-256 as a primitive, would have been a rather small complication. A better compression algorithm (zstd?) is far more complex.
I don't think we can compare reading json to writing a bespoke, secure hashing algorithm across a broad set of languages.
Reading JSON that contains a sort of hash tree already. It’s a simple format that contains a mess of hashes that need verifying over certain files.
Adding a rule that you hash the files in question in, say, 1 MiB chunks and hash the resulting hashes (and maybe that’s it, or maybe you add another level) is maybe 10 lines of code in any high level language.
That does not make any sense; as the network usually is a much bigger bottleneck than compute, even with disk reads. You’re paying quite a lot for “simplicity” if that were the case
I’m no expert on docker but I thought the hashes for each layer would already be computed if your image is built
It's complicated.
If you are using the containerd backed image store (opt-in still) OR if you push with "build --push" then yes.
The default storage backend does not keep compressed layers, so those need to be recreated and digested on push.
With the new store all that stuff is kept and reused.
That's true, but I'd assume the server would like to double-check that the hashes are valid (for robustness / consistency)... That's something my little experiment doesn't do, obviously.