back
1 comments
> And lastly, part three of the dream: it would be great if accessing storage from our services or batch jobs were as easy as accessing in-memory data.

I'm a believer!

The analytics/columnar store folk are well on the way. Voltron for instance,

> Voltron Data is building open source standards that support a multi-language language future. Central to this vision is the Apache Arrow project. https://voltrondata.com/blog/zero-copy-sharing-using-apache-...

Being able to send data around between different data processing systems is definitely shifting from specially tuned narrow systems to a core piece of infrastructure. First for analytics/columnar loads. Eventually more.

There's other pieces ideally slotting into place. Dma-p2p between/across nvme, gpu's, and systems is the target. Is what the object/data stores need, increasingly, to be bringing to the table. Unison is itself an object store, in the fascinating live process kind of way.

Im actually struggling to think of good references. Apache Ray's Plasma object store has a thread with plenty of theorycrafting (but no progress) on how to add some mind of RDMA. There's not a ton of other systems dedicated just to holding objects in shared memory, afaik. https://github.com/ray-project/ray/issues/30094