Item — HN Reader

I disagree, but maybe we're talking about different kinds of computation.

- There are tons of classes of data that work rather well with streaming modes of computation, especially video, music, and images, which make up most of the average user's data. It can even work for bigger data. I recently had my whole genome sequenced. They're sending me the data on a 500GB hard drive, which I'll upload to the cloud and use tools like iobio[0] (which I work on) to analyze it remotely. Even in cases where you can't use sampling, often the actual computation is as much of a bottleneck as the network throughput is.

- I'm thinking more along the lines of a true hard drive analog, ie the apps can store whatever arbitrary data they want, including binary blogs and special config formats. The service provides a simple API for storing/retrieving files, notifying on updates, and handling auth/permissions.

- Again there are a huge class of problems that work just fine with flat filesystems. I think databases are heavily overused, especially in cases where only 1-100 people need to use the instance. You could run into issues with large collections of music or photos, but I don't know anyone who keeps more than a few hundred photos in any one directory.

[0] http://iobio.io/

1 comments

Honestly, video, music, and images are the least-interesting use case, because they are large blobs of static data. Yeah you can put those in a flat filesystem just fine. We even already have open standards for this and a healthy ecosystem of file hosting providers and file-consuming applications that integrate with them.

However, the flat filesystem model would be awful for most productivity apps. Think GMail, Google Docs, Calendar, Slack, Trello, GitHub, Jira, etc. These are the use cases I care about.

Video, music, and images might be the bulk of most users data by raw volume but certainly not by frequency of access or importance.

> We even already have open standards for this and a healthy ecosystem of file hosting providers and file-consuming applications that integrate with them.

I'm not aware of any providers that offer the ease of google drive and the access of S3 (ie can I host a website on it). As far as I know both GDrive and Dropbox removed the ability to host files on the web in the last couple years. Do you know of any comparable products that offer both of those? I'd be very interested in looking at them. I think S3 is the closest, because you can access the filesystem through their browser APIs, and publicly over HTTP. And it got popular enough for people to make open server implementations. The problem with S3 at the moment are that it wasn't designed for this use case, so it's missing features like Firebase-style update events.

> However, the flat filesystem model would be awful for most productivity apps. Think GMail, Google Docs, Calendar, Slack, Trello, GitHub, Jira, etc. These are the use cases I care about.

GMail, for sure. GDocs, I can't think of why you couldn't implement it for a reasonable number of users with nothing but texts files using atomic updates, or maybe come up with a standard protocol for invoking text-based CRDTs. Or just send the change updates as git diffs and if someone tries to change a line that would result in a conflict reject that change and send back the current state of the document. This would be sufficient for any app for the which the data can be expressed as text files. Worst case you could even allow something like CF Workers for cases where you absolutely need centralized logic, but I think there are a large number of tasks you can accomplish without that. Even something like slack I think you could implement with text files on a disk. Pre-SSDs I'd have to agree with you, but nowadays I think a generic filesystem is good enough for most things that don't involve scale.

> Video, music, and images might be the bulk of most users data by raw volume but certainly not by frequency of access or importance.

True