Item — HN Reader

The idea of creating an abstraction layer over thousands of MySQL nodes is a novel one, and should be commended. I like the idea a lot, and by DropBox's scale and success, it must work well.

I would be interested in a granular point-by-point comparison between this layer and native MySQL replication. In other words, if someone were to rewrite MySQL replication such that it did everything that this abstraction layer did (in addition to replication), what would it need to do?

Now (and this is strictly academic/theoretical), I'm curious what would be necessary to modify in the abstraction layer if the abstraction layer was to support a whole bunch of disparate SQL databases underneath it, i.e., Postgres, SQLite, SQL Server, Oracle, etc. (No, that wouldn't be practical, but it would be an interesting exercise to really learn where the gotchas might be where working with different SQL dialects...)

5 comments

> The idea of creating an abstraction layer over thousands of MySQL nodes is a novel one, and should be commended

Twitter did it eight years ago with Gizzard. See http://highscalability.com/blog/2011/12/19/how-twitter-store... and https://blog.twitter.com/engineering/en_us/a/2010/introducin...

Adding abstraction over MySQL is not novel. Facebook did it many years ago https://www.usenix.org/system/files/conference/atc13/atc13-b...

AWS DynamoDB is built on top of MySql according to this post: https://news.ycombinator.com/item?id=13173927

Amazon's Sable is built on top BerkeleyDB, acc to this: https://news.ycombinator.com/item?id=17595644

Vitess [1] is a similar kind of abstraction layer that's compatible with the native MySQL wire protocol. It also supports atomic cross-shard transactions using 2PC.

[1] https://vitess.io/

Vitess is a bit less of an application abstraction layer in the sense of Edgestore, Tao, and others, and more of an operational abstraction that hides physical sharding and taking the pain out of repartitioning to accommodate new capacity.

I've had some opportunity to work with the Vitess (now PlanetScale) folks on some other databases-related work and it's a great community and product so I would definitely encourage people to check it out.

Notably, from that site: "Vitess has been serving all YouTube database traffic since 2011, and has now been adopted by many enterprises for their production needs."

(I work at Google, but not on Vitess or YouTube)

Does anyone know of a vitess equivalent for Postgres? I’d love to use it

Citus seems to be doing pretty well.

Yugabyte is trying to do it but by using the storage-api (and not a proxy)

You can make one, it's open source. You just have to rewrite the MySQL pieces to work for Postgres.

It seems like you are looking for TiDB? It is deployed as a single MySQL compatible database that transparently partitions data so that writes scale horizontally. TiDB is actually the MySQL compute layer. TiKV is the underlying distributed Key-Value store that different protocols can be implemented on (one company made a TiKV-backed Redis).

Disclaimer: I work on this database.

https://pingcap.com/docs/ https://tikv.org/