Item — HN Reader

Amazing work by the Team. Congrats y'all. It was one of the best presentations in the PGcon22.

I did email Heikki the following questions, in case if someone from Neon is around here.

a) How does Neon compare to polardb https://github.com/ApsaraDB/PolarDB-for-PostgreSQL.

b) The readme mentions a component "Repository - Neon storage implementation". Does it use any special FileSystem? Any links to read more about it?

c) Heard the cold start is a second (IIRC), how does that value differ if one runs Neon on bare metal instead of k8s?

3 comments

Thank you!

a. PolarDB is based on a similar idea. https://www.cs.utah.edu/~lifeifei/papers/polardbserverless-s.... This paper describes it. The biggest difference that I see glancing through the paper is that we really integrated S3 into the storage. In Neon architecture branches, backups, checkpoint are all the same thing and instant to run. This simplifies a good amount of database management AND deliver on better costs. S3 is cheap.

b. Neon doesn't need a special filesystem. Neon storage is in a way a filesystem, however it doesn't expose filesystem API. It's a key value store - serves 8k pages to Postgres and a consensus - update API to the key value store. Pages are organized in LSM trees and background processes put layers of the LSM trees to S3.

c. The cols start is 2sec right now. There is a dependency on K8S. Bare metal implementation will require new code to orchestrate starts and stops.

> S3 is cheap.

S3 has its limitations though, like too many small files and the get/delete/list ops get very expensive. There's also an upper-limit on throughput per S3-bucket partition. I guess, sstables that pageserver flushes periodically help work around these issues?

> Neon storage is in a way a filesystem, however it doesn't expose filesystem API.

Genuinely curious: When would anyone consider using filesystems like Amazon FSx for Lustre instead which is backed by S3 anyway over implementing a filesystem-esque abstraction of their own (like neon.tech does, and other solutions like rocketset.com, tiledb.com, xata.io, and quickwit.io do).

> Pages are organized in LSM trees and background processes put layers of the LSM trees to S3.

Curious how merges are handled? Also, are you using RocksDB / some other engine underneath?

> Bare metal implementation will require new code to orchestrate starts and stops.

Speaking of new code... SingleStore started as a very high-throughput OLTP database and eventually evolved to into a HTAP (?) database. Do you see Neon evolving in a similar manner, too?

Thanks!

1. Yes. Our first attempt at storage implementation had a problem with many small file. Then the team rearchitected it around LSM trees and it got a LOT better. Our benchmarks show that we are very close in performance with vanilla Postgres and Aurora. There are some "worst case" scenarios where Neon is worse than vanilla Postgres. Aurora has similar problems too.

2. It's best to custom build a storage system here. External distributed filesystems introduce complexity, cost, and bottlenecks that you don't control.

3. Purpose built. LSM trees also have a temporal dimension - LSN. You can fetch a page by pageId and LSN. This is what allows time machine and branching.

4. I call it convergence when OLTP and OLAP is one system - ultimate dream for a database systems engineer. Since I spent 10 years building it I have both scars and aspirations. I think it will come, but this will take a long time. HTAP is in a way a subset of convergence - most systems will have some HTAP. Neon will have some too, but for now it squarely focused on OLTP and helping developers build apps.

The way you describe it, to me, is one of those “this sounds obvious in retrospect”. Sounds completely elegant and “right”. Congratulations on a great idea. I really hope you pull it off!

Thank you! We are super hard at work. You can see our velocity here: https://github.com/neondatabase/neon

> we really integrated S3 into the storage

Will it be possible to use something else in place of S3? I'm thinking on-premise or what some would call a private cloud.

Right now, it should be possible to use anything that is compatible with the S3 API, as our current focus is on getting the product to the market. Once the business model is proven, we'll likely branch out to other clouds, with their storage providers.

If you can't wait that long to run Neon on your own cloud, feel free to contribute an integration to your persistent blob storage: the code is available under APLv2 here: https://github.com/neondatabase/neon/

>serves 8k pages to Postgres

will page size be tunable on neon cloud for larger datasets?

No Postgres only requires 8K. One can imagine adapting Neon storage to other engines then of course this can be extended.

> It was one of the best presentations in the PGcon22.

I can't find it on Youtube, do you have the link?

edit: I found the link, seems it is not on the Youtube yet: https://www.pgcon.org/events/pgcon_2022/schedule/session/236...

I can't recommend this presentation enough!

> c) Heard the cold start is a second (IIRC), how does that value differ if one runs Neon on bare metal instead of k8s?

Yeah, as Nikita mentioned it's 2 seconds now. We did some tests and measurements and on bare metal, it's sub 500 ms usually, so the remaining part is the k8s (+ our own control plane) orchestration overhead. For example, with plain Docker (which we use in CI in addition to k8s) it's around 1 second already.

K8s provides a convenient abstraction layer, though. So I think that we'll continue using it and optimization will come with pods pool / over-provisioning and it'll be realistic to bring the startup time closer to bare-metal.

-- Cloud engineer @ Neon