back
434 comments
Congratulations to the Spanner team for becoming part of the Google public cloud!

And for those wondering, this is why Oracle wants billions of dollars from Google for "Java Copyright Infringement" because the only growth market for Oracle right now is their hosted database service, and whoops Google has a better one now.

It will be interesting if Amazon and Microsoft choose to compete with Google on this service. If we get to the point where you have databases, compute, storage, and connectivity services from those three at equal scale, well that would be a lot of choice for the developers!

> It will be interesting if Amazon and Microsoft choose to compete with Google on this service. If we get to the point where you have databases, compute, storage, and connectivity services from those three at equal scale, well that would be a lot of choice for the developers!

There are also plenty of choices evolving for developers who aren't looking for hosted solutions (which can sometimes be a showstopper for enterprise on-prem deployments). There's a growing ecosystem of distributed open-source databases to look out for too.

Take Citus, for instance – a Postgres-compatible distributed store which automatically parallelizes normal SQL queries across machines. It's as easy to set up as adding an extension, and people are doing some staggering things in prod with it.

Different audience from BigQuery and Spanner, but no less exciting.

Disclaimer: no professional association, but love their product and the team.

It'll be interesting to see how well customers will adopt this. When I was at one of the two companies you mentioned above, we tried adding global snapshots (a la TrueTime, which is the real innovation in Spanner not the clocks) and demoed it to our DBAs/MVPs. They didn't understand what on-earth was going on. Wanted something that worked with existing clients. We just gave them classic 2PC and they went home happy. I think that's the reason why Oracle will keep on chugging. There just aren't that many workloads that need this sort of scale. It is real cool technology though and we always used to wonder why Google wasn't offer Spanner as a service.
As a bit of a veteran in the database industry, I concur (at least about the impact on Oracle's database business). There is a lot of pent-up demand for anything that offers distributed consistency.

We've been seeing this demand at Fauna. FaunaDB offers distributed consistency, based on Raft and the Calvin protocol instead of depending on specific networking and clock hardware. We've seen a big part of our appeal is the ability to run FaunaDB across multiple cloud services.

>well that would be a lot of choice for the developers!

A sad choice though. The centralization of computation is likely not a good thing in the long run.

>the only growth market for Oracle right now is their hosted database service, and whoops Google has a better one now

It could end up that way, but lacking INSERT and UPDATE will likely limit this to a niche market for now.

Amazon's Aurora databases seem to be solving the same problem, and are MySQL or Postgres compatible to boot.
It seems like Oracle could have a play here, working on adapting cloud infrastructure tools for managing on-premises data centers. That keeps them in play for customers who can't put their data in the cloud and those that haven't because they're already Oracle customers.

This pendulum swings. We're pretty near the apex now. A little work on ergonomics and these tools could be turn-key, and back we go to decentralized hardware.

> the only growth market for Oracle right now is their hosted database service

This is not true. Oracle is far more than a database company nowadays in the same way that Microsoft is more than Windows. Oracle has been acquiring high-growth startups at a significant rate.

Well the product manager is from Oracle.
Really a CP system but with the Availability being five 9s or better (less than one failure in 10^6)

How: 1)Hardware - Gobs and Gobs of Hardware and SRE experience

"Spanner is not running over the public Internet — in fact, every Spanner packet flows only over Google-controlled routers and links (excluding any edge links to remote clients). Furthermore, each data center typically has at least three independent fibers connecting it to the private global network, thus ensuring path diversity for every pair of data centers. Similarly, there is redundancy of equipment and paths within a datacenter. Thus normally catastrophic events, such as cut fiber lines, do not lead to partitions or to outages."

2) Ninja 2PC

"Spanner uses two-phase commit (2PC) and strict two-phase locking to ensure isolation and strong consistency. 2PC has been called the “anti-availability” protocol [Hel16] because all members must be up for it to work. Spanner mitigates this by having each member be a Paxos group, thus ensuring each 2PC “member” is highly available even if some of its Paxos participants are down."

The team here at Quizlet did a lot of performance testing on Spanner with one of our MySQL workloads to see if it's an option for us. Here are the test results: https://quizlet.com/blog/quizlet-cloud-spanner
This release shows the different philosophies of Google vs Amazon in an interesting way.

Google prefers building advanced systems that let you do things "the old way" but making them horizontally scalable.

Amazon prefers to acknowledge that network partitions exist and try to get you to do things "the new way" that deals with that failure case in the software instead of trying to hide it.

I'm not saying either system is better than the other, but doing it Google's way is certainly easier for Enterprises that want to make the move, and why Amazon is starting to break with tradition and release products that let you do things "the old way" while hiding the details in an abstraction.

I've always said that Google is technically better than AWS, but no one will ever know because they don't have a strong sales team to go and show people.

This release only solidifies that point.

Some interesting stuff in https://cloud.google.com/spanner/docs/whitepapers/SpannerAnd... about the social aspects of high availability.

1. Defining high availability in terms of how a system is used: "In turn, the real litmus test is whether or not users (that want their own service to be highly available) write the code to handle outage exceptions: if they haven’t written that code, then they are assuming high availability. Based on a large number of internal users of Spanner, we know that they assume Spanner is highly available."

2. Ensuring that people don't become too dependent on high availability: "Starting in 2009, due to “excess” availability, Chubby’s Site Reliability Engineers (SREs) started forcing periodic outages to ensure we continue to understand dependencies and the impact of Chubby failures."

I think 2 is really interesting. Netflix has Chaos Monkey to help address this (https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey). There's also a book called Foolproof (https://www.theguardian.com/books/2015/oct/12/foolproof-greg...) which talks about how perceived safety can lead to bigger disasters in lots of different areas: finance, driving, natural disasters, etc.

I wonder how this will affect adoption of CockroachDB [1], which was inspired by Spanner and supposedly an open source equivalent. I'd imagine that Spanner is a rather compelling choice, since they don't have to host it themselves. As far as I know, CockroachDB currently does not support providing CockroachDB as a service (but it is on their roadmap) [2].

[1] https://www.cockroachlabs.com/docs/frequently-asked-question...

[2] https://www.cockroachlabs.com/docs/frequently-asked-question...

For those trying to compare this with AWS Aurora, Aurora is more a regular database (MySQL / Postgres) engine with a custom data storage plugin that's AWS/ELB/SSD/EFS-aware. Because of this the database engine can make AWS specific decisions and optimizations that greatly boost performance. It supports master-master replication in the same region, master-slave across regions.

Global Spanner looks like a different beast, though. It looks like Google has configured a database for master-master(-master?) replication, across regions and even continents. They seem to be pulling it off by running only their own fiber, each master being a paxos cluster itself, GPS, atomic clocks and lot of other whiz-bangery.

The white paper is available here: http://static.googleusercontent.com/media/research.google.co...

for anyone interested

I wonder why they charge a minimum of $0.90 per node-hour when they offer VMs for as little as $0.008/hr. This is hugely useful even for single-person startups, so why charge a minimum of ~$8,000 per year?
Amazon likes to respond to Google with it's own price drops and product launches. It's telling that their announcements are orthogonal instead of direct competition with Spanner.

When Google announced Spanner back in 2012, I'm sure Amazon and Microsoft started teams to reproduce their own versions.

Spanner is not just software. The private network reduces partitions. GPS and atomic clocks for every machine help synchronize time globally. There won't be a Hadoop equivalent for Spanner, unless it includes the hardware spec.

Thomas Watson in 1943 amd his famous quote: “I think there is a world market for about five computers".

If he was alive, he could say these computers are Google, Apple, Microsoft, Amazon and Facebook.

How does this compare to AWS Aurora in terms of pricing and performance?

With Aurora the basic instance is $48/month and they recommend at least two in separate zones for availability, so it's about $96/month minimum. Storage is $.10/GB and IO is $.20 per million requests. Data transfer starts at $.09/GB and the first GB is free.[1]

Spanner is a minimum of $650/mo (6X the Aurora minimum), storage is $.30/GB (3X), and data transfer starts at $.12/GB (1.3X).

Of course with Aurora you have to pick your instance size and bigger faster instances will cost more. Also there's the matter of multi-region replication, although it appears that aspect of Spanner is not priced out yet. So maybe as you scale the gap narrows, but it's interesting to price out the entry point for startups.

[1] https://aws.amazon.com/rds/aurora/

Forgive my ignorance, but could someone explain in layman's terms in which situation this would be helpful? E.g. if I have 1TB of data would I use this? If I have 1GB with a growth rate of 25GB/daily would I use this?
> Today, we’re excited to announce the public beta for Cloud Spanner, a globally distributed relational database service that lets customers have their cake and eat it too: ACID transactions and SQL semantics, without giving up horizontal scaling and high availability.

This sounds too good to be true. But it's Google, so maybe not. Time to start reading whitepapers...

Interesting reading: Spanner Whitepaper

https://cloud.google.com/spanner/docs/whitepapers/SpannerAnd...

Link to the actual OSDI paper (not the simpler whitepaper) https://static.googleusercontent.com/media/research.google.c...
Looks cool, but the pricing seems a bit non-cloud-native (or at least non-GCP-native).

"You are charged each hour for the maximum number of nodes that exist during that hour."

We've been educated by Google to consider per-minute, per-instance/node billing normal - and presumably all the arguments about why this is the right, pro-customer way to price GCE apply equally to Cloud Spanner.

While everyone is puzzling over how Spanner seems to be claiming to be CA, I would like to take this opportunity to bring up PACELC[1].

The idea is that the A-or-C choice in CAP only applies during network partitions, so it's not sufficient to describe a distributed system as either CP or AP. When the network is fine, the choice is between low latency and consistency.

In the case of Spanner, it chooses consistency over availability during network partitions, and consistency over low latency in the absence of partitions.

1: http://cs-www.cs.yale.edu/homes/dna/papers/abadi-pacelc.pdf

> clients can do globally consistent reads across the entire database without locking

How is this possible across data centres? Does it send data everywhere at once?

Seems too good to be true of course but if it works and scales it might be worthwhile just not having to worry about your database scaling? Still I don't believe it ;-)

EDIT: further info...

> Spanner mitigates this by having each member be a Paxos group, thus ensuring each 2PC “member” is highly available even if some of its Paxos participants are down. Data is divided into groups that form the basic unit of placement and replication.

So it's SQL with Paxos that presumably never get's confused but during a partition will presumably not be consistent.

I was like YES!!! Then I read for a single node it is 90 cents per hours then I was like NO!!! so absolute minimal cost for me is $648/month? I was hoping there was like a dev version. Maybe I didn't read the fine print?
One thing to note is Spanner's transactions are different compared to what you get with a traditional RDBMS. See https://cloud.google.com/spanner/docs/transactions#ro_transa...

An example is the rows you get back from a query like "select * from T where x=a" can't be part of a RW transaction. I believe because they don't have the time-stamp associated with them. So, you have to re-read those rows via primary key inside a RW transaction to update them. This can be a surprise if you are coming from a traditional RDBMS background. If you are think about porting your app from MySQL/PostgreSQL to Spanner, it will be more than just updating query syntax.

Disclaimer: I used F1 (built on top of Spanner, https://research.google.com/pubs/pub41344.html) few years ago.

>> Remarkably, Cloud Spanner achieves this combination of features without violating the CAP Theorem.

This is the best weasel PR language I have seen in a long time.

Note that the sentence does not actually proclaim that they solved (the previously "unsolvable") problem of achieving distributed consensus with unreliable communication while maintaining partition tolerance and availability.

The blog only says they don't "violate" the CAP theorem -- whatever that means. So the statement is technically correct. Still the intention is obviously to mislead the casual reader (why else would you start the sentence with "Remarkably"?).

A litmus test: The same statement is true for MySQL - or _any other_ database in fact:

  >> "Remarkably, MySQL achieves this combination of features without violating the CAP theorem"
It's a bit like saying

  >> "Remarkably, MySQL is not a perpetuum mobile"
Given that CockrochDB is based on Spanner and F1, this DBaaS sounds like it will compete directly with them.
Is JSON data type support in the works? Seems to be a very commonly requested feature these days.
I can vouch for Spanner: it's a badass piece of Google's infrastructure.
Related, I wrote a blog post on the network latency between Google Compute Engine zones and regions. I'm assuming Cloud Spanner will still have these latencies once multi-region is deployed. Cross-zone latency on GCE is very good though.

https://blog.elasticbyte.net/comparing-bandwidth-prices-and-...

Oh this looks really compelling! Though I'm guessing this is targeted to companies? I'd love to use this for some personal projects but the pricing seems really high. Am I reading it right that a single node being used at least a tiny bit every hour is about $670 a month?

Maybe I'm misunderstanding how the pricing works here. Any clarification would be highly welcomed :)

What is TrueTime really? Are their Distributed Systems 'sharing a global clock'?
Does it support spatial objects / can it replace PostGIS?
> This leads to three kinds of systems: CA, CP and AP,

What is a distributed system that is CA? Can you build a distributed system which will never have a partition.

Few questions from reading the docs:

1) How big can all the colocated data for a single primary key get before they don't fit within a split? Can I implement a GMail-like product where all the data for a single user resides within one split?

2) Is there a way to turn off external consistency and fall back to serializability? In return you get better write latencies. This is similar to what CockroachDB provides?

Here is a very interesting video from 2013 of Martin Schoenert explaining the Google Spanner White Paper (In german though): https://www.youtube.com/watch?v=2QKewyoOSL0

Now he works for Google as an Engineering Manager.

Doesn't seem possible to use this yet. No client libraries and no samples: https://cloud.google.com/spanner/docs/tutorials

Have they documented the wire protocol? I couldn't find it.

> If you have a MySQL or PostgreSQL system that's bursting at the seams

Postgresql ? How does this work for people migrating from traditional SQL databases - typically people use ORM. How would this fit in with, say , Rails or SqlAlchemy ?

Very interesting. How does this pricing compare to AWS Aurora? https://aws.amazon.com/rds/aurora/pricing/
So does Cloud Spanner replace the existing Google Cloud SQL offering [1]? What are the pros/cons of each?

[1] https://cloud.google.com/sql/

Is this similar to AWS Aurora or is this something else completely different?