Twitter migrates data to Google Cloud

Disclosure: I work at Google Cloud (and directly with Derek and the Twitter team).

I’ll try to edit this page tomorrow when I’m at a computer, but there’s much more information in Derek’s talk at NEXT [1]. They (rightfully) didn’t want to get into a detailed “this is what we saw on <X>”, but Derek alludes to their careful benchmarking across providers.

While you should always assume smart people make economically reasonable decisions, Derek’s point about savings is about list price differences that result from total system performance (and not any sort of special discounting). I’m hoping a follow-up talk will let us say more about how the migration is going, while this talk was focused on the decision to move part of (!) their Hadoop environment to GCP.

[1] https://m.youtube.com/watch?v=4FLFcWgZdo4

Reading most comments here I feel like nobody wants to give a tiny bit of credit to either Twitter or Google, which I find disenchanting.

It is true that any cloud provider would get good publicity by being able to say that Twitter runs on their cloud, but that is precisely the same reason why we shouldn't just shrug this off as a publicity/business decision, because just like Google might have tried to persuade Twitter other cloud providers will most certainly have tried their best too. If Twitter still went with GCP then I think it is only fair to assume that there must have been some real advantage in going with GCP. I don't think anyone who says this must have been a pure business decision is entirely honest with themselves, because a cloud which cannot handle Twitter's volume is no good even if it was entirely for free.

I have nothing to do with Twitter (actually don't even like Twitter because it is such a toxic social media platform) and I don't like many things which Google does, but as an engineer myself I can only say that after using Azure, AWS and GCP for many years in a commercial setting that GCP is indeed years ahead. The quality, speed and reliability of GCP is second to none from my experience and I honestly couldn't say the same about AWS and especially not at all about Azure.

The scale of these megacorps is crazy! I wonder how you even plan to move that much data. I know that Amazon has a service called Snowmobile for stuff like this. They probably did something similar here.

"You can transfer up to 100PB per Snowmobile, a 45-foot long rugged sized shipping container, pulled by a semi-trailer truck."

https://www.youtube.com/watch?v=8vQmTZTq7nw

Is there anything new here? Twitter announced this move back in May of last year: https://blog.twitter.com/engineering/en_us/topics/infrastruc...

The video linked on the page is also from last August: https://www.youtube.com/watch?v=T1zjmNAuMjs

Throw a 2018 tag on this?

Correct me if I'm wrong, but this doesn't say what Twitter moved to the cloud. It could be literally anything, and may not in fact be core tweet or user data.

This strikes me as a team (or teams) outgrowing the storage options that were offered internally and choosing to outsource to fit their needs. Isolated use for one business case, and not indicative of a broad movement to the cloud on the part of the whole org.

I could be wrong, of course.

I'd like to know more. And I'd certainly like to know what the engineers at Twitter think.

The next global Twitter downtime?

Their credit card failed at Google Payments, acc gets marked for fraud and no one in support can help them recover. Twitter is gone for good.

I've scoured the comments and article, but I might have missed it... What did Twitter migrate from?? Did they have their own data centre before moving to Google Cloud?

Any inside stories on what twitter discovered when they benchmarked Google with other providers. I imagine they must have done a very exhaustive evaluation

Well hopefully google does not decide to abandon google cloud :)

Huge win for Google. I hope this helps Twitter focus on features and not firefighting infrastructure issues.

Just makes the eventual Google acquisition one bit easier.

Hun. I was under impression that at higher scale of operations building your own cloud is more economical. I guess money is not an issue for Twitter.

side note, if you want to know why distributed infrastructure engineering has been the place to be recently, think about the fact that engineers making $200-500k/year are making decisions that can save big corporations $1-5mm per year in compute/storage costs

The issue with GCP and with Google in general is support. I'm certain that's not an issue for Twitter as there are probably multiple people on call just for the Twitter deal. But it might mislead the average company down a bad path believing there is decent support with GCP. Any custom issues (like billing) and it's a headache dealing with Google.

What Twitter is paying for and the level of service they receive from Google, is very different from the typical GCP pricing and service level that would be available for say, your company...

Realistically, it can't be considered as the same service, can it?

One thing to remember with the benchmarks is that when evaluating a cloud provider, the benchmarks are useless if talked about with absolute numbers or with regards to hardware specs. Because they're so horizontally scalable, it's more a question of average cost per compute operation or cost per (peta) byte.

Both AWS and Google clouds are perfectly capable of running Twitter, and running any software that Twitter uses, even if the number of machines is different. The only "benchmark" applicable is actually the total negotiated cost of the machines required to get the job done.

So it's not about being able to do things on Google cloud with fewer processors or less RAM or faster hard drives - Google was willing to give them a lower total cost of ownership for reasons known only to Google.

Do not assume that Google (or anybody else) will give you a similar preferential deal. Ignore the "benchmarks".

How are they gonna move all that 300 pb of data to Google's data center? A fleet of semitrailer trucks?

I'm curious what happens to everyone at Twitter who was employed in managing their old data centre.

So what will happen to the small Cloud team in NY? I thought they were developing their own dc.

This really makes me wonder if Twitter finally made up their mind who they will merge with.

I think by this announcement, it is probably safe to say, even at the size of Twitter, it is probably still more economical to handle your operations to bigger players.

Hosting your own infrastructure makes less and less sense going forward, the cloud is the new king now.

More beneficial for Google I believe.

I know my comment won't move this discussion forward in any fashion, but I have to ask anyway.

What was this data, and why did they feel they needed to keep it?

I mean - 300PB is nothing to sneeze at. I can understand wanting to keep important data around (IP, for instance, or recent server logs for the last 30 days or even a year) - but this amount seems to go well beyond that.

In fact, it almost seems like they have stored every single tweet ever written. In addition to probably all of their logs, and who knows what else.

Why are they keeping all of that information? If it is all the tweets ever written - they don't have copyright to it, so such information wouldn't belong to them...

I'm not a big twitter user - can I search for a tweet on their system that I (or someone else) made say...5 years ago or longer? If this is a bunch of tweets - is that why they keep them around?

Are people in general aware of this? That is - if they are tweets - that Twitter has stored all of them, forever and ever, and that they are all still accessible (if not by the general public, then by law enforcement at the very least)?

If they are tweets - do users have any recourse at being able to remove those tweets (aside from those that they likely are legally bound to keep - I am thinking of the current POTUS's tweets, for instance)?

What is the ultimate purpose behind keeping all of that data? Even if it wasn't a bunch of tweets, whatever it is can't really be that useful otherwise, except for maybe the most recent stuff created in the past year, at best. Things like that - logs, etc - might be destined for a rotated "cold storage" and would ultimately "age out". Accounting records (financial data, etc) probably isn't that great a percentage of the data (if at all?) - but whatever was there, too, would likely need to be saved for a greater range - but even there maybe only 10-15 years worth (and it certainly wouldn't even comprise a fraction of the bulk).

So what is it? Why is it being kept? To what purpose? While 300PB isn't that great of an amount in physical terms (that is, the storage medium probably doesn't take up a great amount of space in a datacenter), it still isn't anything tiny, either (especially depending on what medium was used - I mean, I doubt they are storing it all on a ton of 256GB microSD cards - though that does bring up an interesting idea/thought to mind - but I digress).

I'm of the thought that companies shouldn't just "throw away" data - but it should be tempered by "importance to humanity over time". Not everything should be kept - but, for instance, it would have been nice if we (humanity) had kept around the original transmissions from the surface of the moon, or the blueprints to the rocket that took humans there. But we probably don't need all the various interoffice memoranda and fluff at the various NASA contractors (we probably don't need the accounting records - but BOMs might be useful).

That's just a couple of examples I can think of off the top of my head, but history is rife with instances of companies imploding or being bought out or restructured in such a manner that data is just "destroyed" instead of kept. Or lost, or otherwise made unretrievable to humanity.

Here with Twitter - an active company to be sure - we have the seemingly exact opposite - almost like r/datahoarders were in charge.

...and that should concern people a bit, regardless of which company it is - but especially a company like Twitter.

This feels like marketing material. Twitter jumped over to GCC because of the offer they received.

Next step - acquire twitter?

wow. talk about "loss of service challenges".

[Removed]

I have had a hard time dealing with twitter engineering since the account activity API debacle.