back
1 comments
Amazon EC2 does not use ECC RAM on anything but their very high end servers either (4XL and above)
This is highly unlikely. I'm on a team that administers an EC2 infrastructure with tens of terabytes of RAM, and I've never seen anything that resembles memory errors. Considering the findings of http://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf, this would be rather extraordinary.
That's disturbing. A quick Googling didn't find anything either way, can you provide a link that confirms this?
The whole point of the "cloud" is to use as commodity hardware as possible... accepting that failures happen. If you aren't using multiple regions and servers for deployment, with failover designs in your applications you are doing it wrong.
How do you suggest detecting corrupted memory that does not cause the instance to fail entirely, but silently corrupts your data or running processes?

Or your page cache:

https://blogs.oracle.com/ksplice/entry/attack_of_the_cosmic_...

Run n times, compare results. The idea of the same bit flipping twice is fairly ludicrous. These n times could be on the same machine, or distributed and compared with paxos. Then back everything up a billion times. Safety is extremely hard to guarantee, so the closest approximations will be expensive.
I doubt that would cause an instance to fail entirely in a majority of instances. Linux has it's own memory corruption checking and fixing (on 64-bit words afaik), also the customer can code their own sanity checking or CRC data to detect and eliminate them as much as possible. Also, many file or copy operations have built in corruption checking as well. Of course, on top of that you have your virtualization platform which I'm sure has it's own series of sanity checks and complex hardware handling logic.

Ultimately, it's a business decision of risk-management. Also, if you're in any kind of serious business like banking, there is already regulations around the standards of hardware/software you can use.

Source, please?
This is the sort of thing for which you want the opposite: a source that they _do_ use ECC RAM for EC2 (heh ECC EC2).
Users will never know because EC2 runs on paravirtualized kernels that don't have access to SMBIOS, which would tell us that information.

However, if we can determine what machine models they use (assuming they buy branded hardware), then we will know their chipsets, and from there, we should be able to deduce whether the RAM is ECC or not.

Actually when I was at NetApp we got pretty good at figuring out ECC failures. Just write a program that allocates 1GB array. Fill it with random data and compute a MD5 hash on it (you can do that fairly quickly). Now recompute the hash once every 5 minutes for a month. Run that on 125 instances. (that ends up being 1 Terabit of data btw) which makes a good detector for cosmic rays if it isn't ECC protected :-).