Supercomputers: Obama orders world's fastest computer

This isn't only about the FLOPS, the big trend of these countries* ordering new supercomputers by 2020/2025 is very focused on power. Current supercomputers consume a lot.

Also the FLOPS measurement is a bit broken: It focuses on dense linear algebra problem, for which GPU or other accelerators boost the results easily. If all you plan to do is running simulations that are easily parallelized on GPU it is fine, for other types of programs it is hard to tell which is the fastest supercomputer.

* France is also ordering a would -be top 10 supercomputer : http://www.hpcwire.com/off-the-wire/the-cea-agency-and-atos-...

The extrapolation on the top500 supercomputer list [1] estimates the first EFlop computer in 2019. The math in the article is weird. They say 20x faster, but 20x33 PFlops is quite a bit less than 1EFlop.

[1] http://www.top500.org/statistics/perfdevel/

It could very well be that there's diminishing returns involved but I agree - they should be aiming to surpass the current tech by at last 100x in the next ten years.

Doubling time: 10 years * log(2)/log(247) ~ 15 months

x20 ought to take about five years and a half.

Computer hardware innovation is a textbook example of diminishing returns. With each improvement in processor performance, size, energy usage, and heat management, it becomes more expensive to push the tech further. We're currently witnessing this effect in action with the recent stagnation in consumer processor speeds. They are still getting better in size, energy and heat management, but average speeds have hovered around 2.5Ghz for years now.

I think this do not advanced more just because of economic reasons.

I have have read last 10 years that Processing is a lot more cheaper did by network of computers and clusters, instead of a expensive supercomputer that also demands an appropriate building and infrastructure.

Then they just can just "order" another one. We're talking government here. Declaring faster computers by government fiat is already a dumb idea.

The real problem is getting 1 exaflop (or around it) within a reasonable power budget. The DOE's power budget for all of their supercomputing resources is 20 Megawatts, so at a full system level we would need to be at 50 GFLOPs per watt, while the best system right now is at 5.

Clearly designed to run AI for cyber-warfare / cyber-defense purposes? William Gibson's Black Ice coming to life?

It is important to not conflate "massively parallel" (HPC) and "massively distributed" (Internet-scale), they have different architectural requirements and solve different classes of problem. People with competency in either of these areas tend to overestimate their understanding of the other but they are not solving the same computer science problems even though they look similar on the surface.

Massively distributed systems do not get much benefit from low-latency interconnects. Massively parallel systems do, and in particular, it is a "throw hardware at the problem" kind of solution that helps cover for the fact that virtually no software designers can engineer efficient, non-trivial, massively parallel systems. MapReduce is a distributed model; outside of some trivial cases, it is a poor parallel model. And while the HPC community has a much better understanding of massive parallelism than the Internet-scale systems community, the HPC community largely doesn't grok massively distributed systems in the way that someone working on Google's infrastructure would.

I benefitted from having spent several years designing software for both HPC and Internet-scale systems. They are not fungible, and both communities grok things that the other is oblivious to. Even within the HPC community though, the number of people skilled at the design of massively parallel software systems is quite small, much smaller than people that know massively distributed systems.

You do not need two systems, you need one system and more people that have figured out how to design massively parallel software -- the real problem. It is difficult to overstate just how rare this skill is even within the HPC community.

> especially when the community has little or no skill at building NG Internet HPC systems

I would argue that the community of people who actually have the skills to take advantage of the interconnects in a classic HPC system is vanishingly small, and in consequence we've overbuilt them on an epic scale.

Allow me to vent. I had the good fortune to have a login on a "petascale" HPC system, and access to an allocation of hours.

The /scratch filesystem would fail weekly, which killed everybody's jobs. If you had a big run going when /scratch failed, you lost everything. Scratch failed so much because the models that were being used often did wildly inappropriate amounts of file IO --- debugging print statements, detailed intermediate calculations, excessively verbose output --- that worked all right in development but when run in parallel brought the filesystem to its knees.

Furthermore, the login nodes were almost unusably slow because of all the Python and Perl post-processing scripts running on them. This isn't even a matter of users being cheap with their hours --- post-processing would have been a tiny fraction of their allocations. Instead, it's that many of them gave no thought at all to how the post-processing might be structured and run through the batch scheduler, and saw no downside to abusing the login nodes for that purpose.

In conclusion, I can attest to at least one HPC system that was badly mismatched to its users' needs and level of sophistication, despite allocations of hours being awarded only to a small number of researchers from across the country through a highly competitive process. Building these things serves national and institutional pride far more than any utilitarian interest.

And it's not like there isn't precedent. I remember a couple years ago when Google showed off gene mapping at I/O (Urs' demo to visualize how easy it was to scale GCP From 1 to 10 to 1000 to 100000 cores), and now they've partnered with the Broad Institute to apply this more generally.

http://www.broadinstitute.org/google

One of the largest challenges in building an exascale cluster is communication. Computing power increases at a higher rate than memory throughput does, and memory throughput increases faster than communication infrastructure advances.

Many argue that an exascale computer can only be cost efficient if the communication capabilities scale highly sublinearly with the computation done in the subsystems [1]. In particular, you can't move the data, and new algorithms are needed that can deal with data that is arbitrarily distributed. This is quite challenging and unfortunately the theoretical computer science community seems to have decided that distributed memory algorithms have been covered since the 90s and are not worth their time. Yet they ignore the progress that has been made in other models of computation since, and many algorithmic improvements of the last decades are not applicable. It is high time to develop communication-efficient algorithms for the basic "toolbox".

I guess what I'm trying to say is that you can't just throw MapReduce at an Exascale machine and expect it to perform well. Instead, you need an environment that is rich in primitives that have been implemented in a communication-efficient way. It's faster and cheaper to spend a little more effort on local communication if that allows for reduced communication volume (and/or the number of connections that need to be established!).

The issue I have with the MapReduce approach is that it doesn't particularly care about data locality. Thus it is very hard to achieve communication volume sublinear in the input size, which is absolutely deadly in an exascale setting.

I also understand the frustration with MPI, it is a very low-level API focused on data movement. It can be rather frustrating to use, but there do exist tools to make it more fun (Boost.MPI with C++11/14 is an excellent example). That said, with a well-engineered set of algorithmic tools, ideally you wouldn't need to use low-level MPI calls at all. However, MPI still remains a useful tool to implement these things.

Exascale computing requires us to rethink a lot of things.

[1] http://www.ipdps.org/ipdps2013/SBorkar_IPDPS_May_2013.pdf Shekhar Borkar (Intel), Keynote presentation at the 2013 IEEE International Parallel & Distributed Processing Symposium

You're assuming the goal is to build the fastest supercomputer in the world. I think the goal is probably closer to "get the computing resources we need at the lowest cost".

You could compare raw FLOPS (Floating point operations per second) but that would only tell part of the story. These supercomputers are highly engineered for low network latency between nodes, which is necessary for many scientific workloads. Google and other companies are generally able to express their algorithms in highly parallel ways, which means there are much reduced requirements for communication between nodes.

Therefore, even if the raw performance in terms of FLOPS sound similar, the two systems will have widely differing performance on real workloads.

This could be slightly misleading. Supercomputers tend to be used for different types of computations.

or the Bitcoin farmers?

Yeah, but everyone knows that thing's doing illegal stuff (illegal now or soon-to-be illegal), and Obama doesn't want that on his record.

Er, do we?

I don't think knowing that is usefull. "Single processors" are all superscalar or pipelined these days, so the colossal single-thread FLOP numbers are counted as an aggregation over many arithmetic units and one has to coordinate these units (mainly avoiding branch mispredictions) in any application that takes advantage of them.

On a per-core basis, probably Power 8: http://www.anandtech.com/show/9193/the-xeon-e78800-v3-review...

Looks like somebody got to ~8.7GHz through overclocking a ~4GHz cpu.

http://hwbot.org/submission/%202615355

I happened to catch an overclocking competition being streamed on Twitch one late night many months ago. It was really interesting to see more about the methods and techniques involved and how the competitions work.

See my comment above for exascale info. The single chip performance will vary wildly. The important measurement is how many operations of useful work per second per watt the system will do. That's the gist of it I've learned from HPC people.

Exascale is power-hungry so power must go way down and efficiency of calculation way up.

Probably an intel xeon (or i7?) overclocked and cooled with liquid nitrogen. There may be a specialized processor with higher performance, but I doubt it.

I use i7 2600k at 5.0 GHz for some time. Probably best option for ordinary human.

I find it odd that a president would sign an executive order for a new kind of computer (even if the new kind of computer is technically impressive). Call me cynical, but I'll bet that there's a number of quid-pro-quo arrangements with big party donors (or soon to be donors) -- regardless of which party is in office.

I think he added an extra "A" there at the very last minute. :-)

Let's hope this is not just bluster :(

Maybe for embarrassingly parallel tasks, but if you require nontrivial interprocess communication, a server farm can't compete with the interconnect of a modern supercomputer.

This is not remotely true for a wide range of codes that matter to the supercomputing industry.