back
58 comments
Controversial opinion here, but all of these distributed streaming architectures are massively overused. They certainly have their place, but you probably don't need them. I see it all the time with ML work. You wind up using a cluster to overcome the memory inefficiency of Spark, when you could have just used a single machine. For example, I've done huge graph clustering models on a single machine just by being smart about memory consumption. It would have taken an enormous and expensive Spark cluster.
This has been my experience, too. I worked at just one place that had a really good handle on high-volume, high-velocity streaming data, and they didn't use Flink or Storm or Kafka or anything like that. They mostly just used the KISS principle and a protobuf-style wire format.[1]

There is definitely a point where these sorts of scale-out-centric solutions are unavoidable. Short of that point, though, they're probably best avoided.

[1]: (It's truly amazing how many CPU cycles you can reclaim just by removing branch instructions from your message deserialization code.)

See McSherry et al, "Scalability! But at what COST?"

https://www.usenix.org/system/files/conference/hotos15/hotos...

That's a smart way of doing it. In much ML work, I've found that the hash trick can be used to train many more things in a constant memory, out-of-core fashion than most people think.
We have also seen this. The huge memory capacities now offered in the cloud makes single node processing very capable with entire TB datasets fitting into RAM, and running quickly enough to offset the hourly cost. Spark clusters are more efficient though for longer-running or continuous background processes.
And spark also has some other goodies: integration with storages, monitoring, fault tolerance, common algorithms implemented.
I've seen the opposite in our company. We did some data science, on a data that wasn't that big, but big enough to not fit into one machine. Everyone was reluctant to move to Spark, so data-scientists computed their models over subset of data: one day instead of one week of data. And after the project finished, and passed to client, they realized that subset was not representative enough to meet the criteria.

Of course, it's kinda their fault to assume that subset is representative, but if you start checking every assumption you make in data science, then you won't get far.

Had they used Spark from the beginning, they would not have that last-moment surprise, because they would work on proper sized dataset from the beginning.

Well, two things there -

First, the case you describe is one that's covered by using Spark for batch processing. Parent was criticizing using it and other tools for stream processing. The two are very different use cases.

Second, just gotta call out that 2nd paragraph. A qualified data scientist should have a solid training in statistics. And someone who has a solid training in statistics should rarely if ever make the assumption that one day is representative of the whole week in the first place, regardless of whether they subsequently check that assumption. They would probably start with the presumption that one day is not representative of the whole week, because that is self-evidently going to be near-universally the case for any data that measures something about human behavior.

What's the easiest way to get started with trying Apache Kafka/Spark/Flink on the cloud? If I want to try out Redis there's RedisLabs, CloudAMQP for RabbitMQ, Compose for Postgres/Redis/RabbitMQ, offerings like Google Cloud SQL/MemoryStore and AWS RDS/ElastiCache, etc. Where do I go for some easy Apache deployments?
Google cloud has "one-click" installation of integrated 3rd-party solutions. Never tried that though.
AWS lets you uplod Flink programs to process to Kinesis streams, and Google Cloud also has support for Apache Beam
Yeah, or any other input. I don't think it's tied explicitly to Kinesis. This is definitely easier than other ways to deploy! EMR also has Flink as an option.
"Prerequisites for building Flink: Unix-like environment (we use Linux, Mac OS X, Cygwin) Java 8 (Java 9 and 10 are not yet supported)"

Sigh is too much to ask for proper crossplatform support? And when the hell they will add support for recent versions of Java????

Pretty much all of these Hadoop-adjacent things are more or less Linux only, and certainly unix-y-thing-only. What other platform do you want to run it on?
Uhh, I don't know... maybe windows?

heck, hadoop itself runs fine on windows 10 [http://hadoop.apache.org/docs/current/hadoop-project-dist/ha...]

Make a PR
I wish people would stop making this predictable and tedious comment that adds nothing to the conversation.
"Make a PR" actually means "This seems not to be relevant enough for the current maintainers to do it, and if it feels important to you, you should write it yourself and post a patch", which is a good answer.
Does anyone have experience running in production?

If you have an article about using it or slides please send links.

Not sure why the Flink github site got linked. Here's a list of companies using it in production, with links to details.

https://flink.apache.org/poweredby.html

> Does anyone have experience running in production?

We do:

https://klaviyo.tech/scaling-klaviyos-real-time-analytics-sy...

Yes. No public material though, sorry.
Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing. I feel like this is a bit overboard. And this is before we talk about the non-Apache stream-processing frameworks out there.

* Apache Flink is an open source stream processing framework

* Apache Flume is a distributed, reliable, and available software for efficiently collecting, aggregating, and moving large amounts of log data.

* Apache Storm is a distributed stream processing computation framework

* Apache Samza is an open-source near-realtime, asynchronous computational framework for stream processing

* Apache Spark is an open-source distributed general-purpose cluster-computing framework.

* Apache Apex is a YARN-native platform that unifies stream and batch processing.

* Apache Kafka is an open-source stream-processing software platform

> Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing.

Well, no, you went too far.

Kafka is primarily used for communication & data transport, by most people (can be used in other ways, and it has the KafkaStreams library that enables you do to some computation on said data - but it is, primarily, a transport & communication mechanism; also maybe storage, if you squint right)

Spark and Flink might be similar on first sight, but if you look a bit closer you realize Spark is primarily geared towards batch workloads, and Flink towards realtime. Sure, you can do micro-batch in Spark and pretend that's realtime stream processing, but the focus of it is fairly clear - as is the focus of Flink. So both have legitimate rights to exist.

I'm not sure about the others, haven't used them. There may be indeed consistent overlap - but I'm sure they are different approaches. What's wrong with that?

On the contrary, I didn't go far enough. I didn't talk about Apache Gearpump, NiFi, Beam, Ignite, or Trident.

I know there are subtle differences in each specific technology, and that's probably the same justifications used to support the thesis when building yet another very similar framework.

> What's wrong with that?

I believe it drastically reduces adoption of these tools because many of us avoid what appears to be bandwagon technologies as we don't want to consciously add layers of future technical debt when a majority of these projects will be abandoned.

In the case of Spark and Flink, I wouldn't say that batch processing versus realtime stream processing are "subtle differences". That's akin to arguing that relational databases vs. document stores vs. timeseries databases just "muddy the waters".

Hacker News and Reddit have a lot of interesting discussion. But the audience skews toward client-side webdev, and students or younger developers. An audience accustomed to libraries and frameworks that you can reason about with fairly low learning curve, and spin up in a Codepen to see visually right away.

Heavy-lifting server side tools, especially those who only earn their keep at scale, are a different beast. And that's OKAY. Quite frankly, if you're "not sure" whether you need a stream processing platform in your architecture, then YOU DON'T. Aside from some consultants and salespeople, no one's really going to push you toward adoption of this stuff.

In the overwhelming majority of use cases, what you need is a tiny microservice (in your language of choice). Which reads from a Kafka or Rabbit topic, and stores state in your cache system of choice. By the time you reach the scale where that's not suitable, your organization probably won't need a web forum thread to educate you on what the vendor landscape looks like.

To be a bit more future-proof you should give Apache Beam a try. The same code should (theoretically) work with any of the supported runners[0] and so you could deploy it on top of the most suitable framework/technology for your specific workload. Moreover, at this point, the community has several examples of how to add an additional runner.

Edit: I didn't see you mentioned Beam in your second salvo :)

[0]: https://beam.apache.org/documentation/runners/capability-mat...

>I believe it drastically reduces adoption of these tools because many of us avoid what appears to be bandwagon technologies as we don't want to consciously add layers of future technical debt when a majority of these projects will be abandoned.

Doesn't seem to be impacting adoption of the major ones from what I can see (Spark, Kafka, Flink). Assuming you're at a scale where you need these technologies. Which most companies are not so it's a good thing they don't implement them just for kicks or as "future proofing." If you have a business problem that the main frameworks can't solve then trying out some of the other ones may be worth the cost. There's also ongoing support for older frameworks (Storm) so it's not like your code becomes useless.

The approach open source is taking to resolve this issue is to embrace the diversity but create unified APIs on top of it. Apache Beam and Arrow for defining data workflows and data exchange formats respectively. There's also always SQL which works with a lot of the more data warehouse solutions out there (with some tweaks per solution unfortunately).

> I believe it drastically reduces adoption of these tools

I really don't see why. From afar they might appear to perform the same task but once you take a closer look you quickly realize they follow fundamentally different architectures and have significantly deployment and performance characteristics.

Just because there are all kinds of hammers, but there are also plenty of different uses, and even in the subset involving driving nails there are significant requirements.

I said "you went to far" to claim Kafka, Spark and Flink do basically the same thing. It's reasonable to use all 3 of them in the same team - so they clearly don't do the same thing.

> I believe it drastically reduces adoption of these tools

As does any competition. Tons of smartphone makers = less adoption for any one of them - and many will close down. Still not a bad thing.

Not quite. All of these are open-source projects but they are very different in many aspects:

* Apache Flink

Sophisticated stream processing framework with focus on robustness (managed memory) and correctness (exactly-once semantics)

* Apache Flume

Tailored towards log data.

* Apache Storm

First stream processing framework. Legacy.

* Apache Samza

Only used at LinkedIn. Tight to Hadoop's YARN.

* Apache Spark

Only great in batch processing.

* Apache Apex

Dead project. Tight to Hadoop's YARN.

* Apache Kafka

A distributed message queue with simple stream processing built on top via the Confluent Platform.

Nice list. There are also non-Apache projects like NATS, hosted by CNCF.

Here are the 19 streaming and messaging projects and products that CNCF is tracking: https://landscape.cncf.io/category=streaming-messaging&forma...

> Only used at LinkedIn.

And Intuit, Uber, Netflix, VMWare, ....

https://cwiki.apache.org/confluence/display/SAMZA/Powered+By

As a frequent user of almost half the programs you've listed... I couldn't disagree more. They're completely different.

I'm surprised to see this level of misunderstanding posted with such confidence.

It makes you wonder about some of the other insightful sounding top comments on topics that you aren't as well versed in
In the beginning, there was Hadoop, which was just a MapReduce clone, more or less.

And then there was Hadoop 2, which was kind of meant to do everything.

And that didn't really work out, so there was specialisation.

I don't really see the problem. Many of these do different things, others do similar things different ways. Some are essentially dead ends (I doubt there are many new users of Storm, say). This all seems pretty normal.

No Hadoop does more than MapReduce, much more. Primarily: MapReduce does not have a distributed file-like block storage.
... I am using Apache Pulsar to replace current Kafka setup * Apache Pulsar is an open-source distributed pub-sub messaging system
What compelled you to replace Kafka with Pulsar?
> Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing.

Yes, conceptually they are very similar. If youu want something radically new then check out Bistro Streams: https://github.com/asavinov/bistro

Solving problems of the internet behemoths -> creating opportunity to be acquired.
> Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing.

That's like saying C++, Java, C#, Scala, Python, Clojure... all do basically the same thing

isn't this just a sign of a still young field (data science)? maybe we'll soon start to enter a consolidation phase.