Two questions:
- If I have some data in Kafka, why would I want to pump it into your platform instead of spawning an Elasticsearch instance and using something like Kafka Connect to write to it and gain visibility?
- If I use Kafka as a permanent data store (with infinite retention), I can easily replay all events with existing clients (or with plumber). What additional functionality does the "replay" feature offer compared to that?
> - If I have some data in Kafka, why would I want to pump it into your platform instead of spawning an Elasticsearch instance and using something like Kafka Connect to write to it and gain visibility?
To avoid having to build, own and maintain the infra you just mentioned. As the number of events on your system increase, you will have to scale ES and other pieces of the system as well.
Our point is just that - if you know what's involved in collecting and indexing the events - that is awesome but maybe you shouldn't have to spend time building the infra around that stuff.
> If I use Kafka as a permanent data store (with infinite retention), I can easily replay all events with existing clients (or with plumber). What additional functionality does the "replay" feature offer compared to that?
I think it depends on your definition of "easily replay" - a kafka replay for a topic that's being consumed by a consumer group would require you to disconnect that consumer group and then run a shell script to move the offsets. You also would not have any way to replay any specific messages - your only point of reference would be an offset (and keyname, if you use it) - not terribly flexible.
With Batch, you get to drill in and replay the _exact_ messages you want (and avoid having to pump and dump potentially millions of messages your consumer doesn't care about).