back
74 comments
https://github.com/coralblocks/CoralRing/blob/1168b047e0183c...

Am I missing something here or does the BlockingRingConsumer not actually block? And worse doesn't it just return garbage if poll is called without first checking availableToPoll?

The example sure looks like it... https://github.com/coralblocks/CoralRing/blob/main/src/main/...

Which if so isn't this like 1/4th a library for doing IPC? It doesn't seem to do much itself

availableToPoll() is the method that blocks, because it can return 0 on an empty ring. When that happens you have to block, by for example busy spinning around availableToPoll() until it returns something to poll.

You are never supposed to call poll() without first calling availableToPoll(). How can you poll something if you don't know if there is something available to poll? This is very different than a ConcurrentLinkedQueue where you can call poll() on an empty queue and get a null to indicate that the queue is empty. Also because the ring is a circular queue, you have to know what you can safely poll before polling. That's all done through availableToPoll().

NOTE: By garbage here I don't think you are talking about GC garbage.

> How can you poll something if you don't know if there is something available to poll

counterexample: you can call ::poll(2) if there isn't anything to available. It will either block or return as desired. That's literally what I would expect a poll function to do: check if something is available; c.f. busy-polling.

Maybe it is a language difference?

Not a language difference, but a design difference. By calling availableToPoll before polling you can poll a batch of messages without blocking or locking on each message. This design decision allows the ring to perform batching naturally. And batching is very important for perfomance. java.util.concurrent.ConcurrentLinkedQueue does not support batching.
> availableToPoll() is the method that blocks, because it can return 0 on an empty ring.

Which means it doesn't block...

Blocking means the call itself doesn't return until there's data available. This is a completely non-blocking API. Which is fine, it's just very wrongly named in that case.

But that also means I don't really know why there's both "non-blocking" and "blocking" variants at that point if blocking isn't an option at all in the first place.

You are going to block/wait yourself when availableToPoll() (now changed to availableToFetch() for clarity) returns 0. When that happens you can block/wait by busy spinning or by using a wait strategy. The blocking term means that the producer has to block/wait when the ring is full and the consumer has to block/wait when the ring is empty.

For the non-blocking ring, the producer never blocks on a full ring. It overwrites the circular ring. The consumer can still block on an empty ring.

So to make it clear:

Blocking ring => producer and consumer can block/wait

Non-Blocking ring => producer never blocks and consumer can still block on an empty ring. Consumer can also fall behind too much and disconnect.

It looks like these are more lower level libraries so clients are supposed to block, but they are responsible for doing the blocking themselves.

And given these are designed for "ultra-low-latency" systems, I don't think that's a big problem because the best blocking strategy is probably just to spin.

It would be nice if the docs were nicer though, considering this is a paid product...

> It looks like these are more lower level libraries so clients are supposed to block, but they are responsible for doing the blocking themselves.

You busy spin when blocking (fastest) or you can use a WaitStrategy from https://www.github.com/coralblocks/CoralQueue. You can see an example here: https://github.com/coralblocks/CoralQueue/blob/main/src/main...

> And given these are designed for "ultra-low-latency" systems, I don't think that's a big problem because the best blocking strategy is probably just to spin.

If you have an isolated and dedicated CPU core for your thread, that's correct. Busy-spinning is the fastest/best strategy.

> It would be nice if the docs were nicer though, considering this is a paid product...

CoralRing and CoralQueue are open-source and free at GitHub. There are also a lot of documentation and explanations on the GitHub README.me page (front page). The code has also a lot of comments.

Oof, I saw "CoralQueue" on the product list on your website and I just immediately assumed it was a paid product. But I didn't even think of looking at the LICENSE files, my bad!

Regarding the docs, I was only looking at CoralRing, and it looks like CoralQueue has some additional documentation that applies to CoralRing as well. After reading through them everything makes a lot more sense.

Weird to see this here! I've used CoralBlocks in the low-latency trading domain previously. Highly recommend. The API is kind, they're very responsive, and the latency is exceptional (and comes with all the basics like thread pinning built-in for convenience)
How does it compare to LMAX Disruptor if you have any experience with both?
They're both similar in design, the main difference is Coral Queue can be used for IPC between JVMs, using a mmap'd file.
See https://github.com/real-logic/aeron (also from the creator of the disruptor)
> CoralBlocks in the low-latency trading domain previously.

Yeah, modern JVM is a true miracle and you can be x5 productive (and safe!) compared to C/C++

Do you have any recommendations for a low latency work queue (with in a jvm)?

I want to spawn millions of micro-second-tasks per second, to worker cores..

I am on a massive cache CPU so memory latency hasnt raised its ugly head yet

EDIT: not LMAX please...

I think you can take a look at the DiamondQueue, which is a Demultiplexer and a Multiplexer combined so that thread A dispatches a bunch of tasks to a fixed set of worker threads (W1, W2, W3, etc.) and then these worker threads use a Multiplexer to deliver the results to another thread B. Thread A and Thread B can also be the same thread. There is an example here => https://www.coralblocks.com/index.php/the-diamond-queue-demu...

The DiamondQueue should be soon available for free at the CoralQueue project on GitHub.

You may find https://jctools.github.io/JCTools/ interesting
>any recommendations for a low latency work queue (with in a jvm)?

I toyed around the ring buffer pattern a decade ago, creating a unicast one (using CAS on entries, and eventually a logarithmic scan for next readable entry, not to brute-force-scan them all), but I'm not sure that its latency is much better than that of a regular ThreadPoolExecutor (the throughput could be better though).

Latency also depends on whether it spins or blocks when waiting for a slot to read or write.

If you want to give it a try: https://github.com/jeffhain/jodk/blob/master/src/net/jodk/th...

You were interested in the diamond queue, so just wanted to say that it is now released in our GitHub => https://github.com/coralblocks/CoralQueue?tab=readme-ov-file...
Given the documentation says that this is supposedly to be between JVMs, how do they handle the serialize/deserialize?
They punt on the actual serialization format: https://www.coralblocks.com/index.php/inter-process-communic...

In most applications like this you'll see direct byte manipulation to byte buffers because you want to pull as much performance as possible.

There are fast serialization formats like SBE that people leverage for this as well.

> Given the documentation says that this is supposedly to be between JVMs, how do they handle the serialize/deserialize?

Your transfer object needs to implement MemorySerializable. Below two examples from CoralRing's GitHub:

https://github.com/coralblocks/CoralRing/blob/main/src/main/...

https://github.com/coralblocks/CoralRing/blob/main/src/main/...

The second one effectively allows you to send anything you want (as bytes) through the ring, making CoralRing message agnostic.

Related: https://github.com/pcdv/jocket

Drop-in replacement for java.net.Socket using shared memory (and optionally, futex for notification)

it's sad how comfy the world for spawning request is versus when you are on the receiving side.
I don’t get it. How is this advantageous as it’s limited to one machine? Why wouldn’t you just have one jvm running multiple threads? What is the point of having multiple jvm processes interacting through this ring? Can someone enlighten me?
A few potential reasons for this design coming to mind:

- Resource allocation; you might want to give just specific amount of memory, CPU, network I/O to specific modules of a system, which is not really feasible within a single JVM

- Resource isolation; e.g. a memory leak in one module of the system will affect just that specific JVM instance but not others (similar to why browsers run tabs in multiple processes);

- Upgrades; you can put a new version of one module of the system into place without impacting the others; while the JVM does support this via dynamic classloading (as e.g. used in OSGi or Layrry, https://github.com/moditect/layrry), this becomes complex quickly, you can create classloader leaks, etc.

- Security; You might have (3rd-party) modules you want to keep isolated from the memory, data, config, etc. of other modules; in particular with the removal of the security manager, OS-enforced process isolation is the way to

Also software design. You can split jvm into those that have to follow strict parameters (eg no allocations) and those that follow more traditional Java patterns.
Yeah, I think the HFT guys use CPU pinning a lot: 1 process - 1 CPU, so you'd need multiple processes to take advantage of multicores server.
JVM does garbage collection, this can stop all threads at safepoints while GC occurs.

Those stops can be enough to ruin your low latency requirements in the high percentiles. A common strategy is to divide workloads between jvms so that you meet the requirement.

CoralRing and CoralQueue (available on GitHub) are completely garbage-free. You can send billions of messages without ever creating garbage for the GC, so no GC overhead. This is paramount for real-time ultra-low-latency systems developed in Java. You can read more about it here => https://www.coralblocks.com/index.php/java-development-witho...
Interesting. But how do you ensure a worker that picks up a task does not pause on gc as well?
I can throw some guesses: 1) apps deployed in separate Docker containers due to organization's tech team separation, 2) apps that require security/performance isolation among tenants, 3) isolation layer around memory-leaky and bug-prone third-party library code.
Does anyone have suggestions for something like this, but in Golang?
This is fascinating. I have no idea what something like this would be used for though... what are the use cases?
Basically if you want to schedule workers on a separate JVM, but don't want to pay the latency cost of something like a DB-backed queue or a library with some FFI component.
Super cool, nice work!
Our kafka isn’t reliable enough. I need to write data on disk before flushing it to kafka. Can I use this lis to write data to disk and then consume inside same jvm. I need data to live through restarts
Can you explain what your issue is with Kafka? What makes it not reliable enough?
Kafka is good. The problem is we don’t have a dedicated person to manage it so sometimes we have kafka outages
You could but its not purpose built for that. You'd probably be happier using some other memory mapped file format for that.
Could you please suggest java library for this if you know one?
Write your data to a MognoDB database. Then use the Kafka Mongo connector to pull data from MongoDB to Kafka.
ultra-low-latency and java in the same sentence... masochists?
Not at all. The constraints for ultra low latency are the same whether you use C++ or Java: No blocking IO, no dynamic memory allocation. Using the right libraries such as this one, Java can do a surprisingly decent job while keeping the code more accessible to a layman developer than if it was C++. You still need C++ for applications where maximum mechanical sympathy is required (cache alignment guarantees, etc.) but if you have low latency / medium volume Java often does the trick.
Well said and I couldn't agree more. There are top market makers and banks using Java for a fact. And other C++ firms as well. Some of them are considering or have considered the move to Java. Some have already done this move. Some will never do. I'm certain that it is trivial for a C++ programmer to code in Java. The opposite of course is not true. The whole point of Java as a language is to be higher-level than C++. I don't know if people have realized, but with GraalVM it is now possible to compile Java code entirely to native code ahead-of-time, like it is done with C++. Even before GraalVM there was already the -Xcomp option to force JIT compilation in the very first pass. However that does not necessarily mean that AOT is always preferable over JIT. It is not. Runtime profiling information so you can determine the critical path (hot spots) is amazing for some optimizations, such as aggressive inlining.
I do it better, I free myself from as much as I can of toxic SDK (c++ and java SDKs are filthy toxic).

Assembly, worst case scenario plain and simple C99+.

dynamic allocation is fine as long as it is under control, as it's just a pointer bump

what you never want is to trigger the GC at a bad time

(you can do cache alignment too relatively easily)

LOL you have a point. But this is subjective. Someone could argue that a highly complex C++ system is masochism. The reality is: a lot of successful market makers, prop trading firms, banks, etc. use Java for ultra-low-latency financial systems. And of course some successful companies use C++ as well.
Numbers aren't subjective. High-performance is a measurable claim and shouldn't be entangled with language-complexity matters.