Item — HN Reader

I wrote a parallel image processing tool (in Swift) that essentially uses Grand Central Dispatch on macOS to process all files in a directory.

I hit the problem that GCD would happily let me enqueue tasks for processing, but each task would need quite an important amount of memory and GPU resources (because it would use CoreFilters that support GPU acceleration "when available"), which made the computer hit swap and slow to a crawl once the available RAM was exhausted.

I had to create an Admission Control component that would let me enqueue tasks only if there was enough memory left to process the image without risking swap, and that part isn't as trivial as it seems, as getting the amount of available memory for a process is a bit vague on UNIX/macOS/Linux since many OSes of this kind assume you can allocate an infinite amount of memory (see linux's overcommit system) and that the Virtual Memory system will do the right thing for you. Also nobody can guess if physical RAM is going to be allocated by some other process somewhere, so I used a fairly conservative heuristic that probably left some small amount of resources unused most of the time, but would almost never slow to a crawl. Not perfect, but "good enough".

4 comments

It seems you have a producer-consumer problem, where feeding in work to do faster than it can be processed only leads to lengthy queues, and running more tasks in parallel than you have cores only increases memory usage and task switching overhead.

Such processes rarely are memory-bound, and from what you write (it would use CoreFilters that support GPU acceleration "when available"), yours aren’t either. So why would you want to limit the number of consumers by looking at memory usage?

ioquatix’ comment to measure latency is better, but more work, so my first go would be to manage queue length, as that is simpler to do and often effective enough.

Determine how many consumer processes your system can really handle in parallel and limit the number of queued GCD tasks to, say, twice that number. That factor of two depends on how bursty the producer produces work items and on how CPU-bound the consumers are.

If memory of I/O aren’t your bottleneck, you want to minimize the number of items ‘in flight’ needed to keep your CPU or GPU at 100% usage.

> Such processes rarely are memory-bound, and from what you write (it would use CoreFilters that support GPU acceleration "when available"), yours aren’t either. So why would you want to limit the number of consumers by looking at memory usage?

In my case it was clearly memory bound. I demonstrated it through instrumentation and profiling, then addressed it by using an admission control scheme exclusively built on memory.

If you can build a system which measures the latency of task processing and dynamically adjusts the resource allocation, I think you can be robust in the face of changing underlying conditions. It's non-trivial, but easier to look for spikes in latency than trying to estimate resource allocation/utilisation.

Right - we need holistic scheduling systems that balance all resources and tasks. For example scheduling an IO heavy-task alongside a compute-heavy task and letting them both run well, rather than picking two compute-heavy tasks at the same time, and only scheduling as many tasks as there is RAM, IO, and other resource capacity for to use productively. Lots of people are trying because at data-centre scale and with serverless that lets you run code wherever they want this could be a huge advantage... not sure how much success yet.

I wonder if something as simple as tagging would give a huge benefit towards that. I should know if my job uses lots of memory(often I know exactly how much), or IO, or CPU.

This is something I have pondered for the Bitbake build system. It compiles a Linux "distribution" for embedded systems and uses a lot of CPU, RAM, and I/O. Often one is starving the other. I use a spinning disk for build dirs currently because they use a lot of space. Even with an SSD there is room for better scheduling.

That’s quite an interesting problem. The mlock [1] system call allows you to lock chosen virtual memory into RAM. What about using that in combination with a memory pool which you manage yourself?

[1] https://developer.apple.com/library/archive/documentation/Sy...

If I had all places where memory is allocated under my "jurisdiction", I think this would be a very applicable solution.

Unfortunately when calling APIs to process images, some opaque code that you don't see ends up doing allocations on your behalf, and you can rarely pass your own pool to allocate memory from.

For the record, mlockall on Linux can lock all future allocations, including those made by shared libraries. You don’t need to manage a memory pool yourself.

https://linux.die.net/man/2/mlock

That's very cool!

It looks like that API doesn't exist on macOS, though... :-(

That’s true. I didn’t consider that. I’d be interested love to see a solution to it though. Maybe there’s some sort of OS-level solution, like how you can set the priority of processes.