Item — HN Reader

Yep deadlocks, you're correct. From the paper:

"Modern Unix systems offer named pipes, also known as fifos, which can be used to hand-craft arbitrary process communication topologies. However, if combined one-to-many and many-to-one piping are setup by using named pipes, another problem will occur. Due to the limited buffering offered by typical programs, deadlocks can easily occur when a process consuming data from many producers with more than one input, blocks waiting for input from one of the processes feeding it. This can cause a second feeding process to block, waiting to send its output to another one of the consumer process’s inputs, and, thereby, blocking the upstream process feeding both processes that provide data to the consumer one."

dspinellis has commented on another dgsh discussion¹(along with you). Interestingly, with a light comparison to pipexec².

I stumbled upon pipexec trying to find a battle tested solution to extend a data munging task where I was relying on zsh's multios³, mostly because orchestrating the interactions with a coproc'd jq for output were fighting me. There is something both frustrating and soothing about finding a seven year old comment pointing out why my path was doomed before I'd even started; people have solved the problem already, plus people far smarter than me also found the trap.

¹ https://news.ycombinator.com/item?id=13352659

² https://news.ycombinator.com/item?id=13358090

³ https://zsh.sourceforge.io/Doc/Release/Redirection.html#Mult...

1 comments

Oh wow, I had stumbled on the zsh behavior of

    echo hi > *.txt  # writes ALL of the files!

No other shell does that.

But I didn't know it was called MULTIOS until now. (I guess that's read "mult I/O's"? I have a hard time not reading it as "multi-OS" :) )

It seems a bit niche to be honest, but it's possible to support in Oils.

---

Oils also uses Unix domain sockets already for the headless shell protocol

https://github.com/oilshell/oil/wiki/Headless-Mode

We could do something like dgsh, but so far I haven't seen a lot of uptake / demand. Every time it's mentioned, somebody kinda wants it, and then it kinda peters out again ... still possible though.

I think flat files work fine for a lot of use cases, and once you add streaming, you also want monitoring, more control over backpressure/queue sizes, etc.

I use mutlios and even I'm not that attached to it. The majority of my use is combined with process substitution, and could be replaced with common-ish tools like pee¹(or pipexec for more complex cases). The only occasion when I'm thankful for it is if I want to use a shell function as a target, but there are workarounds for that too.

As a noclobber user the footgun is largely hidden to me, but I feel its presence. multios without globbing support would be less useful, but would still work for most of my use cases. Scanning my shell history I see various cases of relying on zsh's ability to apply sorting and filtering to globs with multios' input redirection, but only a couple where I want that in output redirection. The input instances could easily be rewritten using cat and globbing.

Even with multios unset the behaviour is different between zsh and bash. For example, nomultios disables all the expansion, so zsh behaves like more like dash with ': >t{1,2}' creating a file instead producing an error like bash does.

[FWIW, I google'd multiios to link the option in mt original comment. It really feels like it needs double-i, and I read the single i name the same way you do.]

---

I'd be one of those people whose desire for dgsh-like functionality wanes. If it was slight DSL that I could "upgrade" pipelines to I'd probably use it, but not enough to warrant working on it or switching other tooling to support it.

The end of result of this morning's pipeline was breaking my jobs up, and applying some judicious use of nq² to keep track of it. I'd follow your advice and move on to more specialist tools if the job grew significantly or if it became a regular occurrence.

¹ https://joeyh.name/code/moreutils/

² https://git.vuxu.org/nq/about/