back
1 comments
This makes me wonder if named pipes is a solution to a problem I'm having.

I use a 3rd party took with a Python API which only takes filenames (not stdin and not Python file objects). It natively understands gzip and non-gzip files.

I would like it to support zstandard compression.

I think I can set up a named pipe with ["zstdcat", "filename.zstd"] as input, directed to a named pipe, and have the 3rd party toolkit read from that named pipe, as a regular text file.

But I'm a bit confused about the coordination.

1) how do I find a valid file system location? I can use tempfile to get a writable file location, but that might be on a network scratch disk. Is that going to be okay, or do I need some mechanism to be able to specify a local writable filesystem?

2) I can start the zstdcat via subprocess, but how do I know when to wait() on it, given that the 3rd party doesn't (easily) tell me when it's done. (I guess I can put a wrapper around the API, and check with the reader finishes.)

3) How do I manage removing the named pipe? Is there a way to do that automatically when both sides close? Or is there some other mechanism I could do?

I can do all this via wrappers of various sorts, but the coordination seems complex, and I'm not sure what will happen if I ^C in the middle of things.

Has anyone here done something like this before? I don't see any relevant package on PyPI for this sort of wrapping.