▲ 133 points
back
27 comments
If you are wondering what PyOxidizer is then here is a good explanation:
https://pyoxidizer.readthedocs.io/en/v0.8.0/overview.html
From that page:
From a very high level, PyOxidizer is a tool for packaging and distributing Python applications. The over-arching goal of PyOxidizer is to make this (often complex) problem space simple so application maintainers can focus on building quality applications instead of toiling with build systems and packaging tools.
On a lower, more technical level, PyOxidizer has a command line tool - pyoxidizer - that is capable of building binaries (executables or libraries) that embed a fully-functional Python interpreter plus Python extensions and modules in a single binary. Binaries produced with PyOxidizer are highly portable and can work on nearly every system without any special requirements like containers, FUSE filesystems, or even temporary directory access. On Linux, PyOxidizer can produce executables that are fully statically linked and don’t even support dynamic loading.
The Oxidizer part of the name comes from Rust: binaries built with PyOxidizer are compiled from Rust and Rust code is responsible for managing the embedded Python interpreter and all its operations. But the existence of Rust should be invisible to many users, much like the fact that CPython (the official Python distribution available from www.python.org) is implemented in C. Rust is simply a tool to achieve an end goal (albeit a rather effective and powerful tool).
So does this use a custom build of CPython linked against a different libc that's part of the package? That has been IME one of the major issues with distributing binary packages of Python apps on Linux, because code built using glibc tends to use recent versions of common symbols, so in practice you can't run them on a system with a glibc older than the build system's glibc, meaning you have to build on some ancient Linux where everything is a pain.
Edit: Sounds like it:
> PyOxidizer uses custom Python distributions that are built in such a way that they are highly portable across machines.
> Specifying --target x86_64-unknown-linux-musl will cause PyOxidizer to use a Python distribution built against musl libc as well as tell Rust to target musl on Linux.
[1] https://pyoxidizer.readthedocs.io/en/stable/packaging_static...
> meaning you have to build on some ancient Linux where everything is a pain.
Well, PyPA provides Docker images for manylinux﹡[1] and manylinux2014 isn’t that ancient, I found it mostly tolerable in my experience of building wheels for various computational packages of my own (depending on gmp, etc.). In fact even manylinux1 and manylinux2010 have been okay for me, but that of course depends on what libraries you need.
The future of manylinux is defined in PEP 600[2], though I’m not sure about what has happened implementation-wise regarding the new scheme.
The page states that a static build is possible so you'll get a binary you can distribute to any machine at that point.
Without any more clarification that doesn't quite answer the question because libc is such a special beast. Typically (but maybe not always) "static build" on Linux really means "all libraries are statically linked except libc which is dynamically linked".
To make something very portable across Linux distributions, rather than statically linking libc you would dynamically link against an old version of it, as newer versions are always backwards compatible with older versions.
I've been using PyOxidizer sind 0.7 for a personal CLI project and I really like it. Apart from being able to ship a single executable, it also speeds up the startup time of my CLI, as it doesn't need to scan loads of Python files during startup.
Really great stuff!
How big are your binaries?
I packaged with a very simple script with a couple of dependencies with 0.7 and it's ~84 Mb: https://github.com/kissgyorgy/cloudflare-dyndns
Ooof that's a lot. PyInstaller is considerably lower than that (~60 MB for a full desktop application including PyQt, NumPy and a plotting toolkit)
A simple CLI with only a few commands and not many dependencies is 56M here.
PyOxidizer generates standalone binaries for python.
It requires the rust (and optionally C) toolchain to be installed first.
I think requiring another language development environment set up to package code is a big turn-off. A friendlier approach is either to have a package in the same language so it interpolates with the ecosystem, or have a standalone binary which doesn't need dependencies.
At this point, I look at it as a python distribution tool for rust developers.
How does this compare to pyinstaller or nuitika?
Nuitka compiles Python to C, so it can make your code run faster, but you're no longer using the tried and tested CPython interpreter so you may run into Nuitka-specific issues (I haven't used it and don't know how common they are, I assume it's usable though)
PyOxidizer and pyinstaller are rather bundlers that build a CPython interpreter and all your dependencies into a single binary. There is also https://github.com/pantsbuild/pex which is a bit like jar files for Java, but it also bundles your code, dependencies and an interpreter into a single executable file.
Nuitka compiles Python to C using the Python C-library (something Cython is also capable of), with a focus on portability. Nuitkas main goal is not speed (compared to Cython), but the ability to output an executable binary (plus a bunch of dynamic libraries) needed to run your program in a portable, "interpreter independent" way. From the "What is Nuitka" page[0]:
> It translates the Python into a C program that then is linked against libpython to execute in the same way as CPython does, in a very compatible way.
Once they managed full compatibility (I think it was in 0.4?) then they started to focus on optimizing and translating even more things to C (again, keeping compatibility at 100% -- so you shouldn't event run into Nuitka-specific issues).
I've used it to deploy a few custom triage programs in the field for DFIR operations where we'd want to run a few specific things from portable drives without installing anything in the triaged computer.
Nuitka is great! I use it to compile some of the command line tools which I have written. Command lines tools compiled with Nuitka have a much better startup time.
Nuitka also has a single file binary target in the works.
Compiled single file binaries for python. Wohoo!
Single file binaries would be great for my use case! It all works fine now, but the instructions would go from "copy all this directory with everything inside, MAKE SURE EVERY FILE IS COPIED, then make sure again" to "copy the file".
Nuitka uses libpython, which is CPython, not some random Nuitka-specific interpreter.
I don't think it's that simple
Nuitka-specific issues are real: https://github.com/Nuitka/Nuitka/issues
My comments aren't meant as a criticism of Nuitka though, I am very glad to see this project existing.
I initially thought it's Python to Rust conversion tool.
while not being a full conversion tool it actually also can function as a bridge between python and rust, from the docs
"While solving packaging and distribution problems is the primary goal of PyOxidizer, a side-effect of solving that problem with Rust is that PyOxidizer can serve as a bridge between these two languages. PyOxidizer can be used to easily add a Python interpreter to any Rust project. But the opposite is also true: PyOxidizer can also be used to add Rust to Python. Using PyOxidizer, you could bootstrap a new Rust project which contains an embedded version of Python and your application. Initially, your project is a few lines of Rust that instantiates a Python interpreter and runs Python code. Over time, functionality could be (re)written in Rust and your previously Python-only project could leverage Rust and its diverse ecosystem. Since PyOxidizer abstracts the Python interpreter away, this could all be invisible to end-users: you could rewrite an application from Python to Rust and people may not even know because they never see a libpython, .py files, etc."
How does it compare to Poetry?
From what I gather, the focus of PyOxidizer is on enabling you to create executable Python binaries which embed a Python interpreter rather than providing Python packaging and dependency management.
Sounds like PyInstaller is the closer comparison
These two are totally different tools, not even comparable. With Poetry, you handle dependencies, just like with npm, cargo or pip itself. PyOxidizer can build a standalone binary which doesn't need an installed Python interpreter, so you need both.
thx, I misunderstood what it does