▲ 262 points
back
163 comments
For those asking "who uses PyPy?": the truth is that very few people using it get back to us and say how it helps them or what they are doing with it. For instance crossbar.io uses it [0]. Some shops use it as a second step until they refactor code into a compiled language, but often that second step never materializes.
But the value of a second implementation of a language goes beyond the immediate "who uses this in practice". It can be a fertile bed for innovation and for new ideas, and provides a contrast to the nay sayers. For instance, the recent pitch to vastly improve CPython's speed has some roots in ideas that were tested out in PyPy. CFFI [1], revdb [2] and vmprof [3] all started as PyPy projects. Some of these turned out to be very popular, some less so. The next project in this line is HPy [4] (still alpha-quality), which is trying to re-think the C-API for Python to make it even easier to interface with.
RPython [5], the language behind PyPy, is also an accessible playground for dynamic language research.
[0] https://crossbar.io/about/FAQ/#python-runtime
[1] https://cffi.readthedocs.io/en/latest/
[2] https://morepypy.blogspot.com/2016/07/reverse-debugging-for-...
[3] https://vmprof.readthedocs.io/en/latest/
If you want more examples of real world use cases, PyPy is pretty stress-tested by the competitive programming community already.
https://codeforces.com/contests has around 20-30k participants per contest, with contests happening roughly twice a week. I would say around 10% of them use python, with the vast majority choosing pypy over cpython.
I would guesstimate at least 100k lines of pypy is written per week just from these contests. This covers virtually every textbook algorithm you can think of and were automatically graded for correctness/speed/memory. Note that there's no special time multiplier for choosing a slower language, so if you're not within 2x the speed of the equivalent C++, your solution won't pass! (hence the popularity of pypy over cpython)
The sheer volume of advanced algorithms executed in pypy gives me huge amount of confidence in it. There was only one instance where I remember a contestant running into a bug with the jit, but it was fixed within a few days after being reported: https://codeforces.com/blog/entry/82329?#comment-693711 https://foss.heptapod.net/pypy/pypy/-/issues/3297.
> If you want more examples of real world use cases, PyPy is pretty stress-tested by the competitive programming community already.
I think this is the first time I've seen someone suggest that competitive programming has much bearing on “real-world use cases”.
Why do competitive Python programmers use PyPy instead of CPython?
AFAIK pypy improvements also led to cpython compact dictionaries (which led to dictionaries being ordered) and many many cpython performance improvements (the most recent one being LOAD_ATTR caching, with huge performance gains).
I wonder if sponsoring pypy instead of the PSF might be a way for people that want "a focus on performance improvements, instead of fancy features" to vote with their wallets
It is worth noting that none of the PSF donations arrive at core development anyway. They support the bureaucracy and the infrastructure.
python.tar.gz would be exactly the same if the donations were zero.
All the more reason to switch to donating PyPy.
To add I believe PyPy also inspired asm.js the precursor to Web Assembly.
I have turned to pypy quite extensively for pure-Python text processing tasks, where I often get a 10-100x speed up just by changing the command line invocation. For example, I wrote a proof of concept Rabin-Karp hashing approximate string matching algorithm to perform plagiarism analysis while I was working at Udacity in around 2017. The system never went into production, but pypy was super helpful in crunching all historical user submissions for analysis.
I’ve also had great success using pypy to accelerate preprocessing steps (when they don’t rely on incompatible c libraries) for machine learning pipelines. It’s the most painless performance enhancement trick in my toolbox before I reach for concurrency (in which case I reach for joblib or Dask).
The one oddity I’ve noticed is that using tuples (including things like named tuples) often speeds up CPython by a lot, but even plain tuples can slow down pypy on the same code—in some cases pypy winds up slower than CPython.
In any case, I’m low key in love with pypy, even though I can’t use it for _everything_.
> The one oddity I’ve noticed is that using tuples (including things like named tuples) often speeds up CPython by a lot, but even plain tuples can slow down pypy on the same code—in some cases pypy winds up slower than CPython.
I'd love for this to be addressed. namedtuple should be lowered into a fixed sized struct, great opportunity for PyPy to show wins in both memory, memory bandwidth and compute. Add in type stability and PyPy should approach native speeds just by adding in namedtuple.
We use pypy pretty extensively at our firm for analytics/olap work.
We've actually tried using some of the more traditional libs (Pandas et al) with CPython, but there's always a pure-python bottleneck (e.g. SQLAlchemy).
Performance is important to our clients and trying to keep everything performance critical in C extensions / NumPy would be kind of risky for us when adding new functionality, so pypy's guarantee of more speed pretty much across the board is awesome.
There are downsides of course - higher memory usage, longer boot times, some more obscure libraries being unsupported - but on the whole, it's a good choice for us
Why isn't Ubuntu shipping it as the default python?
A couple of years ago we reached out to the python community about wheels and arm64 - how it should be handled and whether they plan on embedding non-x86 blobs.
We received the standard "we'll think about it and let you know".
Now that Apple switched to arm64, all communities are suddenly interested in porting things to arm64.
And of course, Apple is not investing in these ports, at least as far as I know. They just rely on what other arm64 players did in the ecosystem before Apples rolled out M1; respectively lets developers figure out the remaining porting.
As much as I hate to say this, IBM does handle porting things to ppc64 right - you can find IBM contributed code and optimizations anywhere you look. For many packages, porting to arm64 was a matter of "does it have ppc64 support? if so, it can be reused for arm64" ...
Disclaimer - used to be a contractor porting stuff to arm64 for a couple of years.
> And of course, Apple is not investing in these ports
What count as "investing in these ports"? Does submitting patches to CPython (and many other open source projects, including NumPy) for macOS 11 and Apple Silicon count? Here's a list of Apple-submitted PRs on python/cpython: https://github.com/python/cpython/pulls?q=is%3Apr+author%3Al... Also some co-authored patches excluded by the search. See https://bugs.python.org/issue41100 for more related PRs. Your subsequent comments about IBM seem to imply that these do count.
I'm also curious about who's "we" in "we reached out to the python community about wheels and arm64".
---
Edit: Forgot to say, arm wheels have been supported for many years now (not sure about the specific timeline of aarch64 support, but if 32-bit arm was supported I don't see why aarch64 wouldn't be). Maybe most famously there's https://www.piwheels.org/ for RPis. Are you talking about aarch64 support on PyPI/warehouse?
For what it's worth, I think Apple sent the Homebrew devs ARM Mac Minis for building and when the Homebrew devs asked for more, they sent more. So they're doing something, even if they're not doing as much as one would hope.
Why would Apple invest in third-party toolchains? Aren't they incentivized to only support Xcode, Swift, and other tools that are as tightly bound to their ecosystem as possible?
They can't. They've put themselves on the path where they can't abide by the GPL. GPL3 says you must preserve the right to run the software and apple takes that away from their customers then sells it back.
So apple nopes out of it. They rely on the initial boost they got from GPL software. They do some MIT/BSD licensed stuff (and publish some of it but not everything). Every year less and less happens, unless it's their own language/etc.
They do have nice gradients and round corners though. :)
Amazon AWS will soon add support for the M1 hardware to the macOS EC2 managed offering announced a few weeks ago. AWS also offers free credits to prominent Open Source projects.
It's not available yet but it should be a good option once they become available, you can immediately start the process of applying for the credits.
This way you would get access to the required hardware without any costs and without having to maintain it. They are designed from the ground up towards such CI/development use cases.
You can also use the same credits for getting access to Intel and ARM Linux, as well as Windows EC2 instances. This may improve your project's CI and build times.
Disclaimer: I work at AWS.
This is no use unless you reduce the minimum tenancy. Might as well buy a Mac Mini and shove it on a shelf.
PyPy volunteers are being too nice asking for a couple of thousand dollars. Don't be afraid to ask for enough money to support development and maintenance for this new platform, purchase the hardware, as well as earning replacement income for opportunity loss. $50K minimum is more reasonable. Get the funding you deserve from these free riding companies using the software.
This seems like the kind of thing that Apple themselves are in a good position to help with. Seems pretty improbable unfortunately.
I imagine that PyPy is way too obscure for Apple to care about it. Amazon would probably care a lot more if they're planning on adopting more ARM in AWS.
Also: the people or organizations making these requests. They know making software isn't a free lunch.
> Either we get sufficient money, and maybe support, and then we can do it quickly; or PyPy will just remain not natively available on M1 hardware for the next 3-5 years
Out of curiosity, what is a ballpark, a rough estimate of the "sufficient money" for a project like this?
I too became frustrated reading the article and trying to figure out what they were asking for.
Protip: patrons and sponsors do not want to know all the details, and certainly not up-front. What they want is to know what you're asking for, what you offer in return and then (briefly) how one leads to the other. Thus:
We make Pypy, which does X and is used by Y people, including projects such as Z, z, and z'. We'd love to bring Pypy to Apple's exciting new M1 hardware, but that needs resources.
We're looking for $CASH_MONEY to buy some M1 hardware that we can develop, test against, and eventually run the repository on. With #NUM_MACHINES available, we believe was can have PyPy available to all M1 developers by %CALENDAR_MONTH. We also like eating and drinking and if you send us more money we will spend some of it on !GOOD_TIME(HAVING).
The whole approach of 'we're not asking for much, just x would help' is how volunteers talk to each other, minimizing their ask so that reorienting effort is not too burdensome for anyone. But how you talk to each other is not a good way to talk to external readers, because it requires effort to parse what you are getting at and then further effort to negotiate and arrange everything which is a distraction from their own concerns. Many people would rather just give you some cash and sign up for a monthly progress update.
There's a reason services Kickstarter etc. work so well: they save the donors time and entanglement. Just set up : PyPy on Mac M1 for $5000?' or similar and make it easy for people to throw money at you.
From the rest of the blog post it seems enough to buy a single M1 machine with 8+ GB of RAM, so my estimate is that it should be a bit over $1k for a basic Mac Mini + shipping + Apple Care.
I wish I could contribute more generally, but I think the PyPy project is a great cause and will put down a small amount to help with this.
I've come to the realization that, to get more performance out of python, the most pythonic way is to simply use PyPy, rather than try and hack your way around issues like attributes requiring dict lookups, access to locals() being faster than globals(), etc.
> and will put down a small amount to help with this
I'd politely ask you to reconsider. You'd basically be subsidising work that benefits the World's highest-capitalised company, to whom the investment in this port would be a fraction of a second's profit.
I've never been able to find a good use case for pypy. Why would you ever want to run CPU bound tasks in python? I'm sure there are arguments about the huge ecosystem, not having to rewrite code in another language, etc. But most of the widely used stuff in python already has underlying compiled code for the heavy lifting. Also, given the large amount of parallelism in modern hardware architecture, and python's lackluster concurrency support, I just don't see a reason for using it.
I used Luigi [1] to automate data processing at a previous job. It's a simple job queue with a UI. You request jobs from it, and then run them for minutes or hours, so it shouldn't normally be a bottleneck and it makes sense to use a language that's quick and easy to write.
It's written in Python and works fine to process thousands of jobs per day. Once you start having tens of thousands of jobs in the queue, it gets slow enough that it can back things up. This compounds the problem, eventually resulting in the whole thing crashing.
By switching the interpreter to PyPy, I was able to keep the data pipeline running at that scale without having to rewrite anything.
> Why would you ever want to run CPU bound tasks in python? ... But most of the widely used stuff in python already has underlying compiled code for the heavy lifting.
Haven't you just answered your own question? We know that people want to run CPU bound tasks in Python so much that they went to the effort of writing native modules because they couldn't do it in Python.
> python's lackluster concurrency support
This is a common misconception - Python actually has fully concurrent threads already.
A well-tuned web application layer is CPU bound at scale - your database is well designed and not a source of latency, and the lack of concurrency support in a language doesn’t matter if the interpreter is so much slower than context switches, which is absolutely true for CPython.
There are many sites and services where a rewrite in a new language is just not viable, and I still would recommend Python-everywhere to startups doing things remotely associated with data. So PyPy would be a tide that would lift many boats.
> Why would you ever want to run CPU bound tasks in python?
0. Because you don't have time to deal with the mess that C++ has become, and the amount of please-repeat-yourself-a-million-times crap you have to deal with (cmake files, header files, they give us a goddamn spaceship operator but not basic necessities like string split/join methods)
1. There are many use cases where faster execution time is nice to have, e.g. when results can be cached for a long time, or if it's a one-off data analysis script, but human time is far more expensive. If it costs $1000 more in engineer hours to write C++ instead of Python for that script that's only going to be run 10 times, that isn't a worthwhile tradeoff. Hell you could buy a new GPU for that money.
2. Because the same exact file can be deployed on arm32, arm64, and x86
3. Because CPU-intensive stuff is largely already optimized by numpy, numba, tensorflow, pytorch, etc.
I can either spend a lot of time rewriting my already tested and working code if my application scales to the point of hitting a CPU bottleneck OR I can just try using PyPy.
I've done it for one-off data munging scripts - processing archives of one format of data to another, etc. Python is easy to write and for these tasks, you can get PyPy to execute at twice the speed for basically no cost.
FWIW, you don't have to be faster than the bear.
Strictly speaking, "CPU bound" is not an adjective that can be used to describe tasks. It's one that describes a particular program solving a particular task under a particular configuration. I've done no analysis on this myself, but I would be more than willing to believe that a CPU-bound job might be only a CPython-to-MyPy's worth of speedup away from instead being memory- or disk-bound.
It can be hard to tell the difference, too, since being stalled out while waiting on the memory controller shows up as CPU activity in htop.
Killing efforts like PyPy is what creates the Julias and Chapels of the future.
As others mentioned, it's generally an easy choice to write one-off data processing & analysis stuff in. I can get all the multithreading support I need with the multiprocessing library.
For me this always just consists of reading a bunch of files in and then doing some basic aggregation on them. Years ago I benchmarked Python vs. PyPy and found no real benefit to it.
Here is the link to that if you'd like to read it:
Code can end up being CPU bound that wasn’t meant to be. For example, having to use on a database driver or some other low level library written in Python can make your web app pretty CPU-bound. The main issue with PyPy is compatibility for me.
I guess they're not asking for individual donations in this post, but still, the "donations page" link in the sidebar 404s. It appears from https://www.pypy.org/howtohelp.html that https://opencollective.com/pypy is the correct link.
How much money and how do we contribute?
I'd add GRPC to the wishlist too: https://github.com/grpc/grpc/issues/4221
So they just need $900?
Apple should pay some of their vast horde of cash.
Apple?
Who uses PyPy in production?
Who uses Apple Silicon in production?
There are enough engineers posting here from well paid FAANG roles to fund this effort...