Item — HN Reader

I absolutely love Polars, and I use the Rust crate all the time!

I think my only gripe is that the Rust API seems (to me at least) to be less well documented than the Python API. I guess Python is the de facto data science language, so maybe that explains it.

4 comments

Likewise, I would prefer to stay in Rust when developing command line apps that might do DataFrame thingies and would love more examples of how to transform DataFrames in pure Rust.

I have written several rust/python apis now and I think what you’re feeling is the result of the api being designed for Python as well as being primarily tested in Python.

No matter the target, I test rust things using python. idk. Food for thought.

Curious why you use the Rust API over the Python API? I'm just wondering if you've also explored the datafusion crate, because I think on the rust side it's about equivalently ergonomic to use either and Datafusion seems a little more modular/pluggable. The Polars python API is definitely miles above the datafusion python API.

Oh, I actually had not yet heard of the DataFusion crate! Thanks for sharing!

And to answer your question, we use the Rust API because all of our backend services are in Rust, and we like to stay in Rust whenever possible.

With GPT4 helping with the refactor, there's no reason to start migrating code away from Pandas imo. A lot of people say they think Pandas is fast enough for their needs, but you're literally getting a 95% speed improvement for free.

This is a huge difference in productivity, especially when running code and doing a lot of slicing in notebooks.

Polars is an immense project, and I hope it continues to gain traction. But there's lots more factors than just speed.

The main one in my team is ubiquity- i.e. lots of people know pandas, who might not be traditional "developers". I.e. data scientists, data analysts etc. Having a data scientist put together some code, it gets optimized by an engineer, and they can talk back and forth about the same code is a massive benefit.

Shifting to polars (and keeping that ability to collaborate) would require not just training the engineers to use a new framework, but all the analysts, data scientists etc that they are adjescant to. That's a huge business cost, and in a lot of cases it might be worth it. But I wouldn't describe it as "getting 95% speed increase for free".

While that's fair, it's fairly easy to fit it in only the most intensive operations and then seamlessly convert back to a pandas data frame.

I understand why you wouldn’t do this on an organizational level for production workflows, but for personal workflows in my opinion, it’s a no-brainer to incrementally learn and adopt it.

Where ive seen resistance to polars in Python land its been either "Pandas is already used/standard and people understand/know it" or "If you really need speed, you can probably do it in numpy which should be faster again".

I think you meant to say there's no reason not to start migrating? Otherwise I can't parse how the rest of your post matches up with your conclusion.

What does GPT-4 have to do with this?

It's easier than ever to do a drop-in replacement for data workflows. The whole decision making process between migrating libraries like this is how much time investment is it going to take and how much is it going to pay off?

Not a good example - polars is new and has changed so much in the past couple of years that GPT-4 often gives outdated code for it.

It's still pretty good posting documentation as context or using phind.