back

by pdet·9h ago·view on hn ↗
The main reason I decided to use a beefier machine is that it gives me flexibility when benchmarking, without the need to set up different environments. The CSV data, for example, is >80 GB. We can also “scale down the machine” for experiments where we want to stress-test lower-memory scenarios or use fewer threads by configuring DuckDB’s settings (e.g., SET memory_limit = '10GB'; or SET threads TO 1;).

(Disclaimer: I’m the author of the blog post.)

1 comments
Note that SET memory_limit is a soft limit and can be completely ignored for some tasks, so this wouldn’t be the same as having a limit on physical memory.
In past experience with DuckDB I have found this soft-limit distinction to be very important.