Tweaking DuckDB

Note

These recommendations are based on DuckDB’s performance guide

DuckDB is the embedded OLAP database used by Liwan. It is fast out of the box, but you can tune memory and thread limits for smaller servers or unusual workloads.

By default, DuckDB can use up to 80% of system memory and all available CPU cores. That is usually fine, but resource-constrained systems may need lower limits.

Memory Limit

The memory limit of DuckDB can be set using the LIWAN_DUCKDB_MEMORY_LIMIT environment variable or duckdb.memory_limit in the configuration file. At an absolute minimum, DuckDB requires 128MB of memory per thread. For better performance, DuckDB recommends around 5GB per thread. Try lowering the thread count before lowering the memory limit.

Thread Limit

The number of threads DuckDB uses can be set using the LIWAN_DUCKDB_THREADS environment variable or duckdb.threads in the configuration file. By default, DuckDB uses all available CPU cores.

Disk Usage

For large queries that do not fit in memory, DuckDB can spill intermediate results to disk. Use fast local disks and avoid network storage when possible.