What Is Polars? – Definition and How It Works

Informational P2 what is polars

Learn what Polars is, how lazy evaluation and query optimisation work, why it uses all cores by default, and where it fits.

Polars is a data frame library implemented in Rust with Python bindings. It is columnar, multi-threaded by default, and offers a lazy execution mode that optimises a whole query before running any of it.

1. What Makes It Different

Multi-threaded by default. Operations use available cores without configuration.
Columnar memory layout, based on the Arrow format.
Lazy mode, where operations build a plan that is optimised before execution.
Strict typing, with less implicit coercion than older libraries.

2. Eager and Lazy

import polars as pl

# eager: each operation runs immediately
df = pl.read_parquet("data.parquet").filter(pl.col("price") > 10)

# lazy: build a plan, optimise, then execute
result = (
    pl.scan_parquet("data/*.parquet")
      .filter(pl.col("price") > 10)
      .group_by("category")
      .agg(pl.col("price").mean())
      .collect()
)

scan_parquet reads nothing until collect. In between, the optimiser can rearrange the work.

3. What the Optimiser Does

Optimisation	Effect
Predicate pushdown	Filters applied while reading, so fewer rows are loaded
Projection pushdown	Only referenced columns are read
Common subexpression elimination	Repeated computation done once
Slice pushdown	A row limit propagates to the source

On partitioned Parquet these combine with file-level pruning, so a filtered query may touch a small fraction of the data — the same principle a query engine applies, inside a data frame API.

4. The Expression API

Operations are expressed as expressions over columns rather than as index manipulation, which composes cleanly and gives the optimiser something to reason about.

df.with_columns(
    (pl.col("price") * pl.col("quantity")).alias("total"),
    pl.col("title").str.strip_chars().str.to_lowercase().alias("key"),
)

5. Where It Fits

Larger datasets on one machine, where an eager in-memory library struggles.
Multi-core workstations, where parallelism is free performance.
Parquet-based pipelines, where scanning and pushdown pay off.
Repeated transformations, where the optimiser removes redundant work.

6. Where Something Else Fits

The older ecosystem has broader third-party integration and far more existing code and documentation. For small data, exploratory work, or a codebase already built on it, switching buys little.

For SQL-shaped analysis over files, an embedded query engine may be a more natural fit than either data frame library.

7. Practical Notes

Use lazy mode for anything non-trivial; it is where the gains are.
Prefer expressions over row-wise operations, which forfeit parallelism.
Strict types surface data problems earlier — treat that as a feature.
Interoperates via Arrow, so exchanging with other tools is cheap.

8. Summary

Polars is a Rust-based columnar data frame library with multi-threaded execution and a lazy mode whose optimiser pushes filters and projections down to the source.

It suits larger-than-comfortable datasets on a single machine and Parquet-based pipelines, where scanning, pushdown, and parallelism together produce a substantial difference in both speed and memory.