Decathlon Switches to Polars to Optimize Data Pipelines and Infrastructure Costs

Decathlon Switches to Polars to Optimize Data Pipelines and Infrastructure Costs

Decathlon, a leading sports retailer, adopted the Polars library to improve data pipeline efficiency and reduce costs. The company observed a reduction in compute launch time from 8 to 2 minutes when switching from Apache Spark to Polars for datasets around 50GB.

Why This Matters

Traditional data engineering often relies on distributed frameworks like Spark, even for smaller datasets, leading to wasted resources and increased costs. Ideal models assume optimal tool selection based on data size, but reality frequently involves using a single, powerful framework for all workloads. Decathlon’s experience highlights the significant cost implications of this mismatch, demonstrating that inefficient infrastructure can hinder agility and inflate operational expenses.

Key Insights

• Polars is built in Rust: leverages Apache Arrow for columnar data processing, improving performance.
• Spark’s overhead: can be substantial for smaller datasets, making Polars a more efficient alternative.
• Medallion Architecture: Decathlon utilizes a Bronze/Silver/Gold/Insight architecture for data refinement and governance.

Practical Applications

• Use Case: Decathlon uses Polars for pipelines processing input tables less than 50GB with stable sizes.
• Pitfall: Introducing Polars adds a new tool to the stack, requiring team training and potentially slowing down data pipeline collaboration.

References:

• https://www.infoq.com/news/2025/12/decathlon-spark-polars/