DataFusion Distributed#

A library that extends Apache DataFusion with distributed capabilities.

The problem#

A single DataFusion node is fast, but it has a ceiling. Large scans, wide joins, and heavy aggregations eventually run out of memory or CPU on one machine. The usual escape routes are painful: build your own distribution layer on top of DataFusion or abandon DataFusion for a heavyweight system that dictates how you deploy and serve queries.

You might want this library if:

  • You want to spread the CPU and memory pressure of an existing DataFusion query across several machines.

  • You want your existing queries to run faster by putting more than one machine behind them.

  • You want to have control over how your custom plans are distributed and assigned to workers.

  • You want to distribute your queries without a framework dictating your deployment and networking setup.

It’s still DataFusion#

A distributed plan is a normal DataFusion physical plan, with a few extra nodes that happen to stream Arrow data between machines in a zero-copy manner using gRPC instead of passing it in memory — everything else is the DataFusion you already know.

You can expect all your existing tooling built on top of DataFusion to work seamlessly with Distributed DataFusion.

Going distributed takes exactly three things:

  1. Enable the distributed planner. Just a one-line addition while building the DataFusion session context.

  2. Tell it where your workers are. Provide a list of worker URLs the distributed planner can use.

  3. Run the worker gRPC servers. Mount them onto a gRPC service you already run, expose them on their own port, or deploy them however suits your infrastructure.

Everything else is opt-in.

Note

This project is a library built on top of Apache DataFusion. It is not part of Apache DataFusion itself.

When to use Apache DataFusion Ballista instead of this library#

Apache DataFusion Ballista is a distributed query engine. It has a scheduler that tracks executors, jobs, and stages; executors with per-task memory pools; clients (Rust, Python, the CLI, and Flight SQL); and operational tooling (REST API, event logs, history server, Kubernetes, and KEDA).

This project is a library intended to be embedded in existing DataFusion-based applications, allowing a system to spread a query over workers it already manages. Like DataFusion, it does not persist intermediate results for recovery: if a query fails mid-execution, it needs to be retried from scratch.

Consider using Ballista instead of this project if:

  • You’d rather use an existing distributed engine as a foundation and customize it to your needs, rather than building a distributed query engine yourself.

  • You are comfortable adopting Ballista’s deployment and networking model.

  • Your workload involves large, long-running batch queries or ETL that benefits from materializing intermediate results between stages.