Introduction#

This crate provides a Variant SQL type and functions for querying semi-structured data using Apache DataFusion, and reading and writing such data to Apache Parquet files.

Note

This crate is under active development and aims to match Apache Spark Variant behavior. Compatibility is incomplete, and APIs may change. See the roadmap for more details.

The source code is available on GitHub, the crate is published on crates.io, and the Rust API reference is on docs.rs.

What is Variant?#

Variant is a type for storing semi-structured data, similar to JSON. Each value can hold objects, arrays, and primitive values, with different fields and types in each row.

For example, a single Variant column can contain all of these values, shown as JSON:

Value

Type

{"service":"api","status":200}

Object

["cached","retry"]

Array

42

Integer

"api"

String

Variants are stored using an efficient binary encoding, similar to BSON, with optimizations for repeated field names. For more background, see Variant Type in Apache Parquet for Semi-Structured Data.

This crate also supports the Variant "shredding" optimization for when many rows share a common structure. You specify some subset of the fields to store as typed Parquet columns, and the remaining fields are stored in the standard Variant binary representation. Queries on the common fields use the typed columns for better performance, while preserving all other fields in Variant values. See Write shredded Parquet columns and the shredding guide for more details.

References#