Store and read Parquet

These examples use the events table from Load and query JSON.

Store Variant in Parquet

You can write Variant columns to Parquet like any other column with the normal DataFusion COPY statement:

COPY events TO 'events.parquet' STORED AS PARQUET;

The Variant column is stored in Parquet following the Parquet Variant encoding specification as a struct with metadata and value fields. You can see the structure using the parquet_metadata table function:

-- Note: parquet_metadata is part of dfv and datafusion-cli, not this crate.
SELECT path_in_schema, type FROM parquet_metadata('events.parquet');

+------------------+------------+
| path_in_schema   | type       |
+------------------+------------+
| "event.metadata" | BYTE_ARRAY |
| "event.value"    | BYTE_ARRAY |
+------------------+------------+

Write shredded Parquet columns

You can choose which fields to store as typed Parquet columns with the variant.shredding option. For example, to store the service and status fields of the event column as typed Parquet columns, run:

COPY events TO 'shredded_events.parquet' STORED AS PARQUET
OPTIONS ('variant.shredding' '{"event": "STRUCT<service VARCHAR, status BIGINT>"}');

See the shredding guide for the full format of the variant.shredding option.

Read Variant from Parquet

You can read events.parquet back with the Variant column preserved. The same query works with shredded_events.parquet.

SELECT variant_get(event, '$.service', 'VARCHAR') AS service,
       event:tags[0]::VARCHAR AS first_tag -- Array indexing via `[]` operator
FROM 'events.parquet'
ORDER BY event:status::BIGINT;

+---------+-----------+
| service | first_tag |
+---------+-----------+
| api     | cached    |
| api     | NULL      |
| api     | NULL      |
+---------+-----------+