Step-by-step instructions for the work you'll do most. If you're loading data for the first time, start with Ingest, then Querying; read Data modeling before you create a table you'll keep, and the other guides for file formats, updates, transactions and tuning.
- Data modeling: Choose sorted, partitioned or vector-indexed layouts, projections and row IDs from your queries, since most of these choices can't be undone.
- Ingest: Insert Arrow data with the Python SDK, size batches, import Parquet files on the cluster, load partitioned tables, or use Trino.
- Load JSON, CSV and Parquet files: Load newline-delimited JSON, CSV and Parquet files with PyArrow, fix the types it guesses wrong, and stream large files in batches.
- Load scientific array data (NetCDF and GRIB2): Turn NetCDF and GRIB2 arrays into table rows with xarray, load step by step or in parallel, and query a region.
- Querying: Make the cluster do the filtering: choose columns, write predicates that push down, limit rows, tune QueryConfig and read splits in parallel.
- Updates and deletes: Update and delete rows by row ID with the Python SDK, on sorted, partitioned and user-defined row ID tables, and delete partitions.
- Transactions in practice: Commit and roll back in Python SDK code, recognise conflict errors, know what the SDK retries, and retry a whole transaction yourself.
- Optimize queries: A checklist for cheaper queries: table structure, filters that push down, fewer columns, splits and endpoints, top-N and vector search.
- Indexing and data skipping: With no B-tree indexes, see how min/max statistics, sorting, partition pruning, projections and the vector index skip data, and what each costs.
- Projections: What a semi-sorted projection is, when a second sort order helps filters on other columns, and how to create and check one.
- Profile a query: Use the Query Engine, Trino, the SDK and Spark to see whether filters pushed down, what came back and where time went.
- Event sourcing and CQRS: Build an append-only event table and a current-state table with the Python SDK, using idempotent writes, a watermark and rebuild by replay.