---
title: "Read path and pruning"
slug: "dev-database-concepts-architecture-read-path-and-pruning"
description: "Learn how query filters impact performance across tables, with insights on splits, sub-splits, and the importance of data order for efficient querying."
updated: 2026-10-30T18:19:01Z
published: 2026-10-30T18:19:01Z
canonical: "docs.vastdata.com/dev-database-concepts-architecture-read-path-and-pruning"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Read path and pruning

*Applies to VAST Cluster 5.5. Examples use the VAST DataBase Python SDK 2.1.0.*

This page explains what the cluster does when you query a table, and why the same filter can be cheap on one table and expensive on another. For the practical decisions that follow from this, see [data modeling](/develop/docs/dev-database-guides-data-modeling) and [querying](/develop/docs/dev-database-guides-querying).

A query reads a table where it's stored, in the cluster's [Element Store](/develop/docs/dev-database-concepts-architecture-cluster-architecture#one-element-store). VAST DataBase scans the table's column chunks there, reads only the columns the query uses, and uses each chunk's minimum and maximum to skip chunks that can't match the filter. How much it can skip depends on how the table is laid out in the Element Store ([how tables are stored](/develop/docs/dev-database-concepts-architecture-element-store-layout)): which rows end up next to each other in the same chunks.

## Where a query reads from

![A database is a bucket on a view with the Database protocol; a schema is stored like a directory and a table like a file, so the view's permissions apply. The table is stored as row groups, and within each row group each column is stored separately in small column chunks, each with its minimum and maximum. New rows land in the write buffer (storage-class memory) and background work moves them to flash, into this layout. The chunks sit on the cluster's shared storage, which every CNode reaches, so the splits of one query run on several CNodes at once and each reads only the columns it needs, skipping chunks whose minimum and maximum can't match.](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/es-table-in-store.svg?sv=2026-02-06&spr=https&st=2026-11-08T08%3A55%3A37Z&se=2026-11-08T09%3A55%3A37Z&sr=c&sp=r&sig=mpYzqDlZXKKfZNXtlxdyokF%2BToyZPAuivGpZBl0xugQ%3D)

*A table in the Element Store: where it lives, how it's laid out, where it's stored.*

- **From shared storage, through any CNode.** A table's column chunks and their statistics are on the cluster's shared storage, which every CNode reaches. No CNode owns part of a table, so a query's splits can run on different CNodes and each reads the chunks it needs directly ([shared everything](/develop/docs/dev-database-concepts-architecture-cluster-architecture#shared-everything)).
- **Column by column.** Within each row group, each column is stored in its own chunks, so a query reads only the chunks of the columns it uses ([how tables are stored](/develop/docs/dev-database-concepts-architecture-element-store-layout#rows-are-grouped-and-each-column-is-stored-separately)).
- **Chunk by chunk.** Each chunk keeps its minimum and maximum, so a filter can skip a whole chunk without reading it. It can only skip where matching rows sit together in the same chunks, which is what the rest of this page is about.
- **Including rows just written.** Committed rows can be queried while they're still in the write buffer, before background work has moved them into the layout ([write path](/develop/docs/dev-database-concepts-architecture-write-path)).

## How a query is divided

![The Python SDK divides a query into splits and sends them to different endpoints, so several CNodes work in parallel. Each CNode divides its split into sub-splits that run on its worker threads. Each sub-split first selects the matching rows, using per-chunk minimum and maximum values to skip chunks, then reads only the requested columns for those rows. Results stream back to your client as Arrow record batches.](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/query-flow.svg?sv=2026-02-06&spr=https&st=2026-11-08T08%3A55%3A37Z&se=2026-11-08T09%3A55%3A37Z&sr=c&sp=r&sig=mpYzqDlZXKKfZNXtlxdyokF%2BToyZPAuivGpZBl0xugQ%3D)

*Splits spread a query across CNodes; inside each, sub-splits select rows and then read only the columns you asked for.*

A query from the Python SDK is sent as several **splits**, each its own stream of requests. The SDK sends different splits to different data endpoints (VIPs), so the work is spread across CNodes ([querying](/develop/docs/dev-database-guides-querying#how-a-query-runs)).

On the CNode that receives a split's request, the cluster divides the split again into **sub-splits** and runs them in parallel on that CNode's worker threads. Sub-splits add parallelism; they don't reduce how much data is read.

A split isn't one contiguous range of rows. The table's rows are dealt out to the splits and sub-splits in chunks, in turn, so every split gets chunks from across the whole table. A small table can fit in a single chunk, so one split holds every row and the others return nothing.

The cluster keeps no state for a query between requests. Each response carries a continuation token for each sub-split, which the SDK sends back to get the next batch.

## Two phases: select rows, then read columns

Each sub-split works in two phases:

1. **Select.** The cluster evaluates your filter, column by column, and builds the set of matching rows.
2. **Read.** It reads only the columns you asked for, and only for the matching rows.

Columns you neither filter on nor ask for are never read. If you ask for no columns at all, the cluster only counts matching rows. Both are listed in [supported pushdowns](/docs-5-5/docs/supported-pushdowns) (projection pushdown, `COUNT(*)`).

With several conditions joined by AND, the cluster stops early: once one column's condition rules out every row in a chunk, the other filtered columns aren't read for that chunk.

## Skipping data with min/max statistics

The cluster keeps a small summary for each chunk of each column: its minimum value, its maximum value, and whether it contains nulls. Before reading a chunk, it checks your conditions against the summary:

| The summary shows the condition is... | What the cluster does |
| --- | --- |
| False for every row in the chunk | Skips the chunk without reading it |
| True for every row in the chunk | Keeps the rows without reading the chunk |
| True for some rows | Reads the chunk and evaluates the condition |

This works on every table type, not just sorted ones. Summaries exist for numbers, decimals, strings, booleans, dates, times and timestamps, but not for nested types (lists, maps and structs).

For strings, the summary keeps only the start of each value. Values that share a long prefix, such as timestamps stored as strings, all look alike in the summary, so it can't skip anything for them. Store timestamps as a timestamp type.

### Why data order decides the cost

![The same filter, temperature greater than 35, against four chunks. On the left the rows are in no order for temperature, so every chunk's minimum-to-maximum range includes values above 35 and every chunk has to be read. On the right the rows are sorted by temperature, so three chunks' ranges are below 35 and are skipped; only one chunk is read.](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/data-order.svg?sv=2026-02-06&spr=https&st=2026-11-08T08%3A55%3A37Z&se=2026-11-08T09%3A55%3A37Z&sr=c&sp=r&sig=mpYzqDlZXKKfZNXtlxdyokF%2BToyZPAuivGpZBl0xugQ%3D)

*The same filter reads every chunk when the rows are in no order, and one chunk when they're sorted by the filtered column.*

A chunk can be skipped only when its whole minimum-to-maximum range falls outside your condition. If the table's rows aren't in any order for the column you filter on, nearly every chunk holds a wide spread of values, so nearly every chunk has to be read, even when only a few rows match. A selective filter returns few rows but can still read the whole column.

When rows are ordered by the filtered column, each chunk covers a narrow range, and most chunks can be skipped. That's what sorted tables and projections give you.

## Sorted tables

A [sorted table](/develop/docs/dev-database-guides-data-modeling#sorted-tables) keeps its rows in **sorted runs**, plus an area of recently inserted rows that haven't been sorted yet. The cluster keeps an index of the runs, with each run's range of values for every sort column.

When you filter on sort columns, the cluster first uses that index to pick the runs that can match, and skips the rest without reading them. Inside the runs it keeps, rows are sorted, so the min/max summaries are narrow and skip most chunks too. Recently inserted rows that haven't been sorted yet are always read.

What this means for you:

- **Filter on the sort columns.** Only conditions on sort columns are used to pick runs.
- **Avoid OR with a non-sort column.** If an OR includes a column that isn't a sort column, no runs can be skipped.
- **Expect more work right after a big load.** New rows are read in full until background sorting includes them. See [sort progress](/develop/docs/dev-database-operations-monitoring-your-workload#sort-progress).

The Python SDK also asks the cluster how many rows the matching runs hold, and uses that to choose how many splits to run (SDK 2.1.0).

## Semi-sorted projections

A [semi-sorted projection](/develop/docs/dev-database-guides-data-modeling#semi-sorted-projections) isn't sorted across the whole table. The table is divided into blocks of rows (the sorting chunk size, 512K rows), and each block's rows are sorted within the projection's matching block. That's what "semi-sorted" means. A query that uses a projection still visits every block, but inside each block the min/max summaries can skip most chunks. If the projection hasn't sorted a block yet, that block is read from the table instead.

The cluster uses a projection only if all of these are true. The [KB](/docs-5-5/docs/managing-table-projections) says projections help "queries that use only the columns included in the projection"; the other conditions come from the server.

- The projection holds every column you filter on **and** every column you ask for.
- The query has a filter that's an AND of conditions, each on one column. An OR of values on one column is fine; an OR across columns isn't.
- You didn't ask for row IDs (`internal_row_id=True`).
- Your transaction hasn't written anything yet. After a write, reads in the same transaction use the table, so they see your own changes.
- Projections are allowed for the query. In the Python SDK that's `QueryConfig.use_semi_sorted_projections`, on by default.

When several projections qualify, the cluster prefers the one whose first sort column you filter on.

## Partitioned tables

Each partition of a [partitioned table](/develop/docs/dev-database-guides-data-modeling#partitioned-tables) is queried separately. The client first queries the table's partitions view to find the partitions that match the filter, then sends one query per partition, so non-matching partitions are never read. This is the partition pruning the [KB](/docs-5-5/docs/database-partitioning) describes. The Python SDK does it for you.

## Filters the cluster can't evaluate

The cluster evaluates only filters of one shape: conditions joined by AND, where each condition is an OR over a single column. A condition can be a comparison, a range, a null or NaN check, or a string match. It rejects anything else instead of evaluating part of it. Any remaining filtering has to happen in the client or the query engine; see [supported pushdowns](/docs-5-5/docs/supported-pushdowns). The Python SDK checks the shape before it sends a query; see [rewrite filters that don't push down](/develop/docs/dev-database-guides-querying#rewrite-filters-that-dont-push-down).

## What `limit_rows` does

`limit_rows` limits each sub-split, not the whole query: each one stops once it has found that many matching rows. The SDK collects the results, keeps the number you asked for, and stops the rest. With a rare value, each sub-split keeps scanning until it finds enough rows or reaches the end, so a small limit doesn't make a search for a rare value cheap.

## Long scans can return empty responses

On a long scan, a sub-split periodically returns even if it hasn't found anything, then carries on from where it stopped. So with a very selective filter on unordered data, you may see empty batches while the scan continues. The SDK handles this for you.

## What this means for your queries

| You do this | What happens |
| --- | --- |
| Filter on a column whose values are in no particular order | Few chunks can be skipped, so the column is read for nearly every row |
| Filter on a sorted table's sort columns | Whole runs are skipped, then most chunks inside the rest |
| Filter on several columns with AND | Columns after the first are read only where rows are still in the running |
| Store timestamps or numbers as strings | The summaries can't tell values apart, so nothing is skipped |
| Raise `num_splits` or `num_sub_splits` | More parallel work; the same amount of data read |
| Write, then read, in one transaction | Projections aren't used for the read |

## See also

- [Table structures](/develop/docs/dev-database-concepts-architecture-table-structures): what each layout is, buys and costs.
- [Querying](/develop/docs/dev-database-guides-querying): writing filters that push down.
- [Pushdowns](/develop/docs/dev-database-reference-pushdowns): what each engine sends to the cluster.