Applies to VAST Cluster 5.5. Examples use the VAST DataBase Python SDK 2.1.0.
This page lists the size and count limits of VAST DataBase tables in VAST Cluster 5.5, and which table structures you can combine: sorted tables, partitioned tables, semi-sorted projections, user-defined row IDs, vector indexes and replication.
Sizes
| Limit | Value |
|---|---|
| Cell size, including metadata | 5 MB |
| Cell size for fixed-size binary columns | 64 KB |
| Cell size in a table with a semi-sorted projection | The 5 MB limit doesn't apply. Once a table has a projection, every cell in every column is limited to about 126 KB, including metadata, as before 5.5. Larger inserts fail with SemiSortedIsNotSupportedWithLargeCells, and create_projection() fails on a table that already holds larger cells. Dropping the projection lifts the limit. |
| Statement size | 5 MB |
| SDK request size | A query request (predicate and column list) may use up to about 90% of 5 MiB. Larger ones raise TooLargeRequest. |
| User-defined row ID value | At most 2^47. See User-defined row IDs. |
Counts
| Limit | Value |
|---|---|
| Sort columns per table | Up to 4, and one sort order per table. A 5-column sorting key is rejected. |
| Partition keys per table | Up to 4. The SDK raises ValueError itself for more than 4. See Database partitioning. |
| Rows before sorting takes effect | Tables are sorted in chunks of 512K rows, so smaller tables aren't effectively sorted. Sorting works best above 1 million rows. See Sorted tables. |
Cluster-wide maximums (how many databases, tables, columns or snapshots a cluster can hold) are in the VAST Cluster Scale Guidelines, which aren't published on the KB (see 5.5.0 release notes and, for the maximum number of snapshots, snapshots). Admin task: send to your platform team. Ask for the Scale Guidelines values for your release.
What can't be changed later
| Structure | Fixed at |
|---|---|
| Sorting | Once set, sorting can't be removed, and the sort columns and their order can't be changed. Values in sort columns can't be updated. See Sorted tables. |
| Partitioning | Defined when the table is created; it can't be removed or changed. Partition columns can't be dropped, and their values can't be changed after insert, unless the update leaves the rows in the same partition. See Database partitioning. |
| User-defined row IDs | If you supply row IDs in an insert, every later insert has to supply them too. See User-defined row IDs. |
Combining table structures
| Combination | Allowed? | Details |
|---|---|---|
| Sorted + partitioned | Yes, at creation only | Set both in create_table(). You can't add sorting to an existing partitioned table. A column partitioned with the identity transform can't also be a sort column. A column partitioned with another transform, for example day(ts), can be. See Database partitioning. |
| Sorted + semi-sorted projection | No | In either order: you can't add a projection to a sorted table, or sort a table that has a projection. See Sorted tables. |
| Sorted + user-defined row IDs | No | In either order. See Sorted tables. |
| Sorted + vector index | No | |
| Sorted + replication | No | You can't replicate a sorted table, or sort a replicated table. See Sorted tables. |
| Partitioned + semi-sorted projection | No | |
| Partitioned + user-defined row IDs | No | |
| Partitioned + vector index | No | See Database partitioning. |
| Partitioned + replication | No | See Database partitioning. |
| Partitioned + quotas | Not supported | Quotas on partitioned tables aren't supported. Check with your platform team before you rely on a quota for a partitioned table. |
| Semi-sorted projection + user-defined row IDs | No | You can't add a projection to a table that uses user-defined row IDs. |
| Semi-sorted projection + vector index | Yes | A projection can be added to a vector-indexed table. |
| User-defined row IDs + vector index | No | |
| Vector index + replication | No | You can't add a vector index to a table in a replicated path, or replicate a vector-indexed table. The error says "replication is currently not supported for vector indexed tables". |
| Semi-sorted projection + replication | Partly | The table replicates, but its semi-sorted projections aren't replicated. See Overview of VAST Replication. |
| User-defined row IDs + replication | Not stated |
Plan the table structure before you create the table: most of these choices can't be changed later. Data modeling explains how to choose.
Errors when a combination isn't allowed
The cluster rejects the request when you try a combination that isn't allowed. These are the errors you'll see:
| Combination | Error |
|---|---|
| Sorted, partitioned or projection combinations | BadRequest (400), code InvalidArgument, message Invalid Argument |
| Any combination with user-defined row IDs | BadRequest (400), code TabularRowAllocMethodConflict, message There is a conflict in tabular row allocation method (with a vector index: failed to enable vector index eres=106) |
| Vector index with sorting or partitioning, at creation | NotImplemented (501), code NotImplemented, message vector indexing cannot be combined with elysium sorting or ... with partitioning |
The InvalidArgument message doesn't say what's wrong. If you get it from create_table(), add_sorting_key() or create_projection(), check the table above first. "Elysium" in the message means sorted tables.
Partition transforms
create_table() accepted each of these PyIceberg transforms in a partition spec: identity, year, month, day, hour, bucket and truncate.