Welcome to the new DOCS.vastdata.com. We've added new menus and a version selector, plus new persona and resource pages to help customers get started. Looking for a previous article? Use the docs search.

Press CTRL+K to open search

Release notes

Prev Next

Applies to VAST Cluster 5.5.

This page lists the changes in each VAST Cluster 5.5 release that affect people who build on VAST DataBase: new things you can do, fixes, and known limitations. It's a digest of the full VAST Cluster 5.5 release notes, which also cover storage, networking and administration.

To find out which release your cluster runs, see find your versions. For which SDK and connector versions go with each release, see version compatibility.

5.5.0

What you can now do

Partitioned tables. You can partition a table by up to four columns or column transforms, such as day(event_time), and also give it up to four sort columns. Queries skip partitions that can't match, and you can delete a whole partition at once. Supported operations are creating a partitioned table, listing its partitions, inserting, importing Parquet files, querying and deleting a partition. You can create partitioned tables from the Python SDK, Trino and Spark. Deleting partitions is controlled by the s3:TabularDeletePartitions permission. See data modeling.

The VAST Query Engine doesn't work with partitioned tables in 5.5; use the Python SDK, Trino or Spark.

Vector-indexed tables. You can create a table with a vector index on one vector column, using Euclidean or cosine distance, for similarity search. Newly inserted vectors can be queried straight away. See VAST Vector Search and, for how vector tables combine with other table types, limits.

Blob expansion. Events published to an event topic can be expanded from JSON into rows of a VAST DataBase table, in the same transaction as the event. You query the result like any other table. It needs the s3:TabularCreateBlobExpansion, s3:TabularAlterBlobExpansion, s3:TabularDropBlobExpansion and s3:TabularGetBlobExpansion permissions. See blob expansion.

Larger cells. The maximum cell size, including metadata, went up from 126 KB to 5 MB. Fixed-size binary columns are limited to 64 KB per cell, and the maximum statement size is 5 MB. The new limit doesn't apply to semi-sorted projections. See limits.

More SQL in the VAST Query Engine.

  • DDL: CREATE SCHEMA, CREATE TABLE (including a sort key), DROP TABLE, DESCRIBE, ALTER TABLE to rename a table, rename, add or drop a column, or add a sort key, and CREATE VIEW / DROP VIEW. There are no commands for projections or partitions.
  • DML: INSERT ... VALUES, INSERT ... SELECT, insert from a Parquet file, UPDATE and DELETE.
  • Aggregate functions such as SUM, MIN, MAX and AVG, including the FILTER clause. DISTINCT, ORDER BY and GROUP BY aren't supported with aggregate functions.
  • Sorted tables are supported without caveats.

See the SQL reference.

Trino 478. The VAST connector for Trino supports Trino 478. See version compatibility.

Spark. Enabling table sorting from Spark, and row and column security for Spark, are listed as new in 5.5.0 (5.5.0 release notes). Neither is confirmed; see Spark.

Sorting status for small tables. Before 5.5, the sorting status of a table with fewer than 5 million rows was reported as a constant. That limitation is gone.

Fixed

  • An invalid input to a VAST DataBase operation could restart a CNode container. Fixed.

Known limitations

These are listed in the 5.5.0 release notes and repeated in SP2, SP3 and SP6 unless noted.

Partitioned tables

  • You can't partition on nested columns (list, map, struct).
  • Partition columns can only be updated if the update doesn't move rows to another partition, and they can't be dropped.
  • Partitioning can't be combined with vector indexing, semi-sorted projections or user-defined row IDs.
  • A partitioned table can be sorted, but only if the sort order is set when the table is created.
  • The same column can't be used for both sorting and partitioning.
  • Quotas and replication aren't supported on partitioned tables.

Sorted tables

  • Sorting can't be removed, and the sort columns and their order can't be changed. Values in sort columns can't be updated.
  • Up to four sort columns and one sort order per table.
  • Tables with fewer than 512K rows don't get sorted, although you can enable sorting on them.
  • Sorted tables can't be replicated, and replicated tables can't be sorted.
  • Sorting can't be combined with semi-sorted projections, nested-type sort columns, or exposing the internal row ID through the vastdb_rowid column.

Vector-indexed tables

  • The index can't be removed, and its distance function can't be changed. Only one vector column per table can be indexed, and it can't be called vector.
  • You can't add an index to an existing table, or add or drop columns in an indexed table.
  • Vector indexing can't be combined with sorted or partitioned tables.
  • The VAST connectors for Trino and Spark don't support vector search.

Views (table views)

  • View properties aren't supported, and queries in a view must use full table names.
  • Only Spark can redefine a view. If a redefined view's query changes schema, user-defined column names and comments are lost.
  • The Trino connector doesn't support the SECURITY clause of CREATE VIEW.

Row and column security

  • Row filtering and column masking are listed as supported for Trino only.
  • The Trino user's identity policy must allow end-user impersonation and getting row and column security; the action names are s3:EndUserImpersonation and s3:TabularGetRowColumnSecurity. See security.
  • With protocol auditing on, impersonated access is logged with the original user's UID only, not their login name or SID.

VAST Query Engine

  • The map type isn't supported.
  • A virtual IP pool with the Query Engine role can't be modified after it's created. Your platform team has to delete it and create a new one, so expect the Query Engine address to change if they do.

Ingest

  • Importing into a table that's both sorted and partitioned takes considerably longer than importing into a table that's neither.
  • Blob expansion: one expansion table per event-topic column; the target table must exist (without columns) first; target columns used by the expansion can't be renamed or removed; the source table can't be renamed; updates and deletes on the source don't reach the target.

Replication and other features

  • Asynchronous replication of VAST DataBase replicates only committed transactions. VAST Catalog and the audit log aren't replicated, and multi-tenant replication isn't supported. See snapshots and replication.
  • Global Access doesn't support VAST DataBase views.
  • A view with the Database protocol can only be created on a cluster with data reduction enabled. Fixed in SP6.

Trino as a managed application on the cluster

  • A Trino cluster needs at least three CNodes: one coordinator and two or more workers.
  • Queries may fail during a cluster upgrade, and work again when it's complete.

5.5.0-SP2

No new VAST DataBase features or fixes. The known limitations are the same as 5.5.0. (5.5.0-SP2 release notes)

5.5.0-SP3

No new VAST DataBase features or fixes. The known limitations are the same as 5.5.0. (5.5.0-SP3 release notes)

5.5.0-SP6

Fixed: a view with the Database protocol can now be created on a cluster with data reduction disabled. Before SP6 this failed with DR_IS_NOT_ACTIVE view_create returned an error: ObjectResultCode: INVALID_PARAM, and the limitation is gone from the SP6 list. Other known limitations are the same as 5.5.0. (5.5.0-SP6 release notes)

See also

Related articles