VAST Cluster 5.5.1 Release Notes

To access this release, contact the VAST Customer Success team via a support ticket, Slack, or by sending an email to support@vastdata.com.

Upgrade to VAST Cluster 5.5.1 is supported from the following releases:

• 5.5.0 - 5.5.0-SP6
• 5.4.6 - 5.4.6-SP2
• 5.4.4 - 5.4.4-SP6
• 5.4.3 - 5.4.3-SP8
• 5.4.2
• 5.4.1 - 5.4.1-SP5
• 5.4.0 - 5.4.0-SP3

Note that direct upgrade may not be supported from hotfix builds. Consult VAST Support regarding upgrade if your cluster is running a hotfix build.

• New Features
• Enhancements
• Resolved Issues
• Limitations
• Known Issues

New Features

This release includes the following features that were first introduced in VAST Cluster 5.4.6 but were not included in VAST Cluster 5.5.0:

• QUALITY OF SERVICE (QOS)
  • S3 Connection Limit per CNode
• PROTOCOL AUDITING
  • Logging Extended Error Information in VAST Audit Log
• AUTHENTICATION AND AUTHORIZATION
  • CA Certificate and VMS IP Binding for SMTP Notifications
• PLATFORM AND CONTROL
  • Monitoring of CNode Liquid Cooling Systems

Enhancements

In addition to enhancements listed below, this release also includes enhancements that were first introduced in VAST Cluster 5.4.6.

Install and Upgrade

• ORION-395660: Updated the Advanced Settings page of the VAST Cluster Install wizard for EBoxes:
  • The following fields are mandatory: Shards, DR Shards, DR WB Shards, Stripe groups.
  • The SCM layout field is mandatory and does not provide the Automatic option.
  • There is a new pane named Partition Drives that requires you to specify if the EBox has partitioned drives (mixed media).

Networking

• ORION-355259: Added a validation performed when removing CNodes from a virtual IP pool to ensure that the remaining number of pool's CNodes does not fall below the required minimum (2 CNodes per pool).

Quality of Service (QoS)

• ORION-323979: The qos_wait_for_budget_time metric is now in microseconds. Prior to this change, nanoseconds were used.

S3

• ORION-368187: Starting with version 5.5.1, each CORS rule can optionally be assigned a unique rule ID string (up to 255 characters).
  • In VAST Web UI, the CORS tab in view settings (Element Store -> Views -> choose to create or edit a view) provides a new field named Rule ID where you can enter an ID for the rule being created.
  • In VAST REST API, the rule ID can be set by using the s3_id field in cors_rules or by using the id field in aws_cors_rules when sending a POST request to the /views/<ID>/s3cors_configuration/ endpoint.
    The rule ID must be unique within the S3 bucket.
• ORION-344013: Added support for S3 requests where the hostname is in the URL but no Host header is provided.

VAST Audit Log

• The VAST Audit Log provides a new field, vip_pool_guid, that shows the GUID of the virtual IP pool used for the request. This field helps track the pool in case it has been renamed or deleted.

• ORION-399003: The VAST Audit Log provides two new fields:

  • TotalTransferredResponseBytes shows the total number of bytes in the cluster's response to an S3 request, including both payload and HTTP overhead.
  • TotalTransferredRequestBytes shows the total number of bytes received per request, including both payload and HTTP overhead.

Event Publishing

• ORION-394324: Made updates to display the following details for a configured S3 event notification:
  • In VAST CLI, the eventnotification list and eventnotification show command output includes new fields:
    • Internal broker ID shows the ID of the VAST Event Broker view.
    • Effective broker name shows the name of the third-party event broker or the name of the VAST Event Broker view.
  • In VAST REST API, the new internal_broker_id and effective_broker_name fields are included in response to GET requests to the /views/ and /views/<ID>/ endpoints.

VAST DataEngine

• ORION-391608: The Manage Elements -> Triggers page in VAST DataEngine UI provides a new context menu option, Delete Selected Triggers, that lets you perform a bulk delete of triggers.
• ORION-360940: Added support of output pagination to DataEngine CLI commands.

Global Namespace

• ORION-401755: Added a validation to prevent creating a quota on a path that is a destination of a Global Namespace stream.

VAST on Cloud (VoC)

• ORION-385076: The following user controls have been deprecated:
  • In VAST CLI, the --total-cnode-machines option on the vm expand command,
  • In VAST REST API, the total_cnode_machines parameter in PATCH requests to the /virtual-macchines/expand endpoint.

VMS

• ORION-399003: VAST Prometheus Exporter offers a new endpoint, /api/prometheusmetrics/protocol_transferred_bytes, that exports the metric showing the total number of bytes in the cluster's response to S3 requests. The values are exported per tenant and then per virtual IP pool.

• ORION-398976: Added the NFSv4 rename latency metric (nfs4_rename_req_latency) to VAST Prometheus Exporter.

• ORION-397053: The Alarms & Events -> Alarms page has a new column, Extra Information, that provides additional details about the object for which the alarm is raised.

• ORION-396453: Added an ability to set the severity of an event to None when configuring an event definition (in VAST Web UI: Alarm and Events -> Event Definitions). The new option allows you to reset severity for event definitions which by default had a severity of N/A and were later updated to a different severity level.

• ORION-389036: VAST Prometheus Exporter features a new endpoint, /api/prometheusmetrics/gns/, that can be used to export Global Namespace metrics. The metrics include:

  • bytes read from the the local or remote cluster (gn_remote_read_size, gn_local_read_size),
  • bytes written from the remote cluster (gn_remote_write_size),
  • latency on reads performed from the local or remote cluster (gn_local_read_latency_us, gn_remote_read_latency_us),
  • latency on writes performed from the remote cluster (gn_remote_write_latency_us),
  • latency due to cross-cluster communication and owner-side operations, for write and create operations performed from a satellite (gn_loanee_write_remote_total_latency, gn_loanee_create_remote_total_latency)

• ORION-362662: Added per-view and per-topic metrics to help monitor and troubleshoot the process of publishing events to non-VAST event brokers:

  • Count of events that were successfully published to the event broker (EventResettledPerKafkaMetrics,..,kafka_external_events_published)
  • Count of events for which the event broker returned an error (EventResettledPerKafkaMetrics,..,kafka_external_events_returned_an_error)
  • Count of events that could not be sent to the event broker (EventResettledPerKafkaMetrics,..,kafka_external_events_failed_to_send)
    The metrics can be obtained for the most recently active 1000 views and 1000 topics.
    Prior to this change, the metrics were available per broker only.

• ORION-347602: Added a new metric named lifecycle_unlinks that shows the number of objects (per view) deleted due to the configured lifecycle rules. The metric can be exported with VAST Prometheus Exporter.

VAST Web UI

• ORION-405098: The Alarms & Events -> Alarms page features an inspection pane on the right that provides detailed information on the selected alarm, including the recommended steps to handle it.

• ORION-398518: The VMS settings page features a new option, Metadata rewrite for DBox removal, that lets you initiate the metadata rewrite process before removing a DBox from the cluster.

• ORION-397147: Enhanced the Settings -> Certificates page so that it shows a notification if the certificate of the selected type already exists in the VMS.

VAST CLI

• ORION-369783: The --snapshot-id option on the restorepoint list command has been deprecated.

VAST REST API

• ORION-405168: Updated the /volumes/bulk/ endpoint to require a list of volumes IDs in the request.

Platform and Control

• ORION-330539: Introduced enhancements for Ceres v2 DNodes to automatically identify and recover from XFS filesystem errors on the DNode boot drive.

Callhome and Support

• ORION-401529: Added an ability to exclude binary traces from support bundle obfuscation. To do so, specify the --force-binary-traces together with the --obfuscate option on the supportbundle create command.

Resolved Issues

Install and Upgrade

• ORION-423581: Enhanced cluster NIC identification routines to resolve an issue that could occur when deploying a cluster with non-Mellanix NICs and cause the deployment to fail due to the failed to run get_nic_psid, 'NoneType' object is not subscriptable error.

• ORION-413328: Resolved an issue that could occur when activating a newly installed cluster and cause the CNode containers to restart with Fiber=<...> group=W_DISCO_BOUNCER had a duration timeout!.

• ORION-409929: Resolved a hardware identification issue that could occur when deploying an EBox and cause the cluster deployment procedure to fail with the Triplication requires ebox or voc or ceres or loopback cluster error.

• ORION-384208: Enhanced VAST Cluster Install to ensure that it never sets any PFC-enabled queues when Global Pause is selected as the flow control mechanism. Prior to this change, an attempt to install with Global Pause could fail with the __root__->__root__: When qos_type is GLOBAL_PAUSE, pfc_enabled_queues should not be configured.<...> error.

• ORION-366380: Introduced enhancements to decrease the time needed for the VMS restart during an upgrade.

• ORION-365842: Resolved an issue where an attempt to upload an upgrade bundle could not proceed due to a wait_for_ssh_ready timeout.

• ORION-361912: Made updates to avoid displaying skipped checkpoints as passed in the Checkpoint tab of VAST Cluster Install.

• ORION-339987: Eliminated a VAST Cluster Install flow where the VMS attempted establishing a SSH connection to an external NIC port that was DOWN, causing the installation process to hang.

• ORION-297545: Resolved an issue that could cause an OS upgrade procedure to fail with Error in function "run_ipmi_power_cmd".

Cluster Expansion

• ORION-399476: Improved the device rebalancing logic to prevent a flow that could cause cluster expansion to fail.
• ORION-375910: Resolved an issue where during cluster expansion, the CNodes being added were stuck in the CREATING state with a cluster ID of NULL.

Networking

• ORION-403388: Updated the cluster networking configuration script (configure_network.py) to let it set BGP routing rules on the cluster's DNodes. Prior to this change, the script could configure BGP routing on CNodes only.

• ORION-398919: Resolved an issue that could cause the cluster networking configuration script (configure_network.py) to fail when run on a Supermicro Turin EBox12 1.5 with two dual-port NICs.

• ORION-329029: Added logic to avoid raising false mtu is not configured correctly alerts for cluster's external interfaces.

• ORION-280521: Resolved an issue where during an attempt to add DBoxes to the cluster, the VMS reported the task as failed due to a timeout while configuring cluster networking, although the configure_network.py script finished successfully on the nodes.

• ORION-230183: Added a validation to prevent creation of virtual IP pools with IP ranges that overlap with the IPMI addresses.

Encryption

• ORION-378799: Resolved an issue where following an upgrade of a cluster with configured external EKM servers, the EKM servers could not be displayed by the VMS.

Element Store

• ORION-404738: Resolved an issue that could cause multiple CNode containers to restart with the assertion failed: (is_first_char_of_nfs3_lock_format(proto_info->_client_string.name[0])) error.

• ORION-392557: Resolved an issue that could cause high latency when trying to modify identity policies via VAST REST API upon deletion of a very large number of local users.

• ORION-380364: Updated metadata chunking routines to resolve an issue where after massive deletions, the cluster was decreasing its metadata block usage slower than expected.

• ORION-375394: Enhanced protection of the .vast hidden directory on S3 views to prevent accidental changes to it by user workloads.

• ORION-259716: Introduced internal enhancements to prevent CNode container restarts when handling operations on extremely large directories.

Multi-Tenancy

• ORION-403027: Made updates to prevent a tenant admin from performing certain tenant-level operations (such as deleting its own tenant or modifying tenant properties that could, in some cases, affect the amount of cluster resources available for use by other tenants).

• ORION-401595: Made updates to harden tenant isolation when using VAST Audit Log and VAST Catalog.

• ORION-279664: Made updates so that the names of customized analytic reports created by a tenant admin cannot be seen by other tenant admins.

Quotas

• ORION-387587: Added logic to allow tenant admins to create and edit user quotas on directories within their respective tenants, if the tenant admin manager users have proper permissions configured for them in the VMS. Prior to this change, a 503 Forbidden error was returned, regardless of the permissions assigned.
• ORION-360876: Made updates to avoid raising Quota <...> state changed from SYNCHRONIZED to FAILED. ROOT_DIR_REMOVED alerts on transient internal errors.

Lifecycle Rules

• ORION-287195: Made updates to prevent lifecycle rule routines from deleting objects in buckets with indestructible object mode enabled, and also items on WORM views.

NFSv4

• ORION-397379: Resolved an issue that could occur when processing a specific type of malformed NFSv4 requests and cause the CNode to restart with the signal=11 at address=(nil) <...> [Address not mapped to object] error.

SMB

• ORION-401431: Updated the logic responsible for tracking memory allocation when handling SMB change notifications to prevent a condition that could cause a CNode to stop sending SMB change notify messages to the clients.
• ORION-395371: Enhanced LDAP caching mechanism to resolve an issue that could cause SMB clients to encounter intermittent access denied errors. The issue would typically occur in a multi-domain environment when a user is a member of multiple groups belonging to a domain that is different from the user's home domain, and such user performs a SMB operation permissions for which are granted through one of these groups.

S3

• ORION-418584: Made updates to harden security when handling presigned requests that include unsigned x-amz-* headers.

• ORION-418506: Made updates so that the cluster's response to GET requests with an expired presigned URL includes error code AccessDenied, as well as the Expires and ServerTime elements. Prior to this change, the cluster responded with ExpiredToken with no Expires and ServerTime elements included.

• ORION-391151: Changed the way the cluster handles ListObjectsV2 requests that contain an object key. Starting with version 5.5.1, such request would return the object identified by the object key. In previous versions, a NotImplemented error was returned.

• ORION-380903: Updated the access check logic for DeleteObjects requests so that it does not require the s3:DeleteObjects permission (non-standard permission implemented by VAST) to be allowed in the identity or bucket policy. Prior to this change, unless s3:DeleteObjects was allowed in the policy, a DeleteObjects request could be denied even when the user had the standard s3:DeleteObject permission allowed for all of the object keys in the multi-delete request. If you want to return to the previous behavior that required s3:DeleteObjects for DeleteObjects requests, contact VAST Support.

• ORION-378803: Changed the message for error code 400 in cluster's response to POST uploads with no object key from InvalidObjectName to InvalidArgument.

• ORION-378749: Added an ability to alter handling of ETags for objects that were created in an S3 bucket using a non-S3 protocol (such as NFS or SMB) and are being copying to self via S3 (CopyObject to the same object key). By default, such objects get an ETag with a prefix of 48ed760a74. The cluster can be configured so that it recalculates the ETag based on the object data. Note that enabling this feature may impact performance for S3 workloads. To configure the cluster to recalculate the ETag for copy-to-self objects, contact VAST Support.

• ORION-344032: Changed the error code that the VAST cluster returns in its response to a SigV4 request with an absolute URI where an incorrect host is specified. Starting with 5.5.1, the cluster responds with error code 403. Prior to 5.5.1, error code 200 was returned.

Block

• ORION-388255: Changed the error code returned for requests where the block volume was not mapped to the host. Starting with 5.5.1, the cluster responds with INVALID_NAMESPACE_OR_FORMAT to such requests. In previous releases, a RESERVATION CONFLICT was returned.

VAST Audit Log

• ORION-360698: Changed the way presigned POST requests are logged in the VAST Audit Log. For clusters installed with version 5.5.1 or later, presigned POST requests are logged with a request type of POST. Upgraded clusters by default preserve the pre-upgrade behavior where presigned POST requests were logged with the PUT_OBJECT request type. If you want to configure an upgraded cluster so that it uses the POST request type when logging presigned POST requests, contact VAST Support.

VAST DataBase

• ORION-408452: Resolved an issue that could, in some cases, cause a discrepancy between the reported metadata usage and used capacity when processing tabular workloads.
• ORION-399151: Resolved an issue that could cause increased metadata space consumption when processing tabular workloads.

VAST Catalog

• ORION-383544: Resolved an issue that could cause denylisting of BIG_CATALOG shards with orphaned metadata handles.

VAST DataEngine

• Updated the Update Trigger dialog so that it allows for removing tags from triggers.

Replication

• ORION-418394: Improved the logic that validates member capabilities when configuring group replication in VAST Web UI so that it does not raise the Member capabilities not valid each time group replication was configured.

• ORION-393875: Resolved an issue due to which an attempt to add a third replication group member could hang if the two destination peers ran specific combinations of VAST releases.

• ORION-381609: Introduced a mechanism to prevent a contention condition that could occur when processing S3 PUT requests against directories under protected paths with synchronous replication enabled, causing multiple CNode containers to restart with the waiting for lock for too long error.

• ORION-380151: Added an alert to be raised during creation of a protected path in case a file under the path has hardlinks that are not within the path.

• ORION-378690: Improved user ID mapping during replication to resolved an issue where replicated S3 keys could not be assigned to local users on non-default tenants.

• ORION-372785: Made updates to prevent potential automatic re-creation of deleted IAM roles on replication peers.

Global Namespace

• ORION-381390: Optimized Global Namespace data prefetching to eliminate a contention condition that could cause multiple CNode containers to restart with the timeout expired for @life_type=16,life_gen=<...> (TRAVIS) error.

• ORION-378170: Made updates to prevent timeouts when retrieving metadata for a very large directory on a Global Namespace satellite.

Authentication and Authorization

• ORION-418885: Resolved an issue that could cause Failed identify External user errors on attempts to log in to the VMS using a LDAP user account from a LDAP provider that is not associated with a cluster tenant.
• ORION-371895: Made updates to prevent VMS login failures when using LDAP groups to authenticate a user being a member of more than 440 groups.

VMS

• ORION-412929: Updated categorization of cluster event definitions to ensure consistency between the type of alarms reported by the VMS and the type of user (such as admin user or support user) required to handle them.

• ORION-401264: Updated the logic behind the VAST Prometheus Exporter endpoint /api/prometheusmetrics/user_view to avoid exporting duplicate metric entries.

• ORION-395092: Resolved an issue due to which VAST Prometheus Exporter could stop exporting the vast_cnode_metrics_SchedulerMetrics_* metrics, with the VMS log showing the No metrics for cnode - SchedulerMetrics error.

• ORION-368905: Resolved a caching issue that could cause the Analytics -> Capacity page to occasionally display inaccurate capacity estimations.

• ORION-366993: Updated the calculations for the User QoS Wait Time metric to ensure that the metric provides correct values.

• ORION-292325: Resolved an issue that could occur when creating a customized analytics report for block volumes or hosts and cause some metrics, intended to reflect a maximum latency, to be set to a fixed value when there were no block volumes on the cluster.

VAST Web UI

• ORION-427408: Updated the State column in the Protected Paths page so that it can be filtered to display only paths in the Partially Active state.

• ORION-424019: Made updates so that the Create IAM Role button in the Add IAM Role dialog of the VAST Web UI for tenant admins works as expected.

• ORION-410154: Updated the dialog used to create a managed application (Data Engine -> Applications -> choose to create or edit an application -> Resource Selection) to provide correct guidance on the number of CNodes required for a managed application (at least 2; some application types require more than two CNodes).

• ORION-403079: Resolved an issue that could cause an Invalid Argument error trying to add a tag to a snapshot volume (Element Store -> Volumes -> choose to edit a volume).

• ORION-399000: Updated the Rewrite Status popup (appears after you initiate a data rewrite on the cluster) to prevent showing decreasing progress percentage.

• ORION-394014: Added the Enable bucket replication option to the Add Global Snapshot Clone dialog. This option let you enable bucket replication for the global snapshot clone being configured.

• ORION-390094: Updated the Protocols dropdown in the DataBase -> VAST Audit Log page to include an entry for STS.

• ORION-385292: Updated the Infrastructure -> Racks page to allow for sorting and filtering by usable capacity.

• ORION-379170: Updated the Data Engine -> Compute Clusters page so that it displays a data loading indicator while fetching tenants to for the Tenant dropdown.

• ORION-378605: The Settings -> Certificates page has been updated to display the certificate expiration date for each certificate.

• ORION-346783: Updated the Cluster admin groups field in Active Directory settings (User Management -> VAST Providers -> Active Directory or LDAP -> open a provider -> Advanced tab) to ensure that the field length is enough to fit all of the specified group names.

VAST REST API

• ORION-405183: Updated the /volumes/bulk/ and /blockhosts/bulk/ endpoints to require a comma-separated list of volumes on DELETE requests.

Platform and Control

• ORION-428332: Eliminated a flow that could cause the cluster deployment procedure to fail due to the object=DNode<...> failed during system_format - unsupported errors occurring when trying to reformat SCM drives.

• ORION-418826: Updated the stripe allocation logic to resolve an issue that could cause a decrease in the cluster's write throughput.

• ORION-417123: Adjusted an internal timeout to prevent CNode containers from restarting with the assertion failed: (((int)res) != ((int)EStore::EStoreRes::TIMEOUT)) (26 != 26) waiting for the completion of @truman_tid=<...> for too long! error.

• ORION-414389: Eliminated a potential CNode memory leak that could occur after DBox removal.

• ORION-391742: Resolved an issue that could cause the CNode container to restart due to the more than a single update for same hash and tid error.

• ORION-388033: Eliminated a race condition that could occur on a leader HA event during DNode deactivation and cause multiple CNode containers to restart with the Unknown device: <...> at file: src/control/dev_agent/dev_agent.hpp:473 func: Control::RemoteDevice *Control::DevAgent::get_device(P::GUID, bool) error.

• ORION-380651: Resolved an issue that could cause an assertion failed: (false) inconsistent serial error followed by ESTORE TOKEN_SAMPLER denylist alert on the cluster.

• ORION-377519: Added logic to prevent iLORest.log space exhaustion on clusters with HPE Genoa CNodes.

• ORION-367005: Added a validation to prevent activation of DNodes or ENodes with 2 single-port Mellanox NICs where at least one of the NICs has a PCI atomic mode setting not equal to '1', which could cause CNode containers to restart with the (had_an_incomplete_op) failed unlock while no previous incomplete attempts were recorded error due to misconfigured PCI atomic mode.

• ORION-359687: Resolved an issue that could cause the CNode container to restart with the assertion failed: (allow_reentrant || !is_locked_by_me()) spinlock deadlock thread tried to acquire same lock twice silo_id=0x'<unprintable char>' {0x1a} error.

• ORION-357872: Resolved an issue where an SCM failure caused multiple leader restarts with the can't fail nvram=NVRAMType:<...> (can reach DU) error.

• ORION-333348: Made updates to eliminate contention that could occur during synchronous replication with large multi-part uploads and cause multiple handles to be denylisted due to the assertion failed: (_latest_hash_idx == INVALID_HASH_IDX \|| name_hash-\>get_snap().snap_time \<= _latest_hash_snap_line.snap_time) found committed hash entry in newer snap time than latest entry?! error.

• ORION-315927: Improved validation of virtual IPs assigned to CNodes to prevent a flow that could result in multiple node container restarts with the assertion failed: ((shard_info->bucket_info) == (nullptr)) error and temporary service disruption.

• ORION-270084: Made updates to eliminate a flow which could cause service disruption due to the leading node failing while attempting to activate a very large number of CNodes at the same time.

• ORION-259716: Enhanced internal routines responsible for directory listing during to prevent CPU hogging when handling very large directories.

Callhome and Support

• ORION-386212: Resolved an issue that could cause the Support bundle collection not implemented for switch type 'hpe_anw'. Supported types: 'aruba', 'cumulus', 'mellanox' (no-op) error when trying to download a support or callhome bundle from a cluster with an Aruba switch running firmware of versions 10.16.*.
• ORION-381454: Made updates to prevent a scenario where support bundle collection could not be completed due to a BMC issue on one of the cluster nodes.

Limitations

Install and Upgrade

• Automatic drive firmware upgrade is not performed on drives that have been moved manually from an old DBox to a new DBox during the DBox replacement procedure. (Note that this does not apply to EBoxes.)
• VAST DataEngine requires a special upgrade procedure performed in strict compliance with the steps and guidelines provided in the VAST Cluster Administrator's Guide.
• ORION-281679: SSD firmware upgrade is not supported for Supermicro EBoxes.
• ORION-280966: Having imported a JSON configuration file in VAST Cluster Install, you need to manually verify that all the populated fields have expected values. Sometimes, depending on the cluster configuration and environment, some of the fields are not populated as expected during the import.
• ORION-242658: BMC firmware upgrades are not supported for Supermicro Genoa CNodes.
• ORION-222648: NDU that includes automatic adjustment of CNode CPU isolation settings (isolcpus) is not supported for EBoxes.
• ORION-214559: A BMC upgrade cannot be performed with an inactive CNode that has been powered off.

Networking

• The following limitations apply when using Open Telemetry-Based Ethernet Network Monitoring:
  • Up to 70 switches per cluster.
  • The switch must have OTel configured to send telemetry data over gRPC to the VMS IP address using port 4317.
  • TLS on the connection used to obtain the metrics from the switch is not supported.
  • Supported on NVIDIA Cumulus switches 5.12.1 and later. For other switch vendors, inquire VAST Support.
• The following limitations apply when implementing L3 networking:
  • After enabling L3 access for a virtual IP pool, it cannot be disabled.
  • L3 access is not supported on virtual IP pools for which CNode Port Affinity is configured.
  • ORION-333743: L3 access is not supported on virtual IP pools that include more than one range of virtual IPs.
  • L3 networking is not supported on IB clusters.
  • L3 networking is not available for VAST on Cloud.
  • MD5 authentication is not supported.
  • ORION-266297: When using numbered BGP, each CNode can only be involved in one BGP configuration. To learn which CNodes are currently used in which BGP configurations, use the Network Access -> Virtual IP Pools page that displays CNodes and BGP Configurations for each virtual IP pool.
  • Reverting an L3-configured cluster to L2 through VMS is not supported. If you need to revert, contact VAST Support.
• Changing network configuration from VMS (in VAST Web UI: Settings -> Configure Network) is not supported for clusters that have any CNodes that are connected simultaneously to multiple separate client data networks.
• ORION-380243: When reconfiguring cluster networking from non-B2B to B2B, there is no indication of the requirement to physically swap the cables on the nodes so that the task can complete successfully. The task pauses when attempting to deactivate the final CNode and resumes only after the cables are reconnected.
• ORION-374025: Virtual IP pools with a role of Query Engine cannot be modified after creation. This is a built-in safeguard to prevent in-flight queries from failing. If you need to introduce a change into an existing Query Engine pool configuration, delete the pool and create a new one.
• ORION-340819: Before modifying a virtual IP pool, ensure that during the modification all of the views associated with this pool remain assigned to the pool. Switching a view from one pool to another while either old or new pool is being modified, may result in unexpected behavior.

Encryption

• The following rules and limitations apply when using Kerberos providers:
  • MIT Kerberos providers are used for NFSv4 only.
  • A tenant can be associated with a single Kerberos provider only.
  • Up to eight Kerberos providers can be defined on a VAST cluster.
  • Kerberos principals used to authenticate to the VAST cluster must exists as users in an LDAP provider or in a VAST provider associated with the same tenant as the Kerberos provider. Otherwise, they are squashed to the nfsnobody user.
• The default tenant does not use keys created or stored within an EKM. To encrypt data within the default tenant with EKM-based keys, use encrypted paths.
• ORION-388667: Creating an encrypted path on a directory exposed by an ABAC-enabled view is not supported. An attempt to create an encrypted path on the same directory as the one exposed by an ABAC-tagged view causes the assigned ABAC tags to be removed.
• ORION-208004: Enabling VAST OS boot drive encryption requires that the node is inactive. Enabling the encryption on an active node may cause a long reboot sequence.

Quotas

• The following limitations apply when using Remote Quota Protocol (rquota):
  • SETQUOTA requests are not supported.
  • Quota information can be provided for active users only.
  • Information about group quotas is not provided.
• Quotas are not enforced on replication destination directories under a protected path. For example, if the protected path is /ppath, a quota on /ppath/yourdir is not enforced.
• ORION-301453: Reaching a quota's hard limit blocks writes to existing (previously written) offsets. When using ESX, this results in inability to perform operations such as VM clone, VM power-on, or datastore operations.

Lifecycle Rules

• ORION-280461: The maximum size of a lifecycle rule is 15MB.

Quality of Service (QoS)

• The prioritization flag cannot be set for user QoS policies.
• Some high-priority optimizations are applied to NFSv3 only.
• The following limitations apply when using QoS for block volumes:
  • Using more than 100 volumes that have QoS limits applied to them may result in inaccurate performance capping.
  • Prioritization of a volume QoS policy over cluster QoS limits is not supported.
  • ORION-266523: For volumes that share the same subsystem, it is recommended that all of the subsystem volumes have the same QoS limits set, or all of them do not have a QoS policy assigned, to prevent possible host IO performance impact.

NFS

• ORION-324346: NFS aliases are not supported with VAST implementation of Remote Quota Protocol (rquota).
• ORION-115336: If one creates an NFSv4.1-only view and mounts it, and then creates its parent view with NFSv3 only, IO operations on the NFSv4.1-only view succeed but mounts are not allowed.

NFSv4

• TLS encryption with NFSv4.1 is not supported for NFSv.4.1 over RDMA.
• The following requirements and limitations apply when using NFSv4 file delegations:
  • The NFSv4 protocol requires Network Time Protocol (NTP) to be configured on both the VAST cluster and the client. While VAST does not enforce this requirement, the absence of proper time synchronization may, in rare cases, result in unexpected behavior.
  • The client must have an NFSv4 backchannel connection.
  • For multi-protocol views with NFSv4 delegations enabled, close-to-open consistency is only preserved between NFSv4 clients, but not between NFSv4 clients and other protocols (including NFSv3). For example, if a non-NFSv4 client modifies a file on such a view, the modification may not become immediately visible to an NFSv4 client holding an active delegation on the file.
  • Before making a path read-only by using the /folders/read_only endpoint of VAST REST API, ensure that NFSv4 delegations are disabled or returned. Otherwise, unexpected behavior may be encountered.
  • Restoring a snapshot on a tenant that have NFSv4 delegations enabled, may cause unexpected behavior with regard to delegated files. It is recommended to revoke existing delegations before restoring a snapshot.
  • VAST Cluster recalls granted NFSv4 delegations when a failover process is started.
  • In some cases when there is very high NFSv4 callback load per cluster or per client, VAST Cluster might start recalling or revoking NFSv4 delegations due to callback request timeouts.
  • NFSv4 delegations require mounting the client at least as NFS version 4.1. It is recommended to mount as NFS version 4.2 for additional performance improvements. Linux kernels 6.11+ on a client provide more performance advantages (RFC9754), which get backported to older kernels through VAST-NFS.
  • If you are using VAST-NFS, note that NFSv4 delegations are supported with VAST-NFS 4.0.32 and later. VAST-NFS 4.5 includes enhancements related to support of NFSv4 delegations.
  • SUSE Linux Enterprise Server 12.x and CentOS 7.x clients are not supported with NFSv4 delegations enabled.
• The following applies when using NFSv4 extended attributes:
  • The VAST cluster does not enforce any limitation on the maximum number of extended attributes per file. However, some clients may have such a limitation when listing extended attributes.

SMB

• The following limitations apply when using SMB hardlinks:
  • Creating a hardlink to another SMB-enabled view is not allowed.
  • Creating hardlinks to directories is not allowed.
  • Linking an Alternate Data Stream (ADS) is not supported.
  • Creating a hardlink on a Global Access satellite cluster is not supported.
  • Creating a hardlink, on the Global Access origin cluster, of a file that has already been opened from a satellite cluster, is not allowed.
• The following limitations apply when managing SMB open files:
  • In VAST Global Namespace, operations on open file handles are only supported for the current cluster.
  • ORION-289428: SMB open file queries do not show open SMB hardlink destination files.
  • ORION-288057: When listing files on a path where there is an open ADS stream (path\file:stream), the stream is shown as if the file is a folder (\path\file\stream).
  • For a path with an open Alternate Data Stream (ADS) on it, the following actions are not supported:
    • Closing the ADS
    • Listing individual open handles associated with the ADS.
• Views exposed as SMB shares work only if the cluster is joined to Active Directory. This includes both SMB-only and multiprotocol views.
• The following limitations apply when using SMB change notifications for subdirectories:
  • SMB recursive change notifications are not available for paths for which Global Namespace is configured.
  • Using SMB recursive change notifications for nested SMB shares may entail unexpected behavior.
• The following limitations apply when using Alternate Data Streams (ADS):
  • ADS are not seen or accessible by protocols other than SMB.
  • ADS will not be retained when files and directories are replicated to S3.
    ORION-169707: When the Hyper-V management tool tries to list VAST Hyper-V SMB shares on an SMB server, the The RPC server is not available error can occur if the SMB server is specified using its FQDN. To avoid this error, specify the IP address of the SMB server instead of the FQDN.
• ORION-160323: After updating permissions for an SMB share in Windows Explorer, a duplicate SMB share can be displayed. The duplicate SMB share disappears upon a refresh (F5).
• ORION-134730: An attempt to restore a file can fail if after the restore has started, a quota is set on the path where the file resides.

S3

• The following limitations apply when using S3 Versioning:
  • After versioning has been enabled on a bucket, it cannot be disabled.
  • Multiprotocol access to versioned objects is not supported.
  • MFA Delete is not supported.
  • S3 conditional writes are not supported for versioned buckets.
• The following rules and limitations apply when using S3 Server-Side Encryption with Customer-Provided Keys (SSE-C):
  • If an S3 request contains an x-amz-server-side-encryption-customer-* header, it is required to use HTTPS for the request to be accepted by the cluster.
  • Only AES256 encryption algorithm is supported.
  • Objects encrypted with SSE-C cannot be accessed through other access protocols.
  • For replication environments, it is strongly recommended to avoid using SSE-C unless all of the clusters involved support SSE-C.
• A POST policy (used for S3 presigned POST requests) can be up to 4800 bytes.
• The following limitations apply when using S3 Indestructible Object Mode:
  • An S3 Bucket view with Indestructible Object Mode cannot have other protocols enabled.
  • Indestructible Object Mode cannot be set for a view that points to / (root directory).
  • It is not allowed to have views under the view in Indestructible Object Mode, or at the same path as the Indestructible Object Mode view.
  • Directories that contain views that have indestructible object mode enabled cannot be moved to the trash folder.
  • Indestructible Object Mode cannot be used together with S3 Object Locking or S3 Object Versioning.
  • Indestructible Object Mode cannot be used together with WORM.
  • Indestructible Object Mode cannot be set for a view that exposes the protocol audit log directory.
  • A snapshot cannot be restored on a view that is in Indestructible Object Mode.
  • Views in Indestructible Object Mode are not subject to replication or Global Access.
• The following limitations apply when using CORS:
  • ORION-314829: VAST Cluster does not include the Access-Contol-Allow-Origin header in case of a failed PutObject request.s
• An object to be uploaded via a S3 presigned POST request must have only ASCII characters in its name.
• ORION-340559: Absolute-form URLs (e.g. http://.../.../<object> instead of /<object>) in CompleteMultipartUpload requests are not supported.
• ORION-240888: S3 bucket monitoring does not take into account VMS-originated S3 requests, including those related to features such as:
  • Lifecycle rules
  • Event notifications
  • Bucket policies
  • Object ownership
  • Bucket versioning
  • Object locking
  • Bucket logging
• ORION-197281: VAST Cluster disables bucket logging set on a bucket from which data is synchronously replicated to another bucket once you set up bucket logging on the replication destination bucket and configure it to use a different logging destination bucket.
• ORION-190674: Once created, an S3 bucket cannot be renamed or moved to a different path. Thus, for example, if you try to change the bucket’s path when modifying a view in VAST Web UI, the change does not take effect and the view will still be listed with the old path.

Block

• VAST Cluster does not support subsystem names that contain a space (in VAST Web UI: Element Store -> Views -> choose to create a view -> Block tab -> Subsystem name field).
• For Rocky Linux-based clients, VAST recommends that the client uses Rocky Linux 9.4 or later.
• An attempt to remove a volume that has snapshots may cause errors for volume objects of snapshots of that volume, if they exist.
• A view that is used to expose block storage cannot have other storage protocols enabled.
• The following VAST capabilities are not available with block views:
  • Access control features (such as ABE, ABAC, WORM)
  • VAST Audit Log
  • Replication to a remote peer
  • Global Access
  • Remote global snapshot clones
• Nesting of a block view inside an existing block view is not allowed.
• The host NQN cannot be modified. To change the NQN, you need to remove the host and then add and map it anew.
• You cannot enable or disable block storage support on an existing view. Block storage support can only be enabled for a view during view creation and cannot be disabled afterwards.
• Block devices can be created on empty directories only.
• If a host defined on the VAST cluster does not have any volumes mapped to it, NVMe auto-discovery does not show this subsystem.
• When using the VAST Web UI or CLI options to bulk create volumes or hosts, the number of items to be created cannot exceed 256. When mapping hosts to volumes, up to 256 items can be mapped at a time.
• The maximum IO block size is limited to 1MB (4GB for unmap).
• ORION-388015: Compare-and-Write (CAW) operations are only supported when performed against a single block (512B).

Attribute-Based Access Control (ABAC)

• ABAC is supported on views controlled with SMB, S3 Native and Mixed Last Wins security flavors. ABAC is not supported with NFS flavor.
• ABAC tags cannot be set on the cluster’s root directory (/).
• If a user does not have any ABAC permissions, the user still can mount an NFSv4 export or map a SMB share to a local drive, but the user is not allowed to perform any operations on the files or directories.
• The following features and capabilities cannot be used together with ABAC-tagged views:
  • If a tenant has ABAC-tagged views, you cannot change or remove the Active Directory provider configured for the tenant.
  • When using NFSv4, it is not allowed to create hardlinks in views that have ABAC tags.
  • When using S3:
    • ABAC cannot be used with anonymous S3 access. You cannot set ABAC tags for views that have anonymous S3 access enabled.
    • It is not allowed to set ABAC tags on a view that is a target for S3 bucket logging.
    • Requests from S3 superusers are handled in the same way as for regular users. This means that an S3 superuser is not granted access if the ABAC access check denies access for this user.
  • A directory under which an ABAC-tagged view exists, cannot be moved to the Trash folder.
  • Bulk permission updates are not available for ABAC-tagged views.
  • Lifecycle rules cannot be set for files or directories with ABAC tags.
  • The tenant with ABAC-tagged views cannot have SMB native authentication enabled.
• Once assigned, you cannot edit or remove the ABAC tags of a view. Assigning new ABAC tags to an existing view or directory (storage path) is not allowed.
• ABAC is not supported with NFSv3.
• If you create a view for a directory that already exists, ABAC tags from the existing directory are assigned to the newly created view. In this case, there can be a delay between the view creation time and the time when the view's ABAC tags can be displayed.
• After a child view inherits ABAC tags from the parent view, you cannot update or remove the ABAC tags on the child view.
• ORION-388667: Creating an encrypted path on a directory exposed by an ABAC-enabled view is not supported. An attempt to create an encrypted path on the same directory as the one exposed by an ABAC-tagged view causes the assigned ABAC tags to be removed.

Write Once Read Many (WORM)

• You can add additional protocols to a WORM-enabled view, but cannot remove a protocol.
• You cannot configure a WORM-enabled view for NFS/SMB and S3 together.
• Nesting views under a WORM-enabled view is not allowed.
• You cannot disable WORM for the view once it is enabled.
• Bulk permission updates are not allowed on WORM-enabled views.

User Impersonation

• ORION-216379: When VAST protocol auditing is enabled on a user-impersonated view, only UID of the original user is included in the log. The user's login name and SID are not included.

Bulk Permission Updates

• A bulk permission update can run only when the target view (the view exposing the files and directories for which you want to update permissions) is on the same tenant as the template view.
• Only one bulk permission update task per tenant can run at a time.
• Running a bulk permission update on a view where the security flavor does not match that of the template view may result in inaccessible or incompatible permissions set.
• Read-only snapshots and VAST special directories (.vast in S3 buckets, .trash, .snap, .remote) are excluded from bulk permission update.
• If a client attempts to set permissions on directories or files being updated via a bulk permission update, the result is unpredictable.
• Bulk permission update cannot run on ABAC-tagged views.

VAST Audit Log

• ORION-385315: When accessing VAST Audit Log through the VMS REST API (including VAST Web UI), sorting/ordering of VAST Audit Log entries is not supported in case the query results include more than 1000 rows. In this case, VAST Audit Log returns random 1000 entries. To mitigate this behavior, narrow your search to return less than 1000 entries, or use other access mechanisms (such as VAST DB SDK or a query engine connected to the VAST Database).

Event Publishing

• The following limitations apply when using VAST Event Broker:
  • Producer API:
    • Messages are limited to 1MB.
    • In the event record, the key is limited to 126KB and the value is limited to 126KB.
    • Access to topics by UUID is not supported.
    • Idempotent producing is not supported.
    • Automatic creation of topics is not supported.
  • Consumer API:
    • No more than 256 consumer groups per view (broker)
    • The following is not supported:
      • Consumer group stickiness parameters (such as group.instance.id)
      • READ UNCOMMITTED isolation level
      • Cooperative rebalancing
      • Client rack awareness
      • Fetch sessions (only full fetch will be applied), delayed fetch parameters
      • Seek by time
  • Admin API:
    • Supported APIs include the APIs to create topics, delete topics, and to delete groups, as well as describeConfigs and alterConfigs APIs that let you get and update the topic configurations.
  • The following Kafka capabilities are not supported:
    • Over-the-wire compression of messages

    Note: VAST compression of data is supported.

    • Transactions
  • Only one virtual IP pool can be associated with a Kafka-enabled view, providing at least one virtual IP per CNode. Once the view has been created, the virtual IP pool cannot be replaced by another one (but it can be modified if needed).
  • The amount of VAST Event Broker views that you can create on a VAST cluster, is limited by the maximum number of views supported by the cluster and by the maximum number of virtual IP pools (since each broker view requires a dedicated virtual IP pool). See VAST Cluster Scale Guidelines for details.
  • The amount of VAST Event Broker views that you can create on a VAST cluster, is limited by the maximum number of views supported by the cluster.
  • The amount of event topics that you can create on a VAST cluster, is limited by the maximum number of tables per VAST Database table and the overall amount of event topic partitions.
  • A topic can have up to 20,000 partitions. The number of partitions in a topic cannot be changed after the topic has been created. Up to 200,000 partitions are supported per VAST Event Broker view.
  • Event queries based on the topic partition are not supported.
  • When listing consumer groups, the response is limited to 256 groups per Kafka-enabled view.
  • Event publishing and consuming operations, as well as topic management operations are not subject to VAST Protocol Auditing or Quality of Service (QoS).
  • VAST Cluster supports Confluent Kafka Python client 2.4 - 2.8.
  • When using aiokafka 0.12 or earlier, set api_version="1.1.0". For aiokafka 0.13 and later, do not specify api_version.
  • Authentication and authorization:
    • Active Directory/LDAP is not supported. The user must be defined as a VAST local user.
    • Only one Kafka TLS certificate can be uploaded per VAST cluster.
  • Data protection:
    • The Kafka-enabled view needs to be manually created and associated with a virtual IP pool at the destination peer. The pool must have the same name as the one at the source peer.
    • Fast restore of a protected path containing a Kafka-enabled view is not allowed.
    • VAST replication of consumer groups is not supported. Consumer group offsets are not replicated.

VAST DataBase

• The following limitations apply when using vector indexing:
  • Once vector indexing is enabled for a table, it cannot be disabled.
  • After a vector index is created, its distance function cannot be changed.
  • Only one vector column can be indexed per table.
  • The word 'vector' cannot be used as the vector column name.
  • Adding a vector index to an existing table is not supported.
  • Adding or deleting columns in an indexed table is not supported.​
  • Vector indexing cannot be used together with sorted tables and partitioned tables.
  • VAST Connectors for Trino, Spark and Dremio do not support vector index-based search.
• The following limitations apply when using table views:
  • View properties are not supported.
  • Queries to a view must include full table names.
  • Redefining a view is supported for Spark clients only.
  • User-defined column names and comments are lost if the schema of the query changes when redefining a view.
• The following limitations apply when using table partitions:
  • Partitions cannot be created based on nested columns (list, map, struct column types).
  • Columns used to create partitions can only be updated if the update does not affect existing partitioning.
  • Columns used to create partitions cannot be dropped once the partition exists.
  • Table partitions cannot be used together with the following database features:
    • Vector indexing
    • Semi-sorted projections
    • User-defined row IDs
  • Sorting can be used on partitioned tables, but the sorting order can only be defined at table creation. Existing partitioned tables cannot be sorted.
  • It is not allowed to use the same column both for sorting and partitioning.
  • ORION-302884: Quotas on partitioned tables are not supported.
  • Replication of partitioned tables is not supported.
  • VAST Query Engine does not support table partitions.
• The following limitations apply to sorted tables:
  • Once enabled for a table, the sorting cannot be disabled.
  • Sorting can be performed for tables that contain 512k or more rows. Although you can enable sorting on a table regardless of its size, smaller tables do not get sorted.
  • Once sorted, tables cannot be unsorted or have their sorted columns changed.
  • Only one sorting order can be defined on a table. Up to four sorted columns are supported.
  • Sorted column values cannot be updated to different values.
  • Sorted tables cannot be replicated. Tables that are replicated cannot be sorted.
  • Sorting cannot be enabled on tables that have semi-sorted projections. Semi-sorted projections cannot be added to sorted tables.
  • Nested data types are not supported for sorted columns.
  • Tables that expose the internal row ID with the vastdb_rowid column cannot be sorted.
• The VAST connector for Trino does not support the SECURITY clause in CREATE VIEW statements.
• The following requirements and limitations apply when using Row/Column-Level Security:
  • Row filtering and column masking are supported for Trino query engines only.
  • The Trino user's identity policy must have the EndUserImpersonation and GetRowColumnSecurity actions allowed.
• VAST Query Engine does not support the nested data type of map.
• The following limitations apply when using Blob Expansion:
  • Only one expansion table can be defined for a given column in the event topic.
  • The expansion target table must be created (without columns) before you start configuring blob expansion for the table being expanded.
  • Renaming or removing target table columns that are included in blob expansion configuration, is not allowed.
  • Updates/deletes to the source columns do not affect the target table.
  • Renaming of the source table is not allowed.
• ORION-353292: Importing data into a VAST Database table for which sorting and partitioning have been enabled, takes considerably longer compared to import to an unsorted and unpartitioned table.

VAST Catalog

• The maximum path length supported by VAST Catalog is 1024 characters.
• ORION-197741: VAST Catalog cannot be enabled on a cluster that uses encryption keys managed through EKM, including per-tenant and per-path encryption keys.

VAST DataEngine

• The following rules and limitations apply when running Trino clusters on VAST:
  • Not less than three CNodes are required per Trino cluster (one for the coordinator, two or more for the workers).
  • In case of CNode HA events, the Trino cluster remains accessible and running queries as long as one coordinator CNode and one of the worker CNodes are up.
  • A VAST cluster upgrade (NDU) performed while running a Trino cluster may cause Trino query failures. The Trino cluster will be fully operational after the NDU is complete.
• If VAST DataEngine is enabled on the cluster, replication of the tenant's root directory is not allowed.
• The following requirements and limitations apply when using NFS-based triggers:
  • Only regular files can be the source for trigger events. Directories, symlinks and other special file types are not supported.
  • Only NFSv4 is supported. Events resulting from NFSv3 or SMB operations on the same source view would not be used for the trigger.
  • The source view must have NFSv4 and S3 Bucket protocols enabled.
  • NFSv4 file delegations must be disabled on the view configured as the trigger's event source view.
  • Events occurring to elements on nested views of the source view, cannot be used as the basis for VAST DataEngine triggers. For example, if an NFS-based trigger is configured on a source view at /abc/, an event on its nested view /abc/xyz/ will not trigger any action in VAST DataEngine.
  • If a file is renamed or deleted while it is being created (e.g. the file has been opened and written to, and then it got unlinked or removed), no create or delete event is produced.
  • ORION-320111: During an upgrade, the VAST cluster does not produce Element Deleted events for files subject to NFSv4 unlink and rename operations.
• Up to 32 engines are supported per VAST cluster.
• The following rules and limitations apply when using VAST compute clusters:

  • Each compute cluster requires at least three CNodes (because three server nodes are required to support HA). If more than 15 CNodes have been selected for a compute cluster, five master nodes will be allocated.

    Note: The role of the node (server/worker) is determined automatically considering the lifecycle state of the nodes, and may change throughout the lifecycle of the VAST cluster (for example, role changes are expected in maintenance operations such as FRU and NDU).

    Depending on the number of active worker nodes, server nodes may also execute workload.

  • A CNode can be used for one compute cluster only.

  • ORION-348513: Compute clusters cannot be deployed in a environment with CNode port affinity (where the LEFT port and the RIGHT port of a CNode can be on different networks).

  • Up to 32 VAST-internal compute clusters can be created per VAST cluster.

  • ORION-291582: VAST-managed compute clusters are not available in VAST on Cloud.

  • ORION-365247: VAST-managed compute clusters cannot be run on CNodes with L3 networking.

  • ORION-288369: VAST-managed compute clusters are not supported in environments with Layer 3 data networking.

  • ORION-371504: IPv6 addressing is not supported.

  • ORION-342312: There is no validation that the IP specified for a compute cluster node is free.

  • ORION-397536: VAST-managed compute clusters cannot be run on ENodes.

Data Protection

• When using NFSv3, in rare cases with large numbers of files and directories, the existence of a view with Global Synchronization enabled under a protected path can block the removal of the protected path.

Replication

• The following limitation applies to VAST Database asynchronous replication:
  • ORION-179909: VAST Database asynchronous replication cannot be used together with Global Access or synchronous replication on the same path.
  • Only committed database transactions are replicated.
  • VAST Catalog and audit log are not replicated.
  • Multi-tenant replication of VAST Databases is not supported.
• The following limitations apply to synchronous replication:
  • A protected path using synchronous replication can only contain S3 buckets. You cannot use synchronous replication for paths that are exposed to any other protocols.
• The following limitations apply to synchronous replication for S3:
  • Synchronous replication is supported for S3 buckets only.
  • It is not allowed to configure local snapshots, global snapshot clones or Global Access on the protected path for which synchronous replication is configured.
  • S3 lifecycle rules are not replicated.
  • S3 keys are replicated asynchronously.
  • Synchronously replicated directories are not subject to bulk permission updates.
• Protected paths with asynchronous replication cannot be nested.
• Data cannot be moved into or out of a path that is protected by either asynchronous replication or S3 replication.
• it is not possible to change which protection policy controls any replication stream.
• The following limitations apply to bucket replication:
  • ORION-338118: If your replication environment includes a cluster where bucket replication is enabled, ensure that all of the replication peers also have it enabled. Otherwise, unexpected behavior may be encountered during asynchronous replication failover after the clusters are upgraded to version 5.5.
• The following block-specific rules and limitations apply when using asynchronous replication of block volumes:
  • If a replication group includes a replication stream that replicates block volumes:
    • All replication peers in the group must be running VAST Cluster 5.5.0 or later.
    • The replication group can include only asynchronous replication streams.
  • Block hosts and mappings between the volumes and the hosts are not replicated.
  • The default block view policy is to be created manually for the relevant tenant at the replication destination peer.
  • When using asynchronous replication of block volumes with ESXi, user intervention is required to unmap volumes from the ESXi host if they are no longer writeable (e.g. after a failover). Otherwise, volumes that are intended to be presented as read-only, are not being detected as such by ESXi.
• ORION-208123: Local user accounts are not subject to replication.

Global Access

• VAST Database and VAST Event Broker (Kafka) views are not supported.

• The following limitations apply when using Global Access for S3 buckets:

  • Identity policies must be enabled at the cluster to which they get replicated.
  • The following VAST capabilities are not supported on destination buckets:
    • S3 event notifications
    • S3 Indestructible Object Mode
    • Lifecycle policies
    • Write Once Ready Many (WORM)
  • Bucket logging is only supported if both the source and destination buckets are in the same protected path.
  • Bucket replication between two clusters is only supported when the bucket is associated with the default S3 view policy.
  • S3 endpoints are not replicated.

• The following combinations of access protocols and security flavors are supported:

  • Access to a Global Access protected path (global folder) on the source cluster: NFSv3, SMB, S3 with any of the flavors; NFSv4 with any flavor except the SMB flavor.
  • Access to a satellite path on the destination cluster: NFSv3 with NFS or S3 Native flavor, SMB (all flavors), S3 (all flavors)
  • Access to a path which is part of both Global Access and asynchronous replication setup: NFSv3 with NFS flavor, SMB (all flavors), S3 (all flavors)
    If a view is configured with both NFSv4 and SMB, it must be controlled with the NFS security flavor.

• Lease expiration time can only be set when creating a global access protected path. You cannot change lease expiration time when you modify a global access path.

• VAST Catalog does not provide information on the cached data on the remote cluster.

• ORION-194805: Applications that use SMB2 Byte Range Locks are not supported when the SMB client is connected via a remote Global Access protected path. Examples of such applications are Microsoft Office suite on macOS, Microsoft Hyper-V, AutoDesk 3ds Max and some Adobe Premiere plugins.

• ORION-194613: If some files have additional hardlinks, the amount of bytes reported as prefetched can be higher than the actual amount prefetched.

VAST DataSpace

• VAST DataSpace requires that each cluster participating in the inter-connection is running VAST Cluster 5.0 or later.
• VAST clusters with IPv6 management IPs cannot be added to VAST DataSpace. Only IPv4 is supported.
• ORION-135966: The inter-connecting clusters must have connectivity to each other through the clusters' management networks.

VAST on Cloud (VoC)

• VAST Cloud (Polaris) UI does not provide an option to run a cluster upgrade. Cluster upgrades are to be performed using the VAST Web UI.
• VAST on Cloud clusters do not support OS upgrade.
• In the event of downtime, data is rebuilt while the cluster comes back online. In case of a subsequent failure during the rebuild, data integrity is not guaranteed.
• ORION-327048: For VoC on GCP, it is recommended to have a multiple of 8 non-VMS VMs per cluster, since GCP has 8 failure domains and imbalance may result in failed stripe allocations.
• ORION-145141: Creating a tenant with EKM encryption is not supported on VoC clusters.
• ORION-113036: After you reregister the same VoC cluster in Uplink, information about the previously registered instance of this cluster is no longer available in Uplink.

Authentication and Authorization

• The following rules and limitations apply when using STS, IAM roles and OIDC providers:

  • STS is supported with HTTPS only.
  • Use of STS requires that the S3 protocol is enabled for the cluster.
  • One OIDC provider per VAST cluster tenant.
  • When replicating IAM roles:
    • ORION-278747: IAM role replication may take several minutes to complete in case the cluster is under high load or there is a very large amount of roles to replicate.
    • STS access keys are replicated only when the protection path is in SYNC state.
    • ORION-278217: When processing the assume role operation, the cluster may be raising a Failed to replicate STS temp keys alert until the synchronous replication stream/protection path reaches the SYNC state.
  • ORION-287563: Only five OIDC keys can be returned for a manual request to get or refresh OIDC keys.

• The following limitations apply when using netgroups:

  • Hosts should have both forward and reverse DNS entries. When VAST Cluster gets the netgroup hostname response from a NIS or LDAP server, it resolves the hostname via DNS.
  • Netgroups are only used to allow or deny clients' access via NFS. VAST Cluster does not accept netgroup entries in host-based access rules for other access protocols.

• The following limitations apply to Multi-Forest Authentication:

  • VAST Cluster does not allow adding two different Active Directory configuration records with the same domain name but different settings for multi-forest authentication and/or auto-discovery.
  • Names of users' domains are not displayed in data flow analytics.
  • If a trusted domain becomes unavailable and then recovers, SMB clients can use it to connect to the VAST cluster only after a period of time, but not immediately upon domain recovery.
  • Clients cannot establish SMB sessions immediately after a trusted domain recovers from a domain failure.
  • If a group exists on an Active Directory domain in a trusted forest and the group scope is defined as DomainLocal, VAST Cluster does not retrieve such a group when querying Active Directory, so members of such a group are denied access despite any share-level ACLs that can rule otherwise.
  • If TLS is enabled, the SSL certificate has to be a CA-signed certificate that is valid for all of the domain controllers in all trusted forests. If the certificate is not valid for a domain controller, this domain controller is not recognized.
  • ORION-156168: In a multi-forest environment, after migrating a group account from the forest of the cluster’s joined domain to another forest, information about historical group membership is not kept, so users in the migrated group might not be able to access resources to which they used to have access prior to the migration.

• The following requirements and limitations apply when using mTLS authentication for NFS:

  • mTLS authentication for NFS requires TLS 1.3.
  • The VASTNFS driver is required to use mTLS certificate-based tenant identification for NFSv3. (NFSv4 does not require VASTNFS.)
  • When using mTLS certificate-based tenant identification for NFSv3, the client must mount with mountproto=tcp.
  • Revocation of an mTLS certificate by using CRL Distribution Points (CDP) is not supported.
  • When using NFSv3, if a parent view does not have TLS or mTLS enforced, the VAST cluster allows unencrypted access to any of its child views, regardless of whether the child view has TLS or mTLS enforcement configured on it.
  • For clients using multiple certificates for different NFS exports (certificate per mount), Linux kernel 6.17 or later is required.
  • ORION-332649: A CRL file can contain up to 100 certificates to be revoked.

• The following limitations apply when using Kerberos/NTLM authentication:

  • ORION-143944: When using Kerberos/NTLM Authentication to authorize SMB users from non-trusting domains, the DOMAIN\username format cannot be used to specify users of remote domains. The username@domain format must be used instead.
  • ORION-141763: Before enabling or disabling NTLM authentication, you need to leave the cluster's joined Active Directory domain. After NTLM authentication is enabled or disabled, rejoin the domain.
  • ORION-134299: When the tenant is set to use Kerberos/NTLM authentication to authorize SMB users from non-trusting domains, both NFS and SMB must use the native SMB authentication (Kerberos), and not Unix-style UID/GIDs.

• The following limitations apply when using an allow-list of Active Directory DCs and GCs for LDAP queries:

  • The list must include at least one Global Catalog.
  • The list can contain up to 10 entries.
  • The Active Directory provider must have multi-forest authentication disabled.

• ORION-344017: When using NFSv3, if a parent view does not have TLS or mTLS enforced, the VAST cluster allows unencrypted access to any of its child views, regardless of whether the child view has TLS or mTLS enforcement configured on it.

• ORION-202335: If the cluster has Active Directory domain auto-discovery enabled, the discovered domains are kept in cache for quite a long time. If you modify an existing provider's configuration while auto-discovery is on, VMS may still report the old cached entries. To avoid this, rerun auto-discovery or remove and re-add the provider.

• ORION-195524: Following a cluster recovery and while the Active Directory provider is still inaccessible, VAST Cluster can resume IO of provider users if they use NFSv3 or NFSv.4.1 with NTLM authentication. IO of provider users accessing through SMB or NFSv4.1 with Kerberos authentication is not resumed during this period.

• ORION-187136: Identity policies are replicated as disabled to the destination peer, where if needed, they can be enabled manually.

• ORION-152475: An access denied error is returned for NFSv3 or NFSv4 requests if they are checked against an identity or bucket policy with an s3:ExistingObjectTag condition statement in it.

VMS

• Tenant client metrics can only be collected for NFSv3 and NFSv4.
• The following limitations apply when installing a SSL certificate for VMS:
  • Only RSA-generated public keys are supported.
  • Password-protected private keys are not supported.
• When setting a VMS login banner text via the VAST CLI, multiple lines are not supported.
• ORION-273819: Duplicate entries of predefined analytic report names may appear after upgrading to a next VAST cluster release. These entries appear only for reports in which the reported metrics have changed between the releases. A report marked with (1) contains the report version from the latest release.
• ORION-212118: If a wrong VMS token is passed, the cluster responds with 403 FORBIDDEN but not with 401 UNAUTHORIZED.
• ORION-187584: An empty realm (which does not contain any objects) cannot be assigned to a role.
• ORION-131386: When there is a parent directory that has a very large number of child directories, a total of children’s capacity values displayed in the Capacity page can exceed the capacity value shown for the parent directory.

VAST Web UI

• The visual identity policy builder in VAST Web UI (User Management -> Identity Policies -> choose to create or edit an identity policy) does not include predefined action statements for Kafka-related objects.
• ORION-249325: When you create the first block view on the cluster via VAST CLI or VAST REST API, the VAST Web UI does not display the Element Store -> Volumes or Hosts tabs until you refresh the page.

VAST REST API

• When listing views through the VMS API, the resulting list is truncated to the first 16,000 entries. To obtain a complete listing, use multiple requests with pagination.

Platform and Control

• The following limitations apply to conversion to write buffer RAID:

  • Conversion from VAST releases prior to 3.4 is not supported.

  • This capability is not supported for clusters with TLC drives, and also for VAST on Cloud clusters.

  • The cluster must include the following minimum number of DBoxes:

    DBox Type	DBox HA enabled	DBox HA disabled
    Ceres	15	4
    Mavericks	22	4

• The following rules and limitations apply when using Rack-Level Resiliency:

  • Every DBox must be associated with a single failure domain.
  • At least seven failure domains must be defined.
  • The number of DBoxes in the rack cannot exceed the number in any other rack by more than one.
  • The total available DBox capacity in a rack cannot be more than twice the available capacity in any other rack.
  • If two of the racks in a rack level-resilient cluster go down, bringing up only one of them does not return the cluster to normal operation. The cluster restores service only after the second of the racks goes up.

• The following limitations apply to metadata rewrite (required during upgrade to VAST Cluster 5.5.0):

  • A metadata rewrite process cannot be run while there is another active rewrite, upgrade, expansion, migration, replacement or removal process running. An upgrade to a major version would be blocked if attempted during metadata rewrite.
  • Metadata rewrite can only be started if the cluster's handle usage is less than 70% and the metadata block usage is less than 80% (these metrics are shown in the Cluster MD Usage analytics report).

• Use of flash write buffers with DBox HA capability enabled requires at least 10 boxes.

• The following limitations apply to EBoxes:

  • ORION-193794: Power cycling of an EBox where the leader was running may result in significant IOPS degradation until the EBox is up again. Contact VAST Support for a workaround.
  • DBox migration is not available for EBoxes.

• The following rules and limitations apply when using DBox removal:

  • A DBox removal process can only be started after metadata rewrite is complete (metadata rewrite is required when upgrading to VAST Cluster 5.5). See Limitations for metadata rewrite.
  • DBox removals reducing the cluster size below the initial installation size may not be allowed.
  • Once the DBox removal process has started, it cannot be stopped.
  • Only one DBox removal process can be run at a time. The process can involve one or more DBoxes.
  • DBox removal cannot be performed at the same time when DBox migration is running (even if it is for a different DBox).
  • DBox removal cannot be performed at the same time when DBox replacement is running.
  • During DBox removal, it is not allowed to start a DBox expansion process. Adding or removing CBoxes is allowed.
  • During DBox removal, the Write Buffers on Flash functionality is disabled.
  • DBox removal is not supported after a Write Buffer RAID rewrite.
  • If the cluster has DBox HA enabled, removing a DBox is not allowed if it would reduce the cluster below the minimum number of DBoxes required for DBox HA.
  • If the cluster has rack resiliency enabled, the DBoxes to be removed must be balanced across all failure domains so that the removal does not impact failure domain balancing. Alternatively, the DBoxes to be removed must constitute a single failure domain.

Callhome and Support

• An attempt to delete a support bundle before it is completely created may cause unexpected behavior.
• ORION-337635: In case of VMS/leader movement at the time prior to support bundle collection, the support bundle may include logs from both old and new nodes hosting the service. However, old nodes are not indicated as such in the log filenames. To differentiate between old and new nodes, use the timestamps within the logs.

Known Issues

Install and Upgrade

• ORION-394488: When upgrading a cluster with block volumes from 5.4.x to 5.5.x, the existing volumes may be shown with a namespace ID of 0 instead of their actual namespace IDs during the upgrade. When the upgrade completes, the correct namespace IDs are shown for all block volumes.

Cluster Expansion

• ORION-220738: In some cases, VMS does not provide any alerts or other status indication when a drive gets disabled while the newly added DBox is being initialized.
• ORION-175762: In some cases, a DBox expansion procedure run on a cluster with similarity-based data reduction enabled can take longer than expected.

Quality of Service (QoS)

• ORION-337326: QoS burst metrics calculated for views or volumes subject to view/volume QoS policy limits, may in some cases show incorrect values.
• ORION-139913: When applying a QoS policy to NFSv3 access, both data and metadata are taken into account in QoS limit calculations, while with NFSv4.1, only data are considered.
• ORION-137986: Enabling a QoS policy for a view on which a mixed (read and write) workload runs, can result in decreased performance for the workload.

SMB

• ORION-402172: Sporadic SMB disconnects may be encountered during SMB server-side copy.
• ORION-142968: If a quota is exceeded during the process of coping a file to the VAST cluster, the copying process is stopped with a misleading error message: A device attached to the system is not functioning.

S3

• ORION-423552: A PutObject request containing the If-None-Match: * header returns InvalidArgument when using chunked encoding or CRC64NVME.

Attribute-Based Access Control (ABAC)

• ORION-196170: When a parent and child NFSv4.1 view both have same ABAC tags on them, an attempt to mount the child view may result in a permission denied error. If this occurs, try setting the ABAC tags for the machine account that the client uses to mount the view.

VAST Audit Log

• ORION-393002: The VAST Audit Log page in VAST Web UI (DataBase -> VAST Audit Log) may show raw values in some fields. These values are taken from the cluster's internal database and can be negative for some S3 operations. For example, the Path/Handle ID field will show a negative value for a CreateMultipartUpload operation since there is no active handle associated with the operation.

VAST DataEngine

• ORION-389933: Scheduled triggers cannot start pipelines while the VMS container is not running (which may occur during NDU or HA events). If the VMS is down at the time when a pipeline is about to be triggered by a scheduled trigger, the pipeline will not start. After the VMS is back up, the pipeline will be triggered as expected. Note that the issue occurs with scheduled triggers only. Element triggers work regardless of the VMS container state.
• ORION-383456: When removing a tag from a trigger in the Update Trigger dialog (opened from the Manage Elements -> Triggers page), the tag does not get removed from the trigger in the backend. Use VAST DataEngine CLI or API to remove tags from triggers.
• ORION-358149: Inappropriate combination of batch event timeout and function timeout settings may result in duplicate entries appearing in the batch event traces. The function timeout must be set to a value exceeding the batch timeout (600 seconds by default).

Replication

• ORION-391617: Simultaneously creating S3 buckets on 2 clusters configured to run synchronous replication with Bucket Replication enabled, where the bucket paths are being synchronously replicated to the other cluster, may fail. Either create both buckets on one cluster, or wait until the first bucket creation completes before starting the second.
• ORION-140894: When attempting to delete a protected path from the destination peer after an ungraceful failover, a Failed to delete following streams or a similar error occurs. The workaround is to manually change the destination peer's role to STANDALONE and retry the deletion.

Global Access

• ORION-145307: Bulk permission updates are not supported for files and directories on satellite clusters.

Authentication and Authorization

• ORION-353874: A Setting mtls certificate for protocol NFS failed: CA certificate chain is empty error occurs when trying to upload a trust chain to tenant's NFS mTLS settings (Element Store -> Tenants -> choose to create or edit a tenant -> Advanced Protocol Settings tab -> Authentication settings for mTLS pane) where the root CA is not the first in the chain.
• ORION-316491: When the bucket has S3 event notifications enabled, no event notification is produced for a copy-to-self operation (modifying object metadata).
• ORION-25479: Latin characters are not supported with LDAP. If you attempt to pass, for example, a username encoded with a Latin character set, the LDAP sanity check res: Invalid credentials error is returned.

VMS

• ORION-398271: The Block Volume IOPS by Size metrics show 0 in case there are less than 15 operations within a 15-second time slot.
• ORION-300508: Some metrics may not be available in VAST Prometheus Exporter. Missing metrics include (but may be not limited to) CNode and DNode hardware metrics or CPU temperature, utilization, memory percentage, PCI error count, retransmitted segments, total correctable/uncorrectable memory errors. If you encounter this issue, contact VAST Support for a workaround.
• ORION-143717: On a cluster with CNode Port Affinity configured, there is no way to expose the VAST DNS IP on a specific port (left or right).
• ORION-131386: When there is a parent directory that has a very large number of child directories, a total of children’s capacity values displayed in the Capacity page can exceed the capacity value shown for the parent directory.
• ORION-89570: In some cases, capacity analytics for subdirectories cannot be reported due to an internal timeout. This issue occurs when there is an extremely large number of subdirectories to be estimated.

VAST Web UI

• ORION-424634: The Performance gauge on the VAST cluster dashboard shows the Kafka bandwidth using KB/s but not MB/s as expected.
• ORION-396255: The VMS does not display LED status for Turin CNodes. The corresponding field in the Infrastructure -> CNodes page shows Unknown or N/a or is always Off.
• ORION-396165: When Kafka is selected in the Performance pane on the cluster's dashboard, the Latency popup has an extra field named Avg that is always 0. This field can be ignored.
• ORION-395373: When viewing top actors among volumes (Analytics -> Top Actors -> Volumes selected), the right-side Top Actors pane may show entries with N/A instead of the volume name. Such entries do not represent real volumes and can be ignored.
• ORION-394425: In the Analytics -> Data Flow page, if the Sort field is set to BW, MD IOPS or Latency, selecting the Scan option in the By field causes the page to hang. Use the Scan option only when the Sort field is set to IOPS or ROW/s.
• ORION-371787: The Default Gateway field in managed application settings (Data Engine -> Applications -> choose to create or edit an application -> Network tab -> Advanced pane ) does not work as expected.
• ORION-308654: When using the Export selected rows as CSV option of the VAST Web UI, the resulting file may contain all of the rows, including those that were not selected for the export.
• ORION-294888: The Infrastructure pages of VAST Web UI may display [object Object] instead of the actual CPLD/UBM values for Dell Turin CNodes.
• ORION-234835: Some VAST Web UI pages might not allow for proper filtering or sorting by column where value presentation differs from that in the VAST internal database.
• ORION-203189: The External Netmask field in cluster networking settings (Settings -> Configure Network) does not accept alphabetic characters.

VAST CLI

• ORION-363096: When associating CNodes with a virtual IP pool using the vippool create or vippool modify command, there is no validation of whether the CNode count is sufficient, which could result in creating a pool that lacks the required number of CNodes.
• ORION-322120: When displaying path names in VAST CLI, the names written in right-to-left languages may appear spelled from left to right.
• ORION-265720: VAST CLI auto-completion does not include the --detach-krb-provider option on the tenant modify command.

VAST REST API

• ORION-178569: The /users/names endpoint always returns only the first 50 entries, regardless of the page size parameter or the total amount of entries to be returned.

Platform and Control

• ORION-432988: Indexes for HPE Genoa EBox slots, SSDs and SCMs in the respective Infrastructure pages start with 0 instead of 1.
• ORION-403644: If an administrative role has OS SSH login enabled (Administrators -> Administrative Roles -> select a role -> Enable OS SSH Login option), an attempt to enable OS SSH login for a second role may result in an internal timeout, which also causes the feature to stop working for the first role.
• ORION-255054: Once a DNode replacement operation is complete, the old node may still be listed in the Infrastructure -> DNodes page of VAST Web UI for some minutes.

Callhome and Support

• ORION-269349: When obfuscating a support bundle, the obfuscation is not applied to files that have already been zipped.
• ORION-239170: When obfuscating a support bundle, the CNode hostname may not get obfuscated in some of the logs included in the bundle.