Snapshots and replication

Applies to VAST Cluster 5.5. Examples use the VAST DataBase Python SDK 2.1.0.

Your platform team can take snapshots of the path that holds your database, and can replicate it to other clusters. This page is for the reader's side: how to query a table as it was in a snapshot, and what asynchronous replication means for the data you see on a replica. Setting up snapshots and replication is an admin task, covered in the KB.

Snapshots

A snapshot is a read-only, point-in-time copy of a path on the cluster. If it covers your database, its tables are in it exactly as they were when the snapshot was taken (Snapshots). Use snapshots to:

• Query yesterday's data, or the data from before a bad load, while the live table keeps changing.
• Compare the live table with an earlier state.
• Recover rows you deleted by mistake: read them from the snapshot, and insert them into the live table.

Partitioned tables work with snapshots (Database partitioning).

Admin task: send to your platform team. Snapshots are taken by hand or on a schedule, from a protection policy. Ask for snapshots of the path of your database, and tell the team how far back you need to go. Procedure: Managing Snapshots.

How snapshots are kept (see Snapshots):

• A snapshot costs space only for what changed. It holds on to data that would otherwise be released: rows you delete or overwrite after the snapshot was taken. A table that changes a lot keeps more data alive in each snapshot.
• Snapshots don't count against quotas.
• The policy sets how many you have. You can't cap a snapshot's size; you choose how often snapshots are taken and how many are kept.
• Some snapshots can't be deleted. A snapshot flagged indestructible stays until it expires.

Find a snapshot

Connect the same way as in the quickstart. In the Python SDK, snapshots belong to a bucket. bucket.snapshots() lists them, and bucket.snapshot(name) gets one by name:

import os
import vastdb

session = vastdb.connect()
BUCKET = os.environ["VASTDB_BUCKET"]

with session.transaction() as tx:
    for snap in tx.bucket(BUCKET).snapshots():
        print(snap.name)

Each snapshot is returned as a Bucket, named <bucket>/.snapshot/<snapshot name>. The list can include snapshots you didn't ask for. For example, if the VAST Catalog is enabled, it takes its own snapshots, named big_catalog_<date and time>.

If you ask for a snapshot that doesn't exist, snapshot() raises MissingSnapshot, or returns None if you pass fail_if_missing=False:

import os
import vastdb
from vastdb import errors

session = vastdb.connect()
BUCKET = os.environ["VASTDB_BUCKET"]

with session.transaction() as tx:
    bucket = tx.bucket(BUCKET)
    print(bucket.snapshot("no-such-snapshot", fail_if_missing=False))
    try:
        bucket.snapshot("no-such-snapshot")
    except errors.MissingSnapshot as e:
        print(type(e).__name__, "snapshot:", e.snapshot)

None
MissingSnapshot snapshot: .snapshot/no-such-snapshot

Read a table in a snapshot

A snapshot Bucket works like the live one: walk down to the schema and table, and query them with the same select() calls. Replace <snapshot name>, <schema> and <table> with your own:

with session.transaction() as tx:
    snap = tx.bucket(BUCKET).snapshot("<snapshot name>")
    then = snap.schema("<schema>").table("<table>").select().read_all()

    live = tx.bucket(BUCKET).schema("<schema>").table("<table>").select().read_all()

print("Rows in the snapshot:", then.num_rows)
print("Rows now:", live.num_rows)

A schema or table that was created after the snapshot was taken isn't in it, so schema() or table() raises MissingSchema or MissingTable. Filters, column selection and internal_row_id=True work as on the live bucket, because a snapshot Bucket gives you the same table objects.

To restore deleted rows, read them from the snapshot and insert them into the live table:

with session.transaction() as tx:
    snap_table = tx.bucket(BUCKET).snapshot("<snapshot name>").schema("<schema>").table("<table>")
    lost = snap_table.select(predicate=snap_table["id"] == 42).read_all()

with session.transaction() as tx:
    tx.bucket(BUCKET).schema("<schema>").table("<table>").insert(lost)

For more than a few rows, your platform team has two tools that work on a whole protected path (Snapshots):

• Fast Restore puts a path back as it was in a snapshot, local or replicated from a peer.
• Global snapshot cloning makes an instantly writable clone of a snapshot, on the same cluster or on a replication peer.

Both act on everything under the path, not on one table. Ask your platform team how they treat a database before you rely on them. Admin task: send to your platform team. See Fast Restore.

Snapshots are read-only

You can't change anything through a snapshot. Creating a schema, inserting rows or dropping a table there fails with vastdb.errors.Forbidden (403), code AccessDenied.

A transaction that has tried to write through a snapshot then fails to commit, also with Forbidden (403) AccessDenied, and loses the changes it made to the live bucket. Keep snapshot reads and live writes apart: read from the snapshot in one transaction, and write to the live table in another, as in the restore example above.

The VAST Catalog has its own snapshots, which you list with tx.catalog_snapshots(). See snapshots and change detection.

Replicas

With asynchronous replication, the cluster copies a protected path to one or more other clusters on a schedule. A database on a protected path is replicated like files and buckets: periodic snapshots of it are sent to the destination clusters as read-only images. If the platform team fails over to a destination, it becomes writable (Overview of VAST Replication).

How replication runs (see Overview of VAST Replication):

• Schedule, or on demand. A protection policy sets how often replication runs. The platform team can also replicate a path on demand at any time, for example after a big load that a replica must see.
• Several destinations. A path can be replicated to more than one cluster, and several clusters can replicate to one.
• Failover and fail-back. A destination can take over as the source, gracefully while the source is reachable, or forced if it isn't. The platform team can also suspend and resume replication.
• Breaking replication. If a destination stops replicating, it becomes standalone, and its copy of the data no longer receives updates.
• Across tenants. A path in one tenant can be replicated to a path in another tenant on the remote cluster, so the database's tenant on the destination can differ.

Synchronous replication. Databases are replicated with asynchronous replication. Synchronous replication, which copies every write at once, is for S3 buckets: a path can be switched to synchronous only if S3 is its only protocol. Whether that includes a database path isn't documented. Don't plan on synchronous replication of a database without confirming it with your platform team.

What a replica contains

These apply to replicated databases:

Some tables can't be replicated at all:

If you need a table on a replica, don't make it sorted, partitioned or vector-indexed. Decide this before you create the table; see limits and data modeling.

Reading from a replica

To read a replica, connect the SDK to an endpoint on the destination cluster, with credentials that work there:

• Endpoint: a virtual IP pool on the destination cluster that serves the database. See setup prerequisites.
• Bucket: with bucket replication turned on for the protected path, the bucket is recreated on the destination with the source's properties, including whether it's a database. Without it, the platform team has to create it (S3 Access to Replicated Data).
• Credentials: access keys and identity policies of users from an external provider, such as Active Directory, are replicated. Keys of users on the cluster's own VAST provider aren't. Replicated identity policies are turned off on the destination by default, so the platform team has to turn them on (S3 Access to Replicated Data).

Don't write to a replica before a failover: it's read-only.

Admin task: send to your platform team. Replication setup: Overview of VAST Replication and the Replication section. S3 and database access on the destination: S3 Access to Replicated Data.

What to ask your platform team

Snapshots
- Database (bucket) and path: <name>
- Snapshot schedule and how long to keep them: <for example, hourly for 7 days>
- Naming, so we can find a snapshot by date: <convention>

Replication (if needed)
- Destination cluster and the endpoint (VIP pool) we should use there
- Replication interval: <how far behind the replica may be>
- On-demand replication after big loads: <yes / no>
- Asynchronous or synchronous replication for this path
- Bucket replication turned on for the protected path
- Credentials that work on the destination (external-provider users, or keys created there)
- Tables we need replicated (they can't be sorted, partitioned or vector-indexed): <list>