A data scientist or machine learning (ML) engineer uses data on the VAST AI OS to train, tune, and serve models. They need fast, reliable access to large datasets over NFS and S3, and they want to find and reuse the right data without waiting on infrastructure teams. On VAST, they store and search embeddings with vector search in the VAST DataBase, locate datasets through the VAST Catalog, build Python functions with the DataEngine Runtime SDK to automate preprocessing and inference, and recover earlier versions of datasets or checkpoints from snapshots.
Common Data Scientist/ML Engineer Tasks
Use NFS performance features: Get the most throughput from NFS clients for training and data loading.
Configure S3 client interfaces: Connect S3 tools and libraries to buckets on the cluster.
Share S3 data with presigned URLs: Give collaborators and applications temporary access to objects.
Access the VAST Catalog from clients: Query file and object metadata to find and curate datasets.
Run vector search: Find similar items by querying embeddings stored in the VAST DataBase.
Create a vector database index: Speed up similarity queries on large embedding tables.
Get started with the DataEngine Runtime SDK: Learn how to write code that runs on VAST DataEngine.
Create a DataEngine function: Package preprocessing or inference code as a reusable function.
Access secrets and environment variables: Pass credentials and configuration to your functions securely.
Restore a file or directory: Recover earlier versions of datasets or model checkpoints from snapshots.