--- title: "vLLM (NVIDIA)" slug: "vllm-nvidia" description: "Optimize LLM inference with vLLM OffloadingConnector by leveraging VAST Data for efficient KV cache management and improved production performance." updated: 2026-10-28T02:14:35Z published: 2026-10-28T02:14:35Z canonical: "docs.vastdata.com/vllm-nvidia" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt > Use this file to discover all available pages before exploring further. # vLLM (NVIDIA) ## vLLM OffloadingConnector As long-context LLM inference scales, GPU High Bandwidth Memory (HBM) rapidly becomes the primary bottleneck due to linear Key-Value (KV) cache expansion. Integrating **vLLM’s native OffloadingConnector**, host DRAM, and VAST Data creates a disaggregated path that spills KV from GPU → CPU → NFS, bypassing GPU memory bounds and cutting warm Time-To-First-Token (TTFT) when prefixes hit the CPU/FS tiers. This KB guide details how to configure vLLM OffloadingConnector against high-performance VAST storage volumes over RDMA-enabled fabrics for production inference — **without LMCache**. ## Prerequisites Please make sure you have completed the [File Prerequisites](/develop/docs/file-prerequisites) before starting this guide, as it can drastically impact performance. ## Motivation **GPUs**: The testing environment utilizes 2x NVIDIA RTX PRO 6000 Blackwell - **Network**: - The host is equipped with 2x Mellanox CX-7 200GbE Single-Port NICs - The network configuration for VAST storage is [NFSv3 over RDMA](https://www.vastdata.com/blog/meet-your-need-for-speed-with-nfs) - **Storage**: The team tested two primary configurations for KV cache offloading: - Local host: OffloadingConnector CPU DRAM tier (`/dev/shm` mmap) - Remote VAST Data storage: FS secondary tier on a VAST partition via NFSv3 over RDMA, with NFS multipath - **Software**: - **vLLM:** Version **0.28.0** - **NVIDIA CUDA:** proprietary accelerated computing platform with programming models, compilers, libraries, runtimes, and deployment tools. - **Model:** Llama-4-Scout-17B-16E-Instruct-FP8, TP-2 We evaluated a tp=2 configuration across concurrency levels, tuning the system according to the [File Prerequisites](/develop/docs/file-prerequisites) guidelines to maximize performance: - nconnect: Set to 32 - Read-ahead increased to 8192 - vLLM OffloadingConnector optimized configurations, detailed below - Rank-local host pin baked into the image (required for large CPU tiers) ## Results TTFT was measured with and without the VAST storage backend (Layer 2). Both setups utilize L0/L1 caching (GPU & CPU). ~35x Warm tok/s gain ![~35x Warm tok/s gain](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/image-20260911-155441.png?sv=2026-02-06&spr=https&st=2026-11-08T09%3A14%3A23Z&se=2026-11-08T10%3A14%3A23Z&sr=c&sp=r&sig=XX4vcQ0Go%2B4wzSQFqd8L7%2FdBWZ0AV4S4qsJEKO0i6cw%3D) ~41x Warm mean TTFT gain ![~41x Warm mean TTFT gain](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/image-20260911-161345.png?sv=2026-02-06&spr=https&st=2026-11-08T09%3A14%3A23Z&se=2026-11-08T10%3A14%3A23Z&sr=c&sp=r&sig=XX4vcQ0Go%2B4wzSQFqd8L7%2FdBWZ0AV4S4qsJEKO0i6cw%3D) ## Software Stack We’ll be using the following NVIDIA Stack: 1. vLLM(*) - [vllm/vllm-openai:v0.28.0-cu129](https://hub.docker.com/layers/vllm/vllm-openai/v0.28.0-cu129/images/sha256-50509e700235cea487715cedeb501d20a1cd15fa6a54ce93688284bd0d96995d) (`vllm/vllm-openai@sha256:ac259a0111c6cf462a72e449962b84f7a624b5cbec24bd7d9ec3b67d40ffd1bf`) 2. Model - `nvidia/Llama-4-Scout-17B-16E-Instruct-FP8` ## Installation - NVIDIA We’ll be working in the working directory `/root/vastdata`, and have `cfg` and `models` underneath it. We’ll also be using the `/mnt/kvcache-2ports` from earlier, so be sure you have this tree ready ```plaintext BASE_DIR="/root/vastdata" MODEL_DIR="${BASE_DIR}/models/llama-4-scout" CACHE_DIR="/mnt/kvcache-2ports" CFG="${BASE_DIR}/cfg"   # contains start.sh mkdir -p ${BASE_DIR} ${MODEL_DIR} ${CFG} ``` Copy this package into `${BASE_DIR}` (Dockerfile, `patches/`, `cfg/start.sh`, `run.sh`). Downloading the model locally ```plaintext python3 -m venv "${BASE_DIR}/.venv" source "${BASE_DIR}/.venv/bin/activate" pip install "huggingface_hub[cli]" hf download nvidia/Llama-4-Scout-17B-16E-Instruct-FP8 \  --local-dir "${MODEL_DIR}" ``` ## Why the Dockerfile patches two files Stock(Unpatched) vLLM `pin_mmap_region` registers the **entire** shared `/dev/shm` offload mmap in **every** tensor-parallel rank. At TP=8 with a 1600 GiB CPU tier that asks the driver for ~12.8 TiB of pinned mappings. On ROCm a failed `cudaHostRegister` leaves a sticky `hipErrorInvalidValue`; the next GPU op then crashes (often far from the pin site). The files under `patches/` change that to **rank-local** registration (page-aligned strided slots) and matching unregister in `cleanup`. Aggregate pinned bytes stay ~tier size. This is a **source replacement in the image**, not a runtime one (a layer that is applied in our Dockerfile). Look for this line once per rank after engine start: ```plaintext Rank-local host registration complete: rank=N rows=... ... GiB (shared region ... GiB) ``` ## Scripts ### `./cfg/start.sh` ```bash #!/usr/bin/env bash set -euo pipefail CPU_GB="${VLLM_OFFLOAD_CPU_GB:-160}" CPU_BYTES=$((CPU_GB * 1024 * 1024 * 1024)) FS_ROOT="${VLLM_OFFLOAD_FS_ROOT:-/kvcache}" KV_TRANSFER_CONFIG=$(cat <