--- title: "vLLM (AMD)" slug: "vllm-amd" description: "Optimize LLM inference with vLLM OffloadingConnector and VAST Data, enhancing performance by offloading KV cache from GPU to CPU and NFS." updated: 2026-10-28T02:14:35Z published: 2026-10-28T02:14:35Z canonical: "docs.vastdata.com/vllm-amd" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt > Use this file to discover all available pages before exploring further. # vLLM (AMD) ## vLLM OffloadingConnector AMD As long-context LLM inference scales, GPU High Bandwidth Memory (HBM) rapidly becomes the primary bottleneck due to linear Key-Value (KV) cache expansion. Integrating **vLLM’s native OffloadingConnector**, host DRAM, and VAST Data creates a disaggregated path that spills KV from GPU → CPU → NFS, bypassing GPU memory bounds and cutting warm Time-To-First-Token (TTFT) when refixes hit the CPU/FS tiers. This KB guide details how to configure vLLM OffloadingConnector against high-performance VAST storage volumes over RDMA-enabled fabrics for production inference — **without LMCache**. ## Prerequisites Please make sure you have completed the [File Prerequisites](/develop/docs/file-prerequisites) before starting this guide, as it can drastically impact performance. ## Motivation **GPUs**: The testing environment utilizes an **AMD MI350X** node. - **Network**: - The host is equipped with 2x Mellanox CX-7 400GbE Single-Port NICs - The network configuration for VAST storage is [NFSv3 over RDMA](https://www.vastdata.com/blog/meet-your-need-for-speed-with-nfs) - **Storage**: The team tested two primary configurations for KV cache offloading: - Local host: OffloadingConnector CPU DRAM tier (`/dev/shm` mmap) - Remote VAST Data storage: FS secondary tier on a VAST partition via NFSv3 over RDMA, with NFS multipath - **Software**: - **vLLM:** Version **0.28.0** (ROCm) - **AMD ROCm:** open foundation with programming models, compilers, libraries, runtimes, and deployment tools - **No LMCache** — `OffloadingConnector` + `TieringOffloadingSpec` only - **Model:** [GPT-oss](https://huggingface.co/amd/gpt-oss-120b-w-mxfp4-a-fp8) 120B, TP-8 (Maverick FP8 / TP-2 smokes use the same image and connector wiring) We evaluated a TP=8 configuration across concurrency levels, tuning the system according to the [File Prerequisites](/develop/docs/file-prerequisites) guidelines to maximize performance: - nconnect: Set to 32 - Read-ahead increased to 8192 - vLLM OffloadingConnector optimized configurations, detailed below - Rank-local host pin baked into the image (required for large CPU tiers on ROCm) ## Results TTFT was measured with and without the VAST storage backend (G4). Both setups utilize G1 & G2 caching (GPU & CPU), and neither uses G3 (Local SSD). ~6x Warm tok/s gain ![~6x Warm tok/s gain](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/image(212).png?sv=2026-02-06&spr=https&st=2026-11-08T09%3A14%3A22Z&se=2026-11-08T10%3A14%3A22Z&sr=c&sp=r&sig=exRKp5Hc%2BbPybplQJKJ2gVqJGDhnA405GK0gvdDnjNo%3D) ~10x Warm mean TTFT gain ![~10x Warm mean TTFT gain](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/image(211).png?sv=2026-02-06&spr=https&st=2026-11-08T09%3A14%3A22Z&se=2026-11-08T10%3A14%3A22Z&sr=c&sp=r&sig=exRKp5Hc%2BbPybplQJKJ2gVqJGDhnA405GK0gvdDnjNo%3D) ## Software Stack We’ll be using the following AMD Stack: 1. vLLM(*) - [vllm/vllm-openai-rocm:v0.28.0](https://hub.docker.com/layers/vllm/vllm-openai-rocm/v0.28.0/images/sha256-e0a3b2bd3fe7ec563916c3a5d949898d133458c18d6b2f460c906885cfb32032) (`vllm/vllm-openai-rocm@sha256:e0a3b2bd3fe7ec563916c3a5d949898d133458c18d6b2f460c906885cfb32032`) 2. Model - `amd/gpt-oss-120b-w-mxfp4-a-fp8` 3. Image overlay - `vllm-fs:0.28.0-rocm-rankpin` (this package’s Dockerfile; two patched files only) ## Installation - AMD We’ll be working in the working directory `/root/vastdata`, and have `cfg` and `models` underneath it. We’ll also be using the `/mnt/kvcache-2ports` from earlier, so be sure you have this tree ready ```plaintext BASE_DIR="/root/vastdata" MODEL_DIR="${BASE_DIR}/models/gpt-oss-120b-w-mxfp4-a-fp8" CACHE_DIR="/mnt/kvcache-2ports" CFG="${BASE_DIR}/cfg"   # contains start.sh mkdir -p ${BASE_DIR} ${MODEL_DIR} ${CFG} ``` Copy this package into `${BASE_DIR}` (Dockerfile, `patches/`, `cfg/start.sh`, `run.sh`). Downloading the model locally ```plaintext python3 -m venv .venv source .venv/bin/activate pip install "huggingface_hub[cli]" hf download amd/gpt-oss-120b-w-mxfp4-a-fp8 --local-dir "${MODEL_DIR}" ``` ## Why the Dockerfile patches two files Stock(Unpatched) vLLM `pin_mmap_region` registers the **entire** shared `/dev/shm` offload mmap in **every** tensor-parallel rank. At TP=8 with a 1600 GiB CPU tier that asks the driver for ~12.8 TiB of pinned mappings. On ROCm a failed `cudaHostRegister` leaves a sticky `hipErrorInvalidValue`; the next GPU op then crashes (often far from the pin site). The files under `patches/` change that to **rank-local** registration (page-aligned strided slots) and matching unregister in `cleanup`. Aggregate pinned bytes stay ~tier size. This is a **source replacement in the image**, not a runtime one (a layer that is applied in our Dockerfile). Look for this line once per rank after engine start: ```plaintext Rank-local host registration complete: rank=N rows=... ... GiB (shared region ... GiB) ``` ## Scripts ### `./cfg/start.sh` ```bash #!/usr/bin/env bash set -euo pipefail CPU_GB="${VLLM_OFFLOAD_CPU_GB:-1600}" CPU_BYTES=$((CPU_GB * 1024 * 1024 * 1024)) FS_ROOT="${VLLM_OFFLOAD_FS_ROOT:-/kvcache}" KV_TRANSFER_CONFIG=$(cat <