---
title: "vLLM & LMCache In-Process (NVIDIA)"
slug: "vllm-lmcache-in-process-nvidia"
description: "Optimize LLM inference with vLLM and LMCache, leveraging VAST Data for high-performance storage and reduced Time-To-First-Token by up to 10x."
updated: 2026-10-28T02:14:35Z
published: 2026-10-28T02:14:35Z
canonical: "docs.vastdata.com/vllm-lmcache-in-process-nvidia"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt
> Use this file to discover all available pages before exploring further.

# vLLM & LMCache In-Process (NVIDIA)

## vLLM & LMCache In-Process

As long-context LLM inference scales, GPU High Bandwidth Memory (HBM) rapidly becomes the primary bottleneck due to linear Key-Value (KV) cache expansion. Integrating vLLM, LMCache, and VAST Data creates an ideal disaggregated infrastructure that offloads KV cache to a scale-out storage fabric, bypassing GPU memory bounds and slashing Time-To-First-Token (TTFT) by up to 10x.

This KB guide details how to configure LMCache and vLLM to mount high-performance VAST storage volumes over RDMA-enabled fabrics for production inference.

## Prerequisites

Please make sure you have completed the [File Prerequisites](/develop/docs/file-prerequisites) before starting this guide, as it can drastically impact performance.

## Motivation

**GPUs**: The testing environment utilizes 2x NVIDIA RTX PRO 6000 Blackwell

- **Network**:
  - The host is equipped with 2x Mellanox CX-7 200GbE Single-Port NICs
  - The network configuration for VAST storage is [NFSv3 over RDMA](https://www.vastdata.com/blog/meet-your-need-for-speed-with-nfs)
- **Storage**: The team tested two primary configurations for KV cache offloading:
  - Local host: OffloadingConnector CPU DRAM tier (`/dev/shm` mmap)
  - Remote VAST Data storage: FS secondary tier on a VAST partition via NFSv3 over RDMA, with NFS multipath
- **Software**:
  - **vLLM:** Version **0.28.0**
  - LMCache: Version **0.53**
  - **NVIDIA CUDA:** proprietary accelerated computing platform with programming models, compilers, libraries, runtimes, and deployment tools
- **Model:** Llama-4-Scout-17B-16E-Instruct-FP8, TP-2

We evaluated a tp=2 configuration across concurrency levels, tuning the system according to the [File Prerequisites](/develop/docs/file-prerequisites) guidelines to maximize performance:

- nconnect: Set to 32
- Read-ahead increased to 8192
- vLLM OffloadingConnector optimized configurations, detailed below
- Rank-local host pin baked into the image (required for large CPU tiers on ROCm)

## Results

TTFT was measured with VAST storage backend (Layer 2) and without. Both setups utilize L0/L1 caching (GPU & CPU).

~35x Warm tok/s gain

![~35x Warm tok/s gain](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/image-20260911-155441(2).png?sv=2026-02-06&spr=https&st=2026-11-08T08%3A55%3A48Z&se=2026-11-08T09%3A55%3A48Z&sr=c&sp=r&sig=CfaBvfRXWDrLCU8%2BcxNpLDQnWKfLd9s8Yv8Q1WP%2BzZ4%3D)

~36x Warm mean TTFT gain

![~36x Warm mean TTFT gain](https://cdn.us.document360.io/89b91113-5dc6-49db-86ec-57b16dc8ce56/Images/Documentation/image-20260911-161243.png?sv=2026-02-06&spr=https&st=2026-11-08T08%3A55%3A48Z&se=2026-11-08T09%3A55%3A48Z&sr=c&sp=r&sig=CfaBvfRXWDrLCU8%2BcxNpLDQnWKfLd9s8Yv8Q1WP%2BzZ4%3D)

## Software Stack

We’ll be using the following NVIDIA Stack:

1. vLLM(*) - [vllm/vllm-openai:v0.28.0-cu129](https://hub.docker.com/layers/vllm/vllm-openai/v0.28.0-cu129/images/sha256-50509e700235cea487715cedeb501d20a1cd15fa6a54ce93688284bd0d96995d)

(`vllm/vllm-openai@sha256:ac259a0111c6cf462a72e449962b84f7a624b5cbec24bd7d9ec3b67d40ffd1bf`)
2. LMCache 0.53 built from source
3. Model - `nvidia/Llama-4-Scout-17B-16E-Instruct-FP8`

## Installation - NVIDIA

We’ll be working in the working directory `/root/vastdata`, and have `cfg` and `models` underneath it. We’ll also be using the `/mnt/kvcache-2ports` from earlier, so be sure you have this tree ready

```plaintext
BASE_DIR="/root/vastdata"
MODEL_DIR="${BASE_DIR}/models/llama-4-scout"
CACHE_DIR="/mnt/kvcache-2ports"
CFG="${BASE_DIR}/cfg"   # contains lmcache.yaml + start.sh
mkdir -p ${BASE_DIR} ${MODEL_DIR} ${CFG}
```

Downloading the model locally

```plaintext
python3 -m venv "${BASE_DIR}/.venv"
source "${BASE_DIR}/.venv/bin/activate"
pip install "huggingface_hub[cli]"
hf download nvidia/Llama-4-Scout-17B-16E-Instruct-FP8 \
  --local-dir "${MODEL_DIR}"
```

## Scripts

### `./cfg/start.sh`

```bash
#!/usr/bin/env bash
set -euo pipefail

exec vllm serve /models/model \
  --host 0.0.0.0 --port 8000 \
  --served-model-name llama-4-scout \
  --tensor-parallel-size 2 \
  --max-model-len 131072 \
  --max-num-seqs 300 \
  --gpu-memory-utilization 0.92 \
  --enable-prefix-caching \
  --block-size=64 \
  --no-disable-hybrid-kv-cache-manager \
  --kv-transfer-config \
  '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_both"}'
```

### `./cfg/lmcache.yaml`

```yaml
chunk_size: 8192
save_decode_cache: false
enable_async_loading: true
blocking_timeout_secs: 120
lookup_timeout_ms: 1800000
local_cpu: true
max_local_cpu_size: 80.0
save_unfull_chunk: false
remote_storage_plugins:
  - fs
extra_config:
  remote_storage_plugin.fs.base_path: /kvcache
  remote_storage_plugin.fs.use_odirect: true
  remote_storage_plugin.fs.read_ahead_size: 8192
  fs_connector_read_ahead_size: 8192
  fs_connector_use_odirect: true
  save_chunk_meta: false
```

### `./cfg/l2-adapter.json`

```json
{
  "type": "fs_native",
  "base_path": "/kvcache",
  "num_workers": 4,
  "use_odirect": true,
  "max_capacity_gb": 0,
  "read_ahead_size": 8192
}
```

### `./run.sh`

```bash
#!/usr/bin/env bash
set -euo pipefail

BASE_DIR="/root/vastdata"
MODEL_DIR="${BASE_DIR}/models/llama-4-scout"
CACHE_DIR="/mnt/kvcache-2ports"
CFG="${BASE_DIR}/cfg"

mkdir -p "${CACHE_DIR}"
chmod +x "${CFG}/start.sh"

docker run --name vllm-lmcache-in-process -d \
  --network host --ipc host --gpus all \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -v "${MODEL_DIR}:/models/model:ro" \
  -v "${CACHE_DIR}:/kvcache:rw" \
  -v "${CFG}:/cfg:ro" \
  -e LMCACHE_CONFIG_FILE=/cfg/lmcache.yaml \
  -e PYTHONHASHSEED=0 \
  -e LMCACHE_DISABLE_BANNER=1 \
  -e SAFETENSORS_FAST_GPU=1 \
  --entrypoint bash \
  vllm-lmcache:0.28.0-0.5.3-cu129 /cfg/start.sh
```

### `./Dockerfile`

```dockerfile
ARG VLLM_IMAGE=vllm/vllm-openai:v0.28.0-cu129
FROM ${VLLM_IMAGE}

ARG LMCACHE_VERSION=0.5.3
ENV CUDA_HOME=/usr/local/cuda
ENV LMCACHE_CUDA_MAJOR=12

# Build the official LMCache source distribution against this image's
# PyTorch/CUDA ABI. No LMCache or vLLM source files are replaced.
RUN python3 -m pip install --no-cache-dir \
      "setuptools>=77" wheel ninja cmake packaging pybind11 \
 && CPATH="$(ls -d /usr/local/lib/python3.12/dist-packages/nvidia/*/include \
      | tr '\n' ':')${CUDA_HOME}/include" \
    MAX_JOBS="$(nproc)" CMAKE_BUILD_PARALLEL_LEVEL="$(nproc)" \
    python3 -m pip install --no-cache-dir --no-build-isolation --no-deps \
      --no-binary lmcache "lmcache==${LMCACHE_VERSION}" \
 && python3 -c "import lmcache; print(lmcache.__version__)"
```

Build the image & Run it

```plaintext
docker build -t vllm-lmcache:0.24.0-rocm-lmcache0.5.3 -f Dockerfile .
chmod +x run.sh
./run.sh
```

---

## Additional Resources

- [https://hub.docker.com/r/vllm/vllm-openai/tags](https://hub.docker.com/r/vllm/vllm-openai/tags)

(*) While optimized, they are frequently not the latest vLLM builds

## Final Note

As workloads and software stacks frequently change, performance opportunities and compatibility shifts may occur. VAST actively expands its capabilities in the KV cache space—contact our team directly to leverage our latest updates and maximize your performance.