---
title: "ModelExpress with vLLM (NVIDIA)"
slug: "modelexpress-with-vllm-nvidia"
description: "Learn how ModelExpress accelerates vLLM safetensors and JIT artifact loading via NIXL RDMA, scaling models across GPU hosts with peer-to-peer transfers."
updated: 2026-10-28T02:14:35Z
published: 2026-10-28T02:14:35Z
canonical: "docs.vastdata.com/modelexpress-with-vllm-nvidia"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt
> Use this file to discover all available pages before exploring further.

# ModelExpress with vLLM (NVIDIA)

**ModelExpress (MX)** is NVIDIA’s fast model loader for inference engines: it pulls safetensors from storage (InstantTensor / ModelStreamer / GDS) or from a ready peer over NIXL RDMA, and can optionally move JIT artifacts the same way. It is not a KV cache tier.

MX also supports **SGLang** (and potentially other LLM frameworks). Engine wiring differs per framework — this guide is the **vLLM** path; SGLang needs its own MX load-format / client setup.

This is an extension of the vLLM (NVIDIA) guide: same host stack (File Prerequisites, Docker, NVIDIA driver, NFSv3/RDMA, VAST mounts). For this guide, we’ll be using **two servers** to showcase model loading across multiple hosts, deploying **gpt-oss-120b / TP=1**, plus MX.

Use MX **with** OffloadingConnector — neither replaces the other.

Complete File Prerequisites and the base vLLM NVIDIA guide first.

## Motivation

Cold model bring-up is slow: stock vLLM weight load plus first-time compile can dominate **minutes of a rollout**. At datacenter scale, you need to **swap or scale models quickly as demand shifts** — not wait on a full cold load every time a replica or model changes. MX shortens that path for **both model weights and compile/JIT artifacts**. Speedup (without NV-link) gets us as fast as 6x on weights alone, and with NV-link we should get even a bigger bump.

A production GPU fleet is typically much larger than the storage that serves it — for example, ~20 racks of GPUs against less than one rack of VAST. Aggregate GPU demand dwarfs that storage footprint, which can limit performance when many GPUs cold load from storage alone concurrently. ModelExpress addresses that: **VAST feeds the first cold load(s);** additional replicas take weights over NIXL RDMA from a READY peer (and can reuse JIT artifacts from a shared VAST root when configs match).

## Setup

Lab used for the numbers below:

**GPUs**: 2× NVIDIA RTX PRO 6000 Blackwell — **two servers, one GPU each**, **no NVLink** (cross-node RoCE only for P2P)

- **Network**: 2× Mellanox CX-7 200GbE per host; VAST over [NFSv3 over RDMA](https://www.vastdata.com/blog/meet-your-need-for-speed-with-nfs); RoCE used for MX NIXL P2P
- **Storage**: Model checkpoint + JIT scratch on VAST (`/mnt/kvcache-2ports` layout from the base guide)
- **Software**: vLLM **0.28.0** + **ModelExpress 0.7.0** (pins below)
- **Model:** gpt-oss-120b, TP=1

## Two components: weights and artifacts

MX (and engine-ready time) has two independent pieces:

| Component | What it is | How MX helps | **Alternatives** |
| --- | --- | --- | --- |
| **Weights** | Model safetensors → GPU | Strategy chain: NIXL P2P, InstantTensor, ModelStreamer, GDS, then default loader | Slow cold read each time |
| **Artifacts** | On-disk JIT / compile caches the engine needs after weights are in GPU — mainly **torch.compile**, **FlashInfer**, **Triton** (plus DeepGEMM / TileLang / CuTe when used). For a given model there can be **many different artifacts** as you change parallelism parameters/kvcahce parameters/data format,.. | NIXL move from an active peer with identical configuration - `MX_ARTIFACT_TRANSFER=1` | - Slow cold creation - or caching via a Shared VAST holding an identical configuration |

You can optimize either without the other (fast weights + cold compile, or shared arts + slow weight path). Best bring-up hits both.

## Technologies MX Uses

[ModelExpress](https://developer.nvidia.com/blog/modelexpress-distributing-model-artifacts-at-the-speed-of-light/) internally chooses among the below technologies for **weights** loading based on a set priority chain (see the chain below)

| Technology | What it does | Typical trigger |
| --- | --- | --- |
| [**NIXL**](https://github.com/ai-dynamo/nixl) **RDMA P2P** | GPU→GPU weight (and optional JIT artifact) transfer from a READY peer over RoCE/IB | Peer published; first in the chain |
| **ServerCache** | Stream weights from the MX server’s cache (no shared checkpoint FS) | `MODEL_EXPRESS_NO_SHARED_STORAGE=1` |
| [**InstantTensor**](https://pypi.org/project/instanttensor/) | Fast local safetensors → GPU (direct I/O; can use GDS under the hood when available) | Default when no peer/streamer |
| [**ModelStreamer**](https://github.com/run-ai/runai-model-streamer) (Run:AI) | Concurrent safetensor reads via bounded CPU staging; local path or `s3://` / `gs://` / `az://` | `MX_MODEL_URI` set |
| [**GDS**](https://docs.nvidia.com/gpudirect-storage/) (GPUDirect Storage / cuFile) | Storage→GPU DMA, bypassing host bounce buffers | InstantTensor/Streamer skipped; host `nvidia_fs` + IOMMU passthrough |
| **Default loader** | Stock vLLM safetensors (CPU-staged) | Last resort |
| **Artifact transfer** | NIXL (or Mooncake) move of torch.compile / FlashInfer / Triton caches | `MX_ARTIFACT_TRANSFER=1` (+ shared VAST root as alternative) |

Upstream overview: [ModelExpress GitHub](https://github.com/ai-dynamo/modelexpress), [DEPLOYMENT.md](https://github.com/ai-dynamo/modelexpress/blob/main/docs/DEPLOYMENT.md).

## Lab Results

Model: gpt-oss-120b in a TP=1 setting

- **Lab scale doesn’t represent a fleet of GPUs** — 2 GPUs across two servers, one model (gpt-oss-120b), short cold loads. This is a bring-up comparison, not a VAST cluster bandwidth benchmark.
- **Cold paths** (InstantTensor / ModelStreamer / GDS / default) measure loading safetensors from VAST into a GPU when no READY peer exists.
- **NIXL P2P** is warm peer GPU→GPU after someone already loaded — not “faster VAST.”
- **Throughput note (InstantTensor cold)** — model is multi-file safetensors; observed ~126–131 Gbit/s for those ~7–9s. Larger models and more concurrent clients are needed before claiming cluster line rate.

## Weight Load by Technology

| Technology | Temp. | Time | Notes |
| --- | --- | --- | --- |
| **NIXL RDMA P2P** | warm | **~4.3–4.5**s | Peer GPU → GPU Warm path after a peer is READY — not a cold storage read. |
| **InstantTensor** (VAST) | cold | **~9s** | VAST safetensors fast path Multi-file; ~126–131 Gbit/s over ~7–9s in this lab — not a line-rate claim for a full cluster. |
| **ModelStreamer** (VAST) | cold | **~15s** | Run:AI streamer |
| **GDS** | cold | **~21s** | Needs host `iommu=pt` + `nvidia_fs` |
| **Default vLLM loader** (VAST) | cold | **~27s** | Baseline |

## Artifacts / compile time by method

| Method | Temp. | Time | Notes |
| --- | --- | --- | --- |
| **NIXL artifact P2P** (torch.compile + FlashInfer + Triton) | warm | **~3.1s** total | Install from peer; compile then ~**0.5**s |
| **VAST-shared JIT root** | warm | ~**0.5s** compile | No P2P; directly from VAST |
| **Cold local compile** | cold | **~15s–22s** | Baseline |

## Other factors (typical)

| Factor | Cold | Warm (Cache reuse) | What controls it |
| --- | --- | --- | --- |
| torch.compile | ~18s–22s | ~0.5s | `VLLM_CACHE_ROOT` (shared `RUN_ID` or art P2P) |

## Why ModelExpress at scale

Reading a model extremely fast onto a large GPU fleet (e.g. thousands of GPUs) implies very high **aggregate** bandwidth — TBps-class if every node cold-reads from storage. Doing that with local CPUs and SSDs on every node wastes power and hardware.

With ModelExpress: VAST supplies the first loader(s); peers fan out over NIXL; matching JIT can stay on shared VAST. Effective model delivery then scales with the GPU fabric rather than by sizing storage 1:1 with GPU racks.

At a realistic GPU∶CNode ratio, ModelExpress can look like ~100× better *effective* model-delivery than naïve all-nodes-from-storage — because one cold VAST load plus P2P fan-out replaces thousands of redundant cold reads. The lab microbenchmark (~2× InstantTensor vs default, ~2× NIXL vs InstantTensor) is only a small-scale illustration.

## Software stack

| Component | Pin |
| --- | --- |
| vLLM base | `vllm/vllm-openai:v0.28.0-cu129` |
| MX Python client | `modelexpress==0.7.0` — wheel `modelexpress-0.7.0-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl`, sha256 `3b3708e44d95c027c39987aff806dad0d98e6e1cf0dedf303b14b48e96c36fd7` |
| MX server image | `sha256:6ff0e82906bfc02672962c825836a5d2603ceae2a0423242bd090e43b25c61fc` (retag as `modelexpress-server:0.7.0`) |
| Redis | `redis:7-alpine` (MX metadata backend) |

Upstream project: [ai-dynamo/modelexpress](https://github.com/ai-dynamo/modelexpress).

## How MX chooses a load path

With `--load-format modelexpress` and default `MX_LOAD_STRATEGY_CHAIN=INFERENCE`, the client runs this **priority chain** (first available success wins; safe fall-through on miss):

```plaintext
1. NIXL RDMA P2P       ← peer already READY (weights; arts if MX_ARTIFACT_TRANSFER=1)
2. ServerCache         ← MODEL_EXPRESS_NO_SHARED_STORAGE=1
3. InstantTensor       ← default local fast path
4. ModelStreamer       ← only if MX_MODEL_URI is set
5. GDS                 ← cuFile / nvidia-fs; earlier paths skipped or failed
6. Default loader      ← stock vLLM safetensors
```

Optional overrides: `MX_MODEL_URI=…` for ModelStreamer; `MX_INSTANT_TENSOR=0` to skip InstantTensor (e.g. force GDS). Start the cold engine first so the joiner’s RDMA step sees a READY peer.

Also set on every engine: `MX_SERVER_ADDRESS`, `MX_REGISTER_LOADERS=1`, `MX_WORKER_HOST` (routable **RoCE** IP), `MX_METADATA_PORT` / `MX_WORKER_GRPC_PORT`, `MX_P2P_METADATA=1`. Share JIT arts via the same `RUN_ID` / `ARTIFACT_ROOT` on VAST (`MX_ARTIFACT_TRANSFER=0`, default); set `MX_ARTIFACT_TRANSFER=1` only when roots are not shared.

## Installation

Reuse the base guide layout (`/root/vastdata`, VAST at `/mnt/kvcache-2ports`). Put the gpt-oss checkpoint under `models/` and add an MX cfg / artifact leaf:

```bash
BASE_DIR="/root/vastdata"
MODEL_DIR="${BASE_DIR}/models/gpt-oss-120b"
CACHE_DIR="/mnt/kvcache-2ports"
CFG="${BASE_DIR}/cfg"
mkdir -p "${BASE_DIR}" "${MODEL_DIR}" "${CFG}" "${CACHE_DIR}/mx"
```

## Engine image (vLLM + pinned MX client)

`Dockerfile` (build context = this directory):

```dockerfile
ARG VLLM_IMAGE=vllm/vllm-openai:v0.28.0-cu129
FROM ${VLLM_IMAGE}

# Pin exact PyPI release (do not use unpinned "modelexpress").
ARG MODELEXPRESS_VERSION=0.7.0
RUN pip install --no-cache-dir "modelexpress==${MODELEXPRESS_VERSION}" \
 && python3 -c "import importlib.metadata as m; v=m.version('modelexpress'); assert v=='0.7.0', v"

ENV MX_REGISTER_LOADERS=1
```

```bash
docker build -t vllm-modelexpress:0.28.0-cu129-mx0.7.0 .
```

Optional stricter pin (copy the wheel next to the Dockerfile after `pip download`):

```dockerfile
COPY modelexpress-0.7.0-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl /tmp/
RUN pip install --no-cache-dir /tmp/modelexpress-0.7.0-*.whl \
 && python3 -c "import importlib.metadata as m; assert m.version('modelexpress')=='0.7.0'"
```

## MX server image (pinned digest)

```bash
# From a registry that already has this digest, or retag a known-good local copy:
docker pull <registry>/modelexpress-server@sha256:6ff0e82906bfc02672962c825836a5d2603ceae2a0423242bd090e43b25c61fc
docker tag <registry>/modelexpress-server@sha256:6ff0e82906bfc02672962c825836a5d2603ceae2a0423242bd090e43b25c61fc \
  modelexpress-server:0.7.0
```

## Run

Both engines use the **same** launch script. Neither is configured as “publisher” or “consumer” — MX’s strategy chain picks **NIXL P2P** if a READY peer exists, otherwise InstantTensor / ModelStreamer / GDS / default. Per-node inputs: RoCE IP, MX server host, `INSTANCE` (docker name), shared `RUN_ID`.

Use **host network**, bind the VAST checkpoint at the **same absolute path** inside the container, and set `MX_WORKER_HOST` to that node’s **RoCE** address.

## Redis + MX server (once; typically on node A)

```bash
docker run -d --name mx-redis --network host redis:7-alpine \
  redis-server --port 16379 --bind 0.0.0.0

docker run -d --name mx-server --network host \
  -e MX_METADATA_BACKEND=redis \
  -e REDIS_URL=redis://127.0.0.1:16379 \
  -e MODEL_EXPRESS_SERVER_HOST=0.0.0.0 \
  -e MODEL_EXPRESS_SERVER_PORT=18001 \
  -e MODEL_EXPRESS_LOG_LEVEL=info \
  -e MODEL_EXPRESS_CACHE_EVICTION_ENABLED=false \
  modelexpress-server:0.7.0
```

> The server image reads `MX_METADATA_BACKEND` + `REDIS_URL` (not `MODEL_EXPRESS_REDIS_URL`). Confirm with `docker logs mx-server` that it connects to Redis and listens on `0.0.0.0:18001`.

## Shared engine script — `${CFG}/start-mx-engine.sh`

JIT / compile caches do **not** all live under `VLLM_CACHE_ROOT`. Point every leaf at a VAST directory (and bind ninja-sensitive ones at a **fixed** container path so P2P extract does not force a rebuild when host paths differ per instance):

| Env | Leaf | Role |
| --- | --- | --- |
| `VLLM_CACHE_ROOT` | `vllm-cache/` | torch.compile / vLLM cache |
| `FLASHINFER_WORKSPACE_BASE` | `flashinfer/` → `/opt/vast/jit/flashinfer` | FlashInfer workspace |
| `VLLM_FLASHINFER_AUTOTUNE_CACHE_DIR` | `flashinfer_autotune/` | FlashInfer autotune |
| `TRITON_CACHE_DIR` | `triton/` → `/opt/vast/jit/triton` | Triton |
| `DG_JIT_CACHE_DIR` / `DEEP_GEMM_CACHE_DIR` | `deep_gemm/` → `/opt/vast/jit/deep_gemm` | DeepGEMM |
| `TILELANG_CACHE_DIR` | `tilelang/` → `/opt/vast/jit/tilelang` | TileLang |
| `CUTE_DSL_CACHE_DIR` | `cute_dsl/` → `/opt/vast/jit/cute_dsl` | CuTe DSL |

`ARTIFACT_ROOT` is keyed by `RUN_ID` (workload fingerprint, e.g. `gpt-oss-120b-tp1`), not by `INSTANCE`. Same `RUN_ID` on every node ⇒ shared VAST warm arts; set `MX_ARTIFACT_TRANSFER=0`. Use art P2P only when engines must keep separate roots (different `RUN_ID` / `ARTIFACT_ROOT`); keep the `/opt/vast/jit/*` binds either way so ninja paths stay stable.

vLLM sizing knobs below match the base guide’s `cfg/start.sh` (context length, GPU util, batching, block size, hybrid KV). Model/TP match the results table (gpt-oss-120b, TP=1). MX deltas: image, `--load-format modelexpress`, MX env, JIT binds. OffloadingConnector (`--kv-transfer-config` from the base `start.sh`) is unchanged if you still want KV tiering — append it the same way.

```bash
#!/usr/bin/env bash
set -euo pipefail

# Required from the environment (set by the per-node wrapper):
#   ROCE_IP            this node's RoCE address (MX_WORKER_HOST)
#   MX_SERVER_HOST     RoCE/host where mx-server listens (often node A)
#   INSTANCE           container name suffix only (e.g. replica-a / replica-b)
#   RUN_ID             shared artifact key for this workload (e.g. gpt-oss-120b-tp1)
# Optional overrides:
#   BASE_DIR MODEL_DIR CACHE_DIR ARTIFACT_ROOT PORT GPU

BASE_DIR="${BASE_DIR:-/root/vastdata}"
MODEL_DIR="${MODEL_DIR:-${BASE_DIR}/models/gpt-oss-120b}"
CACHE_DIR="${CACHE_DIR:-/mnt/kvcache-2ports}"
PORT="${PORT:-8000}"
GPU="${GPU:-0}"
NAME="vllm-mx-${INSTANCE}"
# Shared across nodes for the same workload; do not put INSTANCE here.
ARTIFACT_ROOT="${ARTIFACT_ROOT:-${CACHE_DIR}/mx/${RUN_ID}/artifacts}"

: "${ROCE_IP:?set ROCE_IP}"
: "${MX_SERVER_HOST:?set MX_SERVER_HOST}"
: "${INSTANCE:?set INSTANCE}"
: "${RUN_ID:?set RUN_ID}"

mkdir -p \
  "${ARTIFACT_ROOT}/vllm-cache" \
  "${ARTIFACT_ROOT}/flashinfer" \
  "${ARTIFACT_ROOT}/flashinfer_autotune" \
  "${ARTIFACT_ROOT}/triton" \
  "${ARTIFACT_ROOT}/deep_gemm" \
  "${ARTIFACT_ROOT}/tilelang" \
  "${ARTIFACT_ROOT}/cute_dsl"
chmod -R a+rwX "${ARTIFACT_ROOT}"   # VAST root_squash + Docker-as-root

docker rm -f "${NAME}" 2>/dev/null || true
docker run -d --name "${NAME}" --network host --gpus "device=${GPU}" \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -v "${MODEL_DIR}:/models/model:ro" \
  -v "${ARTIFACT_ROOT}/vllm-cache:${ARTIFACT_ROOT}/vllm-cache:rw" \
  -v "${ARTIFACT_ROOT}/flashinfer_autotune:${ARTIFACT_ROOT}/flashinfer_autotune:rw" \
  -v "${ARTIFACT_ROOT}/flashinfer:/opt/vast/jit/flashinfer:rw" \
  -v "${ARTIFACT_ROOT}/triton:/opt/vast/jit/triton:rw" \
  -v "${ARTIFACT_ROOT}/deep_gemm:/opt/vast/jit/deep_gemm:rw" \
  -v "${ARTIFACT_ROOT}/tilelang:/opt/vast/jit/tilelang:rw" \
  -v "${ARTIFACT_ROOT}/cute_dsl:/opt/vast/jit/cute_dsl:rw" \
  -e SAFETENSORS_FAST_GPU=1 \
  -e MX_SERVER_ADDRESS="${MX_SERVER_HOST}:18001" \
  -e MX_REGISTER_LOADERS=1 \
  -e MX_WORKER_HOST="${ROCE_IP}" \
  -e MX_METADATA_PORT=20000 \
  -e MX_WORKER_GRPC_PORT=21000 \
  -e MX_P2P_METADATA=1 \
  -e MX_ARTIFACT_TRANSFER="${MX_ARTIFACT_TRANSFER:-0}" \
  -e MX_ARTIFACT_READY_URL="http://127.0.0.1:${PORT}/health" \
  -e VLLM_CACHE_ROOT="${ARTIFACT_ROOT}/vllm-cache" \
  -e VLLM_FLASHINFER_AUTOTUNE_CACHE_DIR="${ARTIFACT_ROOT}/flashinfer_autotune" \
  -e FLASHINFER_WORKSPACE_BASE="/opt/vast/jit/flashinfer" \
  -e TRITON_CACHE_DIR="/opt/vast/jit/triton" \
  -e DG_JIT_CACHE_DIR="/opt/vast/jit/deep_gemm" \
  -e DEEP_GEMM_CACHE_DIR="/opt/vast/jit/deep_gemm" \
  -e TILELANG_CACHE_DIR="/opt/vast/jit/tilelang" \
  -e CUTE_DSL_CACHE_DIR="/opt/vast/jit/cute_dsl" \
  vllm-modelexpress:0.28.0-cu129-mx0.7.0 \
  /models/model \
    --host 0.0.0.0 --port "${PORT}" \
    --served-model-name gpt-oss-120b \
    --load-format modelexpress \
    --tensor-parallel-size 1 \
    --max-model-len 131072 \
    --max-num-seqs 300 \
    --max-num-batched-tokens 8192 \
    --gpu-memory-utilization 0.92 \
    --enable-prefix-caching \
    --block-size=64 \
    --no-disable-hybrid-kv-cache-manager
```

> [!NOTE]
> Image entrypoint is already `["vllm","serve"]` (from `vllm/vllm-openai`). Pass only the model path and flags — do **not** prefix with `vllm serve` again.

```bash
chmod +x "${CFG}/start-mx-engine.sh"
```

## Per-node wrappers (IPs only)

Node A (`${CFG}/run-node-a.sh`) — start this first if you want the second engine to have a READY peer for P2P:

```bash
#!/usr/bin/env bash
set -euo pipefail
export ROCE_IP="<node-A-roce-ip>"
export MX_SERVER_HOST="${ROCE_IP}"          # mx-server on A
export INSTANCE="replica-a"                 # docker name only
export RUN_ID="gpt-oss-120b-tp1"            # same on every node
exec "$(dirname "$0")/start-mx-engine.sh"
```

Node B (`${CFG}/run-node-b.sh`):

```bash
#!/usr/bin/env bash
set -euo pipefail
export ROCE_IP="<node-B-roce-ip>"
export MX_SERVER_HOST="<node-A-roce-ip>"    # same mx-server as A
export INSTANCE="replica-b"                 # docker name only
export RUN_ID="gpt-oss-120b-tp1"            # must match A
# Art P2P only if roots diverge:
#   export MX_ARTIFACT_TRANSFER=1
exec "$(dirname "$0")/start-mx-engine.sh"
```

```bash
# On A (after Redis + mx-server):
./cfg/run-node-a.sh
# Wait until http://127.0.0.1:8000/v1/models is up (optional; only affects who is READY first)
# On B:
./cfg/run-node-b.sh
```

Check logs for the path MX actually took (`RDMA transfer complete`, `InstantTensor`, `GDS load complete`, etc.). Starting B after A is READY biases B toward P2P; reverse start order or a missed READY peer falls through to storage.

## Additional resources

- Base guide: [vLLM (NVIDIA)](/develop/docs/vllm-nvidia)
- File Prerequisites: [File Prerequisites](/develop/docs/file-prerequisites)
- [ModelExpress blog](https://developer.nvidia.com/blog/modelexpress-distributing-model-artifacts-at-the-speed-of-light/) · [GitHub](https://github.com/ai-dynamo/modelexpress) · [DEPLOYMENT.md](https://github.com/ai-dynamo/modelexpress/blob/main/docs/DEPLOYMENT.md)
- [NIXL](https://github.com/ai-dynamo/nixl) · [GPUDirect Storage](https://docs.nvidia.com/gpudirect-storage/) · [InstantTensor](https://pypi.org/project/instanttensor/) · [ModelStreamer](https://github.com/run-ai/runai-model-streamer)
- [vLLM tags](https://hub.docker.com/r/vllm/vllm-openai/tags)

## Final note

Software and fabrics change. Pins above (`modelexpress==0.7.0`, server digest, vLLM 0.28.0) freeze the stack that produced the results table; bump deliberately and re-validate load-path logs after upgrades.