--- title: "VastOS Network Configuration Guide" slug: "vastos-network-configuration-guide" description: "Manage VAST node networks with configure_network.py, handling Layer 2/3 topologies, hardware detection, and VMS setup for efficient cluster operations." updated: 2026-10-27T19:37:32Z published: 2026-10-28T18:36:57Z canonical: "docs.vastdata.com/vastos-network-configuration-guide" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt > Use this file to discover all available pages before exploring further. # VastOS Network Configuration Guide The `configure_network.py` script is the primary tool for initializing and managing network interfaces on VAST nodes (CBox, DBox, and EBox). It handles Layer 2 and Layer 3 topologies, hardware detection, firmware configuration, and VMS (management service) setup. The script manages configuration for the following network roles: - **Management** - Cluster management using VMS, SSH, API, etc. Also provides the outbound interface for DNS services, NTP, Active Directory, LDAP, etc. By default, holds the VMS VIP and the DNS VIP. - **External/Northband/Frontend** - Application clients connect to the VAST cluster through this network using common protocols (NFS, S3, SMB, block, etc). - **Internal/Backend** - Handles communication between CNodes and DNodes, within the same cluster and across clusters when using replication / global namespace. - **BMC/IPMI** - Manage server hardware settings. Used to configure BMC, BIOS, etc. Relevant paths on a VAST cluster: - **Script path:** `/usr/bin/configure_network.py` - **Log location:** `/vast/log/configure_network/configure_network.log` - **Config file location:** `/etc/vast-configure_network.py-params.ini` *(Stores the arguments used in the last successful run).* For a full list of supported hardware platforms, see the [Appendix](/kb/docs/vastos-network-configuration-guide#appendix-supported-hardware). --- ## Configuration Modes The script supports two primary network topologies. ### Layer 2 (Standard/Legacy) The default mode. It uses VLANs for internal traffic and assumes a flat Layer 2 network for the external interface. - **External Network:** Configured on a specific physical interface or bond. May be shared with management functions. - **Internal Network:** Uses `bond0` (or physical interfaces) with a specific VLAN (default 69). ### Layer 3 (BGP) Enables BGP routing for leaf-spine topologies (`--l3-network-mode`). Uses FRR to manage BGP capabilities. - **External Network:** IPs are assigned to physical interfaces. BGP advertises these routes. - **Internal Network:** IPs are assigned to loopback or physical interfaces without VLAN tagging. BGP advertises these routes. - **Requirement:** Must specify ASN (`--l3-asn`) and Router ID (`--l3-router-id`). --- ## Usage The script requires root privileges. It can be run in several modes: ### Full install Perform a full configuration of network settings based upon the specified options. ```plaintext sudo configure_network.py [OPTIONS] ``` `**<NODE_ID>**`: Integer representing the node ID (e.g., `1`, `2`). Used to calculate default internal IPs unless overridden. The ID must be unique across the cluster. ### Dry run (new in 5.5) Generates network configuration files without applying changes to the system. Configuration files are stored in the folder `/tmp/configure_network`. ```plaintext sudo configure_network.py [OPTIONS] --dry-run ``` The `--dry-run` flag supports one of the following optional parameters: - `--dry-run=clean` - Remove the /tmp/configure_network folder before starting - `--dry-run=keep` - Keep the /tmp/configure_network folder, overwriting existing files ### Identify Hardware (new in 5.5) Print supported hardware identified by the script on the local node. No changes are made to the system. ```plaintext sudo configure_network.py --identify-hw ``` --- ## Special flags These flags change the default topology. Use the flags below when planning L3, edge, or platform-specific deployments. ### `--combo-ip` **Use when:** A single IP address must serve management, internal, and external roles on a single interface. Common on EBox, single-NIC, and L3 edge deployments where separate address pools are not available. **How it works:** One address replaces the usual split between `--ext-ip` and the internal addresses from `--template`. In L3 mode, it also becomes the internal `mgmt_ip`. Requires `--combo-ip-netmask` and `--combo-ip-gateway`. **Typical scenarios:** - Supermicro Milan EBox (required) - Single-port edge nodes with `--l3-network-mode` - Lab or customer networks with one routable IP per node **Not intended for:** Standard dual-NIC CBox/DBox deployments where management (`--ext-ip`) and data (`--template`) are on separate networks. ### `--mgmt-data-vip` **Use when:** VMS must be reachable via the **data network** in addition to (or instead of) the management network. Usually for L3 resiliency or sites where operators manage the cluster through the data-plane fabric. **How it works:** A second floating VMS VIP on `--mgmt-data-interface`. Unlike `--mgmt-vip`, routing is not inherited from `--ext-gateway`; you must supply `--mgmt-data-netmask`, `--mgmt-data-vip-gateway`, and optionally `--mgmt-data-vip-vlan`. **Typical scenarios:** - L3 inband deployments (e.g., Supermicro Genoa single-NIC L3) - The customer wants a backup VMS path if the OOB management network is down **Not intended for:** Standard outband deployments. `--mgmt-vip` on the management network is sufficient for most clusters. ### `--no-bond` **Use when:** The node should not create `bond0`. Required for **L3****/BGP mode**, which configures internal addresses directly on physical interfaces. **How it works:** Skips `bond0` creation and its virtual-function slaves. Internal IPs land on physical interface aliases (`:a`, `:b`, `:m`) instead of `bond0.69:*`. L3 mode sets this automatically if not already specified. **Typical scenarios:** - Any `--l3-network-mode` deployment - BGP leaf-spine where routes are learned, not bridged **Not intended for:** Standard L2 clusters. `bond0` with VLAN 69 is the default internal topology. ### `--no-vlan` **Use when:** Internal traffic should not be VLAN-tagged (no `bond0.69` or `enp*s0f0.69` style interfaces). **How it works:** Internal IPs are assigned directly to physical interfaces or to untagged bonds. Setting `--data-vlan 0` automatically enables `--no-vlan` if not already set. **Typical scenarios:** - L3 deployments where VLAN 69 is not used on the backend network - Customer fabric carries internal traffic without a dedicated data VLAN tag **Not intended for:** Standard L2 clusters. Internal VLAN 69 is the default. --- ## What’s New in 5.5 The following capabilities were added in the VastOS 5.5 release: - **Dry-run mode** `--dry-run` (`clean` or `keep`) generates configuration files without applying changes; cleans up prior dry-run output by default. - **Standalone hardware identification** `--identify-hw` prints the detected platform and exits (cannot be combined with other flags). - **Custom routing tables** - `--l3-route-table-id` assigns BGP-learned routes to the specified routing table rather than the main table. Applies only when `--l3-network-mode` is not used. - `--l3-default-route-table-id` puts the default route to management into its own table. Provides policy-based routing (PBR) for the management IP so SSH and other traffic sourced from that IP uses the specified route table rather than the main table. - **Internal-interface QoS suite:** the full QoS surface that previously existed only for external interfaces is now available for internal interfaces. Requires `--vxlan` to be set. New options: - `--qos-internal-pfc-enable` - `--qos-internal-pfc-traffic-class` - `--qos-internal-rdma-dscp2prio` - `--qos-internal-cnp-dscp2prio` - **CNP DSCP mapping** - map Congestion Notification Packets to a DSCP/traffic-class pair (previously RDMA-only). - `--qos-external-cnp-dscp2prio` - `--qos-internal-cnp-dscp2prio` - **VLAN-less internal network** `--no-vlan` removes the internal VLAN tag from internal interfaces. Requires that `--data-vlan 0` is also set. - **VXLAN switch support** `--vxlan` now adjusts ARP and `vast_pfc` settings for Cumulus/VXLAN switch fabrics. This setting is required when configuring QoS on the internal network. - **BF3 DPU configuration** `--bf3-dpu` configures networking for BlueField-3 DPU-based deployments. - **L2 port label in L2/L3 hybrid** `--l2-internal-port-in-l2l3` (`a` or `b`) labels, where the internal port stays L2 in a hybrid L2-on-L3 topology. - **Force-reduce internal IPs** `--force-reduce-internal-ips` reduces the internal IP count to 2 regardless of cluster type (whereas the existing `--reduce-internal-ips` only applies to single-interface SuperMicro DPU clusters). The following capabilities have been deprecated in the 5.5 release: - `--enable-pfc` and `--traffic-class` remain in 5.5 but are now explicitly **DEPRECATED**. Use the `--qos-external-pfc-*` flags instead. --- ## Argument Reference ### General Network Settings | **Argument** | **Description** | **Default** | | --- | --- | --- | | `--hostname <NAME>` | Sets the system hostname. | `node-<ID>` | | `--ext-dns <IP>` | External DNS servers (space separated, up to 3 servers). | Empty | | `--ext-dns-domain <DOM>` | Search domain for DNS. | Empty | | `--ntp <IP>` | Custom NTP servers (replaces default pool). | Empty | | `--eth-mtu <BYTES>` | MTU for Ethernet interfaces. | 9000 | | `--ib-mtu <BYTES>` | MTU for InfiniBand interfaces. | Mode dependent | ### Management Interface | **Argument** | **Description** | | --- | --- | | `--ext-interface <NAME>` | Physical interface name (e.g., `eno1`, `bond0`) for management traffic. | | `--ext-ip <IP>` | IPv4 address for the management interface. | | `--ext-netmask <MASK>` | IPv4 subnet mask. | | `--ext-gateway <IP>` | IPv4 default gateway. | | `--ext-vlan <ID>` | VLAN ID if the management network is tagged. | | `--ext-ip-ipv6 <IP>` | IPv6 address for the management interface. | | `--ext-prefix-ipv6 <LEN>` | IPv6 prefix length. | ### Combined IP configuration Use a single IP to server management, internal, and external roles on one interface. | **Argument** | **Description** | | --- | --- | | `--combo-ip <IP>` | Single IP used as external, internal, and management. | | `--combo-ip-gateway <IP>` | Gateway for combo IP. | | `--combo-ip-netmask <MASK>` | Netmask for combo IP (default 255.255.0.0) | ### Management VIP (VMS and related services) | **Argument** | **Description** | | --- | --- | | `--mgmt-vip <IP>` | The cluster-wide management VIP (IPv4). | | `--mgmt-vip-ipv6 <IP>` | The cluster-wide management VIP (IPv6). | | `--mgmt-inner-vip <IP>` | Internal VIP for inter-node communication. | ### L3 / BGP Settings | **Argument** | **Description** | | --- | --- | | `--l3-network-mode` | Enables Layer 3 mode (disables default bond/VLAN logic). | | `--l3-asn <ASN>` | Autonomous System Number for BGP. Requires `--data-netmask 255.255.255.255` | | `--l3-router-id <IP>` | Unique Router ID for this node (usually loopback IP). | | `--l3-bgp-config` | `default` or `community`. Controls route map tagging. | ### IPMI & Back-to-Back (B2B) | **Argument** | **Description** | | --- | --- | | `--b2b-ipmi` | Configures a back-to-back IPMI network on the technician port. | | `--b2b-template` | IP template for B2B IPMI addresses (default `192.168.3.{node-id}`). | | `--ipmi-ip <IP>` | Static IP to assign to the BMC/iDRAC/iLO. | | `--ipmi-netmask <MASK>` | Subnet mask for IPMI. | | `--ipmi-gateway <IP>` | Default gateway for IPMI. | ### QoS & Flow Control (New in v5.5) **Notes:** - Using the options below will erase all previous QoS settings. - `--enable-pfc` and `--traffic-class` are **DEPRECATED**. Use the options below instead. - `--vxlan` must be set when using QoS. - QoS can be set on a live cluster. Example: - `clush -g cnodes "sudo configure_network.py --qos-only --qos-external-pfc-enable --qos-external-rdma-dscp2prio 40,3 --qos-external-cnp-dscp2prio 48,6 --vxlan"` | **Argument** | **Description** | | --- | --- | | `--qos-only` | "Fast Exit". Updates QoS settings only, without configuring other settings. | | `--qos-external-pfc-enable` | External - Enable PFC on external interfaces. | | `--qos-external-pfc-traffic-class` | External - Bitmap (e.g., `0,0,0,1,0,0,0,0`) for PFC priorities. | | `--qos-external-rdma-dscp2prio` | External - Map DSCP to Priority (Format: `dscp,prio`). Default: `26,3`. | | `--qos-external-cnp-dscp2prio` | External - CNP DSCP to traffic-class map (Format: dscp, tc) | | `--qos-internal-pfc-enable` | Internal - Enable PFC on internal interfaces. | | `--qos-internal-pfc-traffic-class` | Internal - Bitmap (e.g., `0,0,0,1,0,0,0,0`) for PFC priorities. | | `--qos-internal-rdma-dscp2prio` | Internal - Map DSCP to Priority (Format: `dscp,prio`). Default: `26,3`. | | `--qos-internal-cnp-dscp2prio` | Internal - CNP DSCP to traffic-class map (Format: dscp, tc) | ### Auto-Ports The `--auto-ports` flag automatically detects the hardware platform type and maps interfaces accordingly. This is the recommended method for supported hardware (see Appendix). **Modes:** - `eth`: Both internal and external networks use Ethernet - `ib`: Both internal and external networks use InfiniBand (IB) - `int_eth_ext_ib`: Configure the internal network to use Ethernet and the external to use IB - `int_ib_ext_eth`: Configure the internal network to use IB and the external to use Ethernet **Helper Flags:** - `--auto-ports-ext-iface`: Defines where management traffic flows. - `outband`: Dedicated management port (default). - `inband`: Shared with internal data traffic. - `northband`: Uses one of the external data interfaces. | **Argument** | **Description** | | --- | --- | | `--auto-ports` | Auto-detect NIC roles per chassis. Choices: `eth`, `ib`, mixed modes, or `disabled` (default). | | `--auto-ports-ext-iface` | Which port carries external/mgmt traffic: `inband`, `outband`, or `northband` | | `--auto-ports-reverse-nics` | Swap internal and external NIC roles on dual-NIC systems | | `--auto-ports-skip-nic` | On dual-NIC chassis, use only internal `int`, only external `ext`, or both `none`. | ### Firmware Configuration | **Argument** | **Description** | | --- | --- | | `--only-fw` | Configure NIC firmware only, then exit. | | `--skip-fw` | Skip NIC firmware configuration | ### Miscellaneous Settings | **Argument** | **Description** | | --- | --- | | `--template` | Format string to use for internal IPs (default `172.16.{network}.{node}`). | | `--data-netmask <MASK>` | Netmask for internal IP template (default `255.255.255.0`). A recommended netmask is `255.255.192.0`. | | `--data-vlan <ID>` | Internal data VLAN ID (default `69`). Set to `0` to turn off VLAN tagging on the internal network. | | `--subsystem` | Subsystem bits are added to the third octet of the internal IP address. | | `--bf3-dpu` | BlueField-3 DPU network configuration mode. | | `--dockerd-bip` | Docker bridge IP/CIDR; validated against internal IP template for overlap. | | `--frr-ipv6-prefix` | Link-local IPv6 prefix for FRR next-hop (default `fc80`). | --- ## Examples ### Scenario 1: Standard CNode Setup (L2) Setting up node 1 with a dedicated management port `eno1` and IPMI configuration. ```bash sudo configure_network.py 1 \  --auto-ports eth \  --ext-ip 10.0.0.101 \  --ext-netmask 255.255.255.0 \  --ext-gateway 10.0.0.1 \  --mgmt-vip 10.0.0.200 \  --ipmi-ip 192.168.10.101 \  --ipmi-netmask 255.255.255.0 \  --ipmi-gateway 192.168.10.1 ``` ### Scenario 2: L3 / BGP Configuration Setting up node 5 in a BGP leaf-spine architecture. ```bash sudo configure_network.py 5 \  --l3-network-mode \  --l3-asn 65001 \  --l3-router-id 192.168.100.5 \  --ext-interface eno1 \  --ext-ip 10.0.5.10 \  --ext-netmask 255.255.255.0 \  --mgmt-vip 10.0.100.1 ``` ### Scenario 3: QoS Update (No Reboot/Network Reset) Updating DSCP mappings on a running system. ```bash sudo configure_network.py --qos-only \  --qos-external-pfc-enable \  --qos-external-rdma-dscp2prio 46,5 ``` ### Scenario 4: "Combo" IP (Single Port Edge) Using a single IP address for Management, Internal, and External traffic (common in Edge/EBox deployments). ```bash sudo configure_network.py 1 \  --l3-network-mode \  --l3-asn 65000 \  --l3-router-id 10.10.10.1 \  --combo-ip 10.10.10.15 \  --combo-ip-netmask 255.255.255.0 \  --combo-ip-gateway 10.10.10.254 ``` --- ## Troubleshooting 1. **Reboot Required:** If the script changes firmware settings (SR-IOV, Link Type), it will create a file `/run/configure_network_requires_reboot`. You must reboot the node and run the script again. 2. **Log Analysis:** Always check `/vast/log/configure_network/configure_network.log` for failures in specific steps (e.g., `ipmitool` failures or `frr` stabilization issues). 3. **Dry Run:** Use `--only-generate-conf-files` to verify the generated `/etc/sysconfig/network-scripts/ifcfg-*` files without applying changes. --- ## Appendix: Supported Hardware The `configure_network.py` script will automatically recognize the following hardware platforms and devices, applying hardware-specific changes as appropriate. ### Legacy CBox / DBox | Platform | Role | Notes | | --- | --- | --- | | Broadwell CNode | CNode | Single dual-port NIC | | Cascade Lake CNode | CNode | 1 or 2 dual-port NICs; Mellanox or Broadcom | | Mercury CNode | CNode | Single dual-port NIC | | Mavericks DNode (Sanmina) | DNode | Two single-port or single dual-port NICs | | Mavericks MLK 1.5 DNode | DNode | Single dual-port NIC | | Supermicro DNode | DNode | Single dual-port NIC | ### Ceres DBox | Platform | Role | | --- | --- | | Ceres DNode | DNode | | Mercury Ceres DNode | DNode | | Ceres V2 DNode | DNode | | Ceres V3 DNode | DNode | ### HPE | Platform | Role | | --- | --- | | HPE Ice Lake CNode | CNode | | HPE Raider CNode | CNode | | HPE Raider DNode | DNode | | HPE Genoa CBox (DL325) | CNode | | HPE Genoa EBox10 | EBox | | HPE Turin (DL325) | CNode/EBox | ### Dell | Platform | Role | | --- | --- | | Dell Ice Lake CNode (C6520) | CNode | | Dell Genoa EBox (R6615) | EBox | | Dell Jump 6620 CNode | CNode | | Dell Jump Turin CNode | CNode | | Dell Turin CNode | CNode | | Dell EBox DPU | DPU | | CoreWeave Dell Turin EBox | EBox | ### Supermicro | Platform | Role | | --- | --- | | Supermicro Genoa CBox | CNode | | Supermicro Genoa EBox24 | EBox | | Supermicro Genoa DPU (BlueField-3) | DPU | | Supermicro Turin CBox | CNode | | Supermicro Turin EBox10 | EBox | | Supermicro Turin EBox 12/15 | EBox | | Supermicro Milan EBox | EBox | ### Cisco | Platform | Role | | --- | --- | | Cisco Turin CBox | CNode | | Cisco Genoa EBox (BlueField) | EBox | | Cisco Genoa CX7 EBox | EBox | | Cisco Genoa BlueField DPU | DPU | ### Third-party EBox / GBox (OEM) | Platform | Role | | --- | --- | | Lenovo Genoa EBox | EBox | | Kaytus Genoa EBox | EBox | | xFusion Genoa EBox | EBox | | Gigabyte Genoa EBox | EBox |