Network Design Best Practices for Using Proxmox with FlashArray over NVMe-TCP

Proxmox
Audience
Public
Source Type
Documentation

This page provides the best practices when designing your network for using Proxmox with FlashArray as storage over NVMe-TCP.

Network Architecture Principles

Principle Reason

Dedicated Storage Network: Always use dedicated physical or VLAN-isolated networks for storage traffic

 Isolates storage I/O from other network traffic; prevents bandwidth contention; enables QoS policies specific to storage; simplifies security and firewall rules

No Single Points of Failure: Redundant switches, NICs, and storage controllers

 Any single component can fail without impacting storage availability; enables zero-downtime maintenance; critical for production environments

Proper Segmentation: Separate storage traffic from management and VM traffic

 Prevents noisy neighbor problems; ensures storage performance is not affected by VM traffic spikes; improves security posture; simplifies network troubleshooting

Optimized MTU: Use jumbo frames (MTU 9000) end-to-end when possible

 Reduces CPU overhead by ~30%; improves throughput by reducing packet count; lowers interrupt rate; essential for high-performance storage

Network Topology Options

Option 1: Dedicated Physical Networks (Recommended): This is the gold standard for production storage networks because it provides complete isolation and maximum performance.

Advantages:

  • Maximum performance and isolation: Storage has dedicated bandwidth with zero contention.
  • No contention with other traffic: VM traffic spikes cannot impact storage performance.
  • Simplified troubleshooting: Storage network issues are isolated and easier to diagnose.
  • Predictable latency: Consistent performance without interference from other workloads.

Requirements:

  • Dedicated NICs on each Proxmox node (minimum 2 for redundancy): Provides redundancy; one NIC can fail without storage outage
  • Dedicated storage switches (minimum 2 for redundancy): Eliminates switch as single point of failure; allows switch maintenance without downtime
  • Direct connections to storage array: Reduces latency; simplifies network path; fewer points of failure

Configuration Example: 

# /etc/network/interfaces
# Both NICs on same subnet (10.100.1.0/24)
auto ens1f0
iface ens1f0 inet static
    address 10.100.1.101/24
    mtu 9000

auto ens1f1
iface ens1f1 inet static
    address 10.100.1.102/24
    mtu 9000

Option 2: VLAN-Based Segmentation (Shared NICs): Choose this option when you have limited physical NICs or want to consolidate infrastructure. Acceptable for environments where cost/port density is a concern and you can accept some performance trade-offs.

Advantages:

  • Efficient use of physical infrastructure: Reduces NIC and switch port requirements
  • Flexible network design: Easy to add new VLANs without physical changes
  • Single set of NICs handles multiple traffic types: Lower hardware costs; fewer cables

Considerations:

  • NICs are shared with management, VM, and container traffic: Storage competes for bandwidth with other traffic; potential performance degradation during high VM traffic.
  • Requires proper VLAN configuration and trunking on switches: Mis-configuration can cause traffic leakage or complete storage outage.
  • Storage VLANs share bandwidth with other VLANs on same NICs: Maximum storage throughput is limited by total NIC bandwidth minus other traffic.
  • Requires QoS/traffic prioritization for storage VLANs: Without QoS, VM traffic can starve storage traffic causing I/O timeouts.
  • Proper switch redundancy still required: Even with shared NICs, you still need redundant switches to avoid single point of failure.

Architecture: In this configuration, physical NICs are trunked and carry multiple VLANs:

  • VLAN 10: Management traffic (Proxmox web UI, SSH) - uses both NICs via bond, which ensures management access even if one NIC fails.
  • VLAN 20: VM/LXC bridge traffic - uses both NICs via bond, which provides redundancy for VM network connectivity.
  • VLAN 100: Storage traffic (NVMe-TCP) - each NIC has separate VLAN interface (no bond) as bonding would reduce 8 paths to 1 path, eliminating multipath benefits; separate interfaces preserve all redundant paths for NVMe native multipathing.

Configuration Example: 

# /etc/network/interfaces
# Physical NICs configured for trunking (no IP, just up)
auto ens1f0
iface ens1f0 inet manual
    bond-master bond0
    mtu 9000

auto ens1f1
iface ens1f1 inet manual
    bond-master bond0
    mtu 9000

# Bond interface for management and VM traffic
auto bond0
iface bond0 inet manual
    bond-slaves ens1f0 ens1f1
    bond-mode active-backup
    bond-miimon 100
    bond-primary ens1f0
    mtu 9000

# Management VLAN 10 on bond (single management interface)
auto bond0.10
iface bond0.10 inet static
    address 192.168.10.101/24
    gateway 192.168.10.1
    vlan-raw-device bond0

# VM/LXC Bridge on VLAN 20 (single bridge using bond)
auto vmbr0
iface vmbr0 inet manual
    bridge-ports bond0.20
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 20-50

# Storage VLAN 100 - NVMe-TCP ONLY (NO BOND - separate paths)
# Each NIC gets its own VLAN interface on same subnet
auto ens1f0.100
iface ens1f0.100 inet static
    address 10.100.1.101/24
    vlan-raw-device ens1f0
    mtu 9000

auto ens1f1.100
iface ens1f1.100 inet static
    address 10.100.1.102/24
    vlan-raw-device ens1f1
    mtu 9000
Important:

  • Physical NICs are trunked to carry multiple VLANs: allows a single physical connection to carry multiple isolated networks.

  • Management and VM traffic: Use bond0 (active-backup) for redundancy: provides failover without requiring switch configuration (unlike LACP), making the configuration simple and reliable.

  • Storage traffic: Does NOT use bond - each NIC has separate VLAN interface for multipath: bonding would aggregate paths into one, defeating the purpose of NVMe multipathing. Separate interfaces maintain 8 independent paths (2 NICs × 4 portals).

  • Storage VLAN (100) is isolated from other traffic at Layer 2: prevents accidental routing of storage traffic and enables VLAN-specific QoS and security policies.

  • Both storage VLAN interfaces on same subnet (10.100.1.0/24): simplifies routing; all paths can reach all portals without complex routing; matches storage array configuration.

  • Single management interface (bond0.10) uses both NICs: consolidates management traffic, provides redundancy and simplifies firewall rules.

  • Single VM bridge (vmbr0) uses both NICs: provides redundant network path for VMs and simplifies VM network configuration.

  • QoS should prioritize storage VLAN traffic: ensures storage I/O is not delayed by VM traffic, prevents I/O timeouts and maintains consistent performance

  • Two switches provide redundancy even in converged infrastructure: eliminates switch as single point of failure and allows switch maintenance without downtime

Network Design Best Practices

Practice Recommendation Rationale
MTU Size 9000 (Jumbo Frames) Reduces CPU overhead by ~30%; improves throughput by reducing packet count; lowers interrupt rate; must be configured end-to-end (NICs, switches, storage)
Number of NICs Minimum 2 per node Provides redundancy (one NIC can fail); enables load distribution across paths; recommended 2-4 NICs depending on performance requirements
Switch Redundancy Minimum 2 switches Eliminates switch as single point of failure; allows switch firmware updates without downtime; each NIC connects to different switch
IP Addressing Same subnet for all paths Simplifies routing (no need for static routes); all NICs can reach all portals directly; easier troubleshooting; matches most storage array configurations
Link Speed 10 GbE minimum, 25/100 GbE preferred 10 GbE provides ~1 GB/s per link; 25/100 GbE needed for high-performance NVMe arrays; must match or exceed storage array capability to avoid bottleneck
Flow Control Enable on storage ports Prevents packet drops during bursts; essential for lossless storage traffic; configure on both NIC and switch ports
QoS Priority for storage VLANs Ensures storage traffic is not delayed by other traffic; prevents I/O timeouts; critical when sharing NICs with VM traffic; recommend highest priority for storage VLAN

Subnet Design Example

  • Storage Network: 10.100.1.0/24 (VLAN 100)

  • Proxmox Nodes:

    • Node 1 NIC 1: 10.100.1.101 (ens1f0 or ens1f0.100)

    • Node 1 NIC 2: 10.100.1.102 (ens1f1 or ens1f1.100)

    • Node 2 NIC 1: 10.100.1.111 (ens1f0 or ens1f0.100)

    • Node 2 NIC 2: 10.100.1.112 (ens1f1 or ens1f1.100)

    • Node 3 NIC 1: 10.100.1.121 (ens1f0 or ens1f0.100)

    • Node 3 NIC 2: 10.100.1.122 (ens1f1 or ens1f1.100)

  • Storage Array Portals:

    • Portal 1 (Controller 1): 10.100.1.10

    • Portal 2 (Controller 1): 10.100.1.11

    • Portal 3 (Controller 2): 10.100.1.12

    • Portal 4 (Controller 2): 10.100.1.13

  • Result: Each node has 8 paths (2 NICs × 4 portals)

Why use this IP scheme:

  • Single /24 subnet: Simplifies routing; all devices can communicate directly without routing
  • Storage array uses .10-.19: Reserved range for storage infrastructure; easy to identify in logs
  • Nodes use .100+: Clear separation from storage; each node gets 10 IPs (e.g., Node 1: .101-.110)
  • Sequential NIC numbering: .101, .102 for Node 1; .111, .112 for Node 2; easy to remember and troubleshoot