Network Configuration

Network Architecture Principles

Dedicated Storage Network: Always use dedicated physical or VLAN-isolated networks for storage traffic
- Why: Isolates storage I/O from other network traffic; prevents bandwidth contention; enables QoS policies specific to storage; simplifies security and firewall rules
No Single Points of Failure: Redundant switches, NICs, and storage controllers
- Why: Any single component can fail without impacting storage availability; enables zero-downtime maintenance; critical for production environments
Proper Segmentation: Separate storage traffic from management and VM traffic
- Why: Prevents noisy neighbor problems; ensures storage performance is not affected by VM traffic spikes; improves security posture; simplifies network troubleshooting
Optimized MTU: Use jumbo frames (MTU 9000) end-to-end when possible
- Why: Reduces CPU overhead and improves throughput by reducing packet count; lowers interrupt rate; recommended for high-performance storage (actual gains vary by workload)

Network Topology Options

Option 1: Dedicated Physical Networks (Recommended): This is the gold standard for production storage networks because it provides complete isolation and maximum performance.

Advantages:

Maximum performance and isolation: Storage has dedicated bandwidth with zero contention
No contention with other traffic: VM traffic spikes cannot impact storage performance
Simplified troubleshooting: Storage network issues are isolated and easier to diagnose
Predictable latency: Consistent performance without interference from other workloads

Requirements:

Dedicated NICs on each node (minimum 2 for redundancy): Provides redundancy; one NIC can fail without storage outage.
Dedicated storage switches (minimum 2 for redundancy): Eliminates switch as single point of failure; allows switch maintenance without downtime.
Direct connections to storage array: Reduces latency; simplifies network path; fewer points of failure.

Option 2: VLAN-Based Segmentation (Shared NICs): Choose this option when you have limited physical NICs or want to consolidate infrastructure. Acceptable for environments where cost/port density is a concern and you can accept some performance trade-offs.

Advantages:

Efficient use of physical infrastructure: Reduces NIC and switch port requirements
Flexible network design: Easy to add new VLANs without physical changes
Single set of NICs handles multiple traffic types: Lower hardware costs; fewer cables

Considerations:

NICs are shared with management, VM, and container traffic: Storage competes for bandwidth with other traffic; potential performance degradation during high VM traffic.
Requires proper VLAN configuration and trunking on switches: Misconfiguration can cause traffic leakage or complete storage outage.
Storage VLANs share bandwidth with other VLANs on same NICs: Maximum storage throughput is limited by total NIC bandwidth minus other traffic.
Requires QoS/traffic prioritization for storage VLANs: Without QoS, VM traffic can starve storage traffic causing I/O timeouts.
Proper switch redundancy still required: Even with shared NICs, you still need redundant switches to avoid single point of failure.

MTU Configuration Best Practices

Jumbo Frames (MTU 9000):

Performance Impact: Improved throughput for large sequential I/O (actual gains vary by workload; validate with benchmarks)
CPU Reduction: Fewer packets to process means lower CPU overhead
Latency: Slightly reduced latency for large transfers

Requirements for Jumbo Frames:

End-to-end configuration: ALL devices in the path must support MTU 9000 (host > switches > storage.

Verification: Test with ping to ensure no fragmentation:

# Test MTU 9000 (8972 bytes + 28 byte header = 9000)
ping -M do -s 8972 <storage_portal_ip>

When NOT to use Jumbo Frames:

Network infrastructure doesn't support it
Mixed environments with devices that don't support it
Troubleshooting network issues (start with MTU 1500, then increase)

Network Performance Tuning

NIC Ring Buffers:

# Increase RX/TX ring buffers to reduce packet drops
ethtool -G <interface> rx 4096 tx 4096

Interrupt Coalescing:

# Reduce interrupt rate for better throughput (trade-off: slightly higher latency)
ethtool -C <interface> rx-usecs 50 tx-usecs 50

RSS (Receive Side Scaling):

# Distribute network processing across multiple CPU cores
ethtool -L <interface> combined 4

Flow Control:

# Enable flow control to prevent buffer overruns
ethtool -A <interface> rx on tx on

IP Addressing Approaches

There are two valid approaches for assigning IP addresses to multiple storage interfaces:

Option A: Same Subnet (Recommended for Simplicity)

Configuration:

Interface 1: 10.100.1.101/24
Interface 2: 10.100.1.102/24
Storage Array: 10.100.1.10-19/24

Advantages:

Simpler routing: All devices communicate directly without routing between subnets
Single VLAN: Only one VLAN needed for storage traffic
Easy to understand: Clear, sequential IP addressing scheme
Fewer firewall rules: No inter-VLAN routing to configure

Considerations:

Both interfaces are on the same L2 broadcast domain
Requires careful planning to avoid IP conflicts
Multipath relies on interface binding, not separate subnets
Requires ARP configuration (see ARP Configuration for Same-Subnet Multipath below)

Best for: Proxmox clusters, environments with straightforward network design

Option B: Different Subnets (Recommended for Isolation)

Configuration:

Interface 1: 10.100.1.101/24 (VLAN 100)
Interface 2: 10.100.2.101/24 (VLAN 101)
Storage Array: 10.100.1.10/24 and 10.100.2.10/24

Advantages:

Network path isolation: Each subnet can use different physical paths
Failure domain separation: Issues in one subnet don't affect the other
Easier troubleshooting: Clear separation of traffic per subnet
Required for some switch designs: Necessary if using separate switch fabrics

Considerations:

Storage array needs IPs in both subnets
Requires two VLANs and potentially more complex switch configuration
May need policy-based routing on hosts

Best for: Enterprise environments requiring strict network isolation

Choosing the Right Approach

Factor	Same Subnet	Different Subnets
Network complexity	Lower	Higher
Failure isolation	Shared L2 domain	Separate L2 domains
Switch requirements	Single VLAN	Multiple VLANs
Multipath method	Interface binding	Subnet-based paths
Routing	None required	May need policy routing

Note: Both approaches work correctly with NVMe-TCP multipath. NVMe-TCP uses --host-iface and --host-traddr parameters to bind connections to specific interfaces. Choose based on your network infrastructure requirements and organizational preferences.

Oracle Linux Network Best Practices

Use NetworkManager for consistency:

# Ensure NetworkManager is running
sudo systemctl enable --now NetworkManager

# Disable legacy network service if present
sudo systemctl disable network

Network Interface Bonding (Optional)

For additional redundancy, consider bonding:

# Create bond interface
sudo nmcli connection add type bond \
    con-name bond-storage \
    ifname bond0 \
    mode active-backup \
    miimon 100

# Add slave interfaces
sudo nmcli connection add type ethernet \
    con-name bond-storage-slave1 \
    ifname ens1f0 \
    master bond0

sudo nmcli connection add type ethernet \
    con-name bond-storage-slave2 \
    ifname ens1f1 \
    master bond0

# Configure IP on bond
sudo nmcli connection modify bond-storage \
    ipv4.method manual \
    ipv4.addresses 10.100.1.101/24 \
    802-3-ethernet.mtu 9000

# Activate
sudo nmcli connection up bond-storage

Note: Bonding provides NIC-level redundancy in addition to NVMe multipath.