Network Configuration

NetworkManager Best Practices

Why NetworkManager:

• Default in RHEL 8/9
• Better integration with systemd
• Dynamic configuration support
• Consistent across RHEL ecosystem

Disable network-scripts (RHEL 8):

# Network scripts are deprecated
sudo systemctl disable network
sudo systemctl enable NetworkManager
sudo systemctl start NetworkManager

Storage Network Configuration

Dedicated Interfaces (Recommended)

# Create connection for storage interface
sudo nmcli connection add type ethernet \
    con-name storage-iscsi-1 \
    ifname <INTERFACE_1_NAME> \
    ipv4.method manual \
    ipv4.addresses <INTERFACE_1_IP>/<INTERFACE_1_CIDR> \
    ipv4.never-default yes \
    ipv4.may-fail no \
    802-3-ethernet.mtu 9000 \
    connection.autoconnect yes \
    connection.autoconnect-priority 10

# Optimize for storage
sudo nmcli connection modify storage-iscsi-1 \
    ethtool.ring-rx 4096 \
    ethtool.ring-tx 4096 \
    ethtool.coalesce-rx-usecs 50 \
    ethtool.coalesce-tx-usecs 50

# Activate
sudo nmcli connection up storage-iscsi-1

# Repeat for second interface
sudo nmcli connection add type ethernet \
    con-name storage-iscsi-2 \
    ifname <INTERFACE_2_NAME> \
    ipv4.method manual \
    ipv4.addresses <INTERFACE_2_IP>/<INTERFACE_2_CIDR> \
    ipv4.never-default yes \
    ipv4.may-fail no \
    802-3-ethernet.mtu 9000 \
    connection.autoconnect yes \
    connection.autoconnect-priority 10

sudo nmcli connection up storage-iscsi-2

Key NetworkManager parameters:

• ipv4.never-default yes - No default route on storage interface
• ipv4.may-fail no - Boot waits for this interface
• connection.autoconnect-priority 10 - Higher priority for storage
• ethtool.* - NIC tuning parameters

MTU Configuration

# Set MTU 9000 for jumbo frames
sudo nmcli connection modify storage-iscsi-1 802-3-ethernet.mtu 9000
sudo nmcli connection modify storage-iscsi-2 802-3-ethernet.mtu 9000

# Verify
nmcli connection show storage-iscsi-1 | grep mtu

# Test MTU
ping -M do -s 8972 <storage_portal_ip>

ARP Configuration for Same-Subnet Multipath

The Problem

When multiple network interfaces are assigned IP addresses in the same subnet (e.g., 10.100.1.101 on ens1f0 and 10.100.1.102 on ens1f1), Linux's default ARP behavior can cause problems:

Default Behavior (arp_ignore=0):

Storage Portal 10.100.1.10 sends ARP: "Who has 10.100.1.101?"
Without arp_ignore: BOTH ens1f0 AND ens1f1 might respond
Result: Storage array gets confused about which MAC address to use
Impact: Packets may be sent to wrong interface, breaking multipath

Why This Breaks Multipath:

• Storage array may cache the wrong MAC address for an IP

• Traffic sent to one interface but routed through another (asymmetric routing)

• Breaks multipath path selection - kernel expects traffic on specific interfaces

• Causes intermittent connection failures and performance issues

The Solution: arp_ignore and arp_announce

arp_ignore controls which interfaces respond to ARP requests:

arp_announce controls the source IP used when sending ARP requests:

Configuration

Create persistent sysctl configuration:

# For NVMe-TCP storage
cat > /etc/sysctl.d/99-nvme-tcp-arp.conf << 'EOF'
# ARP configuration for storage multipath
# Prevents ARP responses on wrong interface when multiple NICs share same subnet

# For dedicated physical interfaces (adjust interface names as needed)
net.ipv4.conf.ens1f0.arp_ignore = 2
net.ipv4.conf.ens1f1.arp_ignore = 2
net.ipv4.conf.ens1f0.arp_announce = 2
net.ipv4.conf.ens1f1.arp_announce = 2

# For VLAN interfaces (uncomment if using VLANs)
#net.ipv4.conf.ens1f0.100.arp_ignore = 2
#net.ipv4.conf.ens1f1.100.arp_ignore = 2
#net.ipv4.conf.ens1f0.100.arp_announce = 2
#net.ipv4.conf.ens1f1.100.arp_announce = 2

# Global settings (applies to all interfaces)
net.ipv4.conf.all.arp_ignore = 2
net.ipv4.conf.default.arp_ignore = 2
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_announce = 2
EOF

# For iSCSI storage (same settings, different file name for clarity)
cat > /etc/sysctl.d/99-iscsi-arp.conf << 'EOF'
# ARP configuration for iSCSI multipath
net.ipv4.conf.ens1f0.arp_ignore = 2
net.ipv4.conf.ens1f1.arp_ignore = 2
net.ipv4.conf.ens1f0.arp_announce = 2
net.ipv4.conf.ens1f1.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 2
net.ipv4.conf.default.arp_ignore = 2
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_announce = 2
EOF

# Apply settings immediately
sysctl -p /etc/sysctl.d/99-nvme-tcp-arp.conf
# or
sysctl -p /etc/sysctl.d/99-iscsi-arp.conf

Verification

# Verify settings are applied
sysctl net.ipv4.conf.ens1f0.arp_ignore
sysctl net.ipv4.conf.ens1f1.arp_ignore
# Expected output: net.ipv4.conf.ens1f0.arp_ignore = 2

# Test ARP behavior from another host on same subnet
arping -I <interface> 10.100.1.101
# Should only get response from ens1f0's MAC address, not ens1f1

# Monitor ARP traffic
tcpdump -i ens1f0 arp &
tcpdump -i ens1f1 arp &
# Each interface should only respond to ARP requests for its own IP

# Check ARP cache
ip neigh show
# Clear ARP cache if needed after configuration changes
ip neigh flush all

Why arp_ignore=2 is Critical for Storage Multipath

• Ensures correct path selection: Each interface only responds for its own IP

• Prevents asymmetric routing: Traffic sent to .101 always uses ens1f0, traffic to .102 always uses ens1f1

• Maintains multipath integrity: Storage array can reliably use all paths

• Avoids connection confusion: Each path remains distinct and predictable

When ARP Configuration is NOT Required

ARP configuration is not needed when using:

• Different subnets (Option B above): Each interface is in a different broadcast domain

• Single storage interface: No ARP confusion possible

• Bonded interfaces: Bond handles the single IP address

Reference: Linux Virtual Server - ARP Configuration