NVMe-TCP on Debian/Ubuntu - High Availability

Path Redundancy Model

Queue-Depth IO Policy

IO Policy Comparison

Policy	Behavior	Best For
queue-depth	Routes to path with lowest queue	Mixed workloads (recommended)
round-robin	Rotates through paths equally	Uniform latency paths
numa	Prefers NUMA-local paths	NUMA-optimized systems

Enabling Native Multipath

# Add kernel parameter
echo 'options nvme_core multipath=Y' > /etc/modprobe.d/nvme-tcp.conf

# Verify after reboot
cat /sys/module/nvme_core/parameters/multipath
# Output: Y

NVMe-TCP Failover Sequence

NVMe-TCP Failover Parameters

Parameter	Default	Recommended	Description
`ctrl-loss-tmo`	600s	1800s	Time before controller considered lost
`reconnect-delay`	10s	10s	Delay between reconnection attempts
`nr_io_queues`	CPU count	-	Number of IO queues per controller

Tip:

NVMe-TCP uses native NVMe multipathing built into the Linux kernel. This is NOT dm-multipath (multipath.conf, multipathd) - those are for iSCSI/Fibre Channel only.

NVMe-TCP APD (All Paths Down) Behavior

When all paths to an NVMe controller are lost:

ctrl-loss-tmo timer starts - Default 600s (recommended: 1800s for production)
Reconnection attempts - NVMe driver attempts reconnection every reconnect-delay seconds
If timer expires - Controller is removed and I/O fails to application
If path recovers - I/O automatically resumes without intervention

Native NVMe Multipath Configuration for HA

NVMe-TCP uses native NVMe multipathing built into the Linux kernel. This is NOT dm-multipath (multipath.conf, multipathd) - those are for iSCSI/Fibre Channel only.

Enable Native NVMe Multipath:

# Enable native NVMe multipathing
echo 'options nvme_core multipath=Y' | sudo tee /etc/modprobe.d/nvme-tcp.conf

# Reboot to apply (required if nvme_core already loaded)
sudo reboot

Configure IO Policy for HA:

# Create udev rule for NVMe IO policy
sudo tee /etc/udev/rules.d/99-nvme-iopolicy.rules > /dev/null <<'EOF'
# Set IO policy to queue-depth for all NVMe subsystems (recommended for HA)
ACTION=="add|change", SUBSYSTEM=="nvme-subsystem", ATTR{iopolicy}="queue-depth"
EOF

# Reload udev rules
sudo udevadm control --reload-rules
sudo udevadm trigger

Configure NVMe Connection Timeouts for HA:

# When connecting, use appropriate timeout values
# ctrl-loss-tmo: Time to wait before declaring controller lost (seconds)
# reconnect-delay: Delay between reconnection attempts (seconds)

# Example: Conservative HA settings
nvme connect -t tcp -a <IP> -s 4420 -n <NQN> \
    --ctrl-loss-tmo=1800 \
    --reconnect-delay=10

# For faster failover (may cause more transient errors):
nvme connect -t tcp -a <IP> -s 4420 -n <NQN> \
    --ctrl-loss-tmo=600 \
    --reconnect-delay=5

Verify Native Multipath Status:

# Check multipath is enabled
cat /sys/module/nvme_core/parameters/multipath
# Should show: Y

# View all paths per subsystem
sudo nvme list-subsys

# Check IO policy
cat /sys/class/nvme-subsystem/nvme-subsys*/iopolicy

Systemd Service Dependencies

Ensure proper boot order:

# Create drop-in for services that depend on NVMe storage
sudo mkdir -p /etc/systemd/system/libvirtd.service.d

sudo tee /etc/systemd/system/libvirtd.service.d/storage.conf > /dev/null <<EOF
[Unit]
After=nvmf-autoconnect.service
Wants=nvmf-autoconnect.service
EOF

# Reload systemd
sudo systemctl daemon-reload

Monitoring and Alerting

Set up monitoring with systemd:

# Create monitoring script for native NVMe multipath
sudo tee /usr/local/bin/check-nvme-paths.sh > /dev/null <<'EOF'
#!/bin/bash

# Check native NVMe multipath status (NOT dm-multipath)
# Count connections that are NOT in 'live' state
FAILED=$(nvme list-subsys 2>/dev/null | grep -c -E "connecting|deleting")

if [ $FAILED -gt 0 ]; then
    echo "WARNING: $FAILED NVMe paths not in live state"
    nvme list-subsys
    exit 1
fi

# Check connection count
EXPECTED_CONNECTIONS=8
ACTUAL=$(nvme list-subsys 2>/dev/null | grep -c "live")

if [ $ACTUAL -lt $EXPECTED_CONNECTIONS ]; then
    echo "WARNING: Only $ACTUAL of $EXPECTED_CONNECTIONS NVMe connections active"
    nvme list-subsys
    exit 1
fi

echo "OK: All NVMe storage paths healthy"
exit 0
EOF

sudo chmod +x /usr/local/bin/check-nvme-paths.sh

# Create systemd timer
sudo tee /etc/systemd/system/check-nvme-paths.service > /dev/null <<EOF
[Unit]
Description=Check NVMe-TCP path health

[Service]
Type=oneshot
ExecStart=/usr/local/bin/check-nvme-paths.sh
StandardOutput=journal
EOF

sudo tee /etc/systemd/system/check-nvme-paths.timer > /dev/null <<EOF
[Unit]
Description=Check NVMe-TCP paths every 5 minutes

[Timer]
OnBootSec=5min
OnUnitActiveSec=5min

[Install]
WantedBy=timers.target
EOF

# Enable timer
sudo systemctl enable --now check-nvme-paths.timer