Connections Drop Intermittently When Using NVMe-TCP or iSCSI on RHEL/Rocky/AlmaLinux

NVMe-TCP or iSCSI on RHEL/Rocky/AlmaLinux

• Connections establish but drop randomly
• I/O errors in logs
• Multipath shows paths going up and down

• Network instability
• MTU mismatch
• Timeout settings too aggressive

Diagnosis:

# Monitor connection state
watch -n 1 'nvme list-subsys'  # NVMe-TCP
watch -n 1 'iscsiadm -m session'  # iSCSI

# Check for network errors
ethtool -S <interface> | grep -i error

# Check system logs
journalctl -f | grep -i "nvme\|iscsi\|multipath"

# Monitor packet drops
netstat -i

Solutions:

1. Network instability:
   • Check for packet loss: ping -c 100 <portal_ip>
   • Check for high latency: ping -c 100 <portal_ip> | tail -1
   • Verify no spanning tree issues on switches
   • Check for duplex mismatches: ethtool <interface> | grep Duplex
2. MTU mismatch:
   • Test MTU: ping -M do -s 8972 <portal_ip>
   • Verify MTU on all devices in path
   • Set consistent MTU end-to-end
3. Timeout settings too aggressive:
   • Increase timeout values
   • Check ctrl_loss_tmo (NVMe-TCP)
   • Check replacement_timeout (iSCSI)