Experiencing High Latency While Using NVMe-TCP on RHEL/Rocky/AlmaLinux

NVMe-TCP on RHEL/Rocky/AlmaLinux

• Slow I/O performance
• High await times in iostat
• Application timeouts

Diagnosis:

# Check I/O latency
iostat -x 1 5

# Monitor queue depth
cat /sys/block/nvme0n1/queue/nr_requests

# Check for CPU bottlenecks
top
mpstat -P ALL 1

# Check network latency
ping <portal_ip>

# Check path status and IO policy
nvme list-subsys
cat /sys/class/nvme-subsystem/nvme-subsys*/iopolicy

Solutions:

1. Network latency:
   • Check for network congestion
   • Verify QoS settings
   • Check for routing issues
   • Verify jumbo frames enabled
2. CPU bottleneck:
   • Check IRQ distribution
   • Enable IRQ affinity
   • Verify offload features enabled
3. Storage array issue:
   • Check array performance metrics
   • Verify no controller failover in progress
   • Check for array-side bottlenecks
4. IO policy not optimal:
   • Verify IO policy is round-robin for balanced distribution
   • Check ANA state if using ANA