NVMe-TCP on Debian/Ubuntu - Quick Start Guide

Linux

Audience

Public

Product

FlashBlade

FlashArray

Technology Integrations

Linux

Source Type

Documentation

Important:

Vendor Documentation Priority: This guide is for reference only. Always consult official vendor documentation for your distribution. Test thoroughly in a lab environment before production use.

Debian 11+ or Ubuntu 20.04 LTS+
NVMe-TCP storage array with portal IPs and subsystem NQN
Dedicated storage network interfaces
Root or sudo access

Note:

New to NVMe-TCP? See the Storage Terminology Glossary.

Install NVMe Tools:

sudo apt update
sudo apt install -y nvme-cli
nvme version

Enable Native NVMe Multipath:

# Enable multipath BEFORE connecting (requires reboot to take effect)
echo 'options nvme_core multipath=Y' | sudo tee /etc/modprobe.d/nvme-tcp.conf
sudo reboot

Configure Network Interfaces:

Configure dedicated storage interfaces using Netplan (Ubuntu/Debian 11+):

sudo tee /etc/netplan/50-storage.yaml > /dev/null <<EOF
network:
  version: 2
  renderer: networkd
  ethernets:
    <INTERFACE_NAME_1>:
      addresses:
        - <HOST_IP_1>/<CIDR>
      mtu: 9000
      dhcp4: no
      dhcp6: no
    <INTERFACE_NAME_2>:
      addresses:
        - <HOST_IP_2>/<CIDR>
      mtu: 9000
      dhcp4: no
      dhcp6: no
EOF

sudo netplan apply
ip addr show

Warning:

If both interfaces are on the same subnet, configure ARP settings. See ARP Configuration.

Configure Firewall:
Allow all traffic on storage interfaces (recommended for dedicated storage networks):
```
# Using UFW
sudo ufw allow in on <INTERFACE_NAME_1>
sudo ufw allow in on <INTERFACE_NAME_2>
```
Note:
Alternative: For port filtering options, see Best Practices - Firewall Configuration.

Generate Host NQN:

sudo mkdir -p /etc/nvme
sudo nvme gen-hostnqn | sudo tee /etc/nvme/hostnqn
cat /etc/nvme/hostnqn

Register this NQN with your FlashArray's allowed hosts list.

Connect to FlashArray:

Option A: Connect-all (simplest method)

Discovers and connects to all available subsystems from a single portal:

# Replace <PORTAL_IP_1> with any storage portal IP
sudo nvme connect-all -t tcp -a <PORTAL_IP_1> -s 4420

# Verify connections
sudo nvme list-subsys

Option B: Individual connections (for specific interface binding)

Use when you need to bind connections to specific host interfaces:

# Replace values: <PORTAL_IP_X>, <SUBSYSTEM_NQN>, <INTERFACE_NAME_X>, <HOST_IP_X>
sudo nvme connect -t tcp -a <PORTAL_IP_1> -s 4420 -n <SUBSYSTEM_NQN> \
    --host-iface=<INTERFACE_NAME_1> --host-traddr=<HOST_IP_1> \
    --ctrl-loss-tmo=1800 --reconnect-delay=10

sudo nvme connect -t tcp -a <PORTAL_IP_2> -s 4420 -n <SUBSYSTEM_NQN> \
    --host-iface=<INTERFACE_NAME_1> --host-traddr=<HOST_IP_1> \
    --ctrl-loss-tmo=1800 --reconnect-delay=10

# Repeat for all portal/interface combinations to create full mesh

# Verify connections
sudo nvme list-subsys

Configure IO Policy:

# Create udev rule for queue-depth IO policy
sudo tee /etc/udev/rules.d/99-nvme-iopolicy.rules > /dev/null <<'EOF'
ACTION=="add|change", SUBSYSTEM=="nvme-subsystem", ATTR{iopolicy}="queue-depth"
EOF

sudo udevadm control --reload-rules
sudo udevadm trigger

Configure Persistent Connections:

# Create discovery configuration
sudo tee /etc/nvme/discovery.conf > /dev/null <<EOF
-t tcp -a <PORTAL_IP_1> -s 4420 --host-traddr=<HOST_IP_1> --host-iface <INTERFACE_NAME_1>
-t tcp -a <PORTAL_IP_2> -s 4420 --host-traddr=<HOST_IP_1> --host-iface <INTERFACE_NAME_1>
-t tcp -a <PORTAL_IP_3> -s 4420 --host-traddr=<HOST_IP_1> --host-iface <INTERFACE_NAME_1>
-t tcp -a <PORTAL_IP_4> -s 4420 --host-traddr=<HOST_IP_1> --host-iface <INTERFACE_NAME_1>

-t tcp -a <PORTAL_IP_1> -s 4420 --host-traddr=<HOST_IP_2> --host-iface <INTERFACE_NAME_2>
-t tcp -a <PORTAL_IP_2> -s 4420 --host-traddr=<HOST_IP_2> --host-iface <INTERFACE_NAME_2>
-t tcp -a <PORTAL_IP_3> -s 4420 --host-traddr=<HOST_IP_2> --host-iface <INTERFACE_NAME_2>
-t tcp -a <PORTAL_IP_4> -s 4420 --host-traddr=<HOST_IP_2> --host-iface <INTERFACE_NAME_2>
EOF

# Enable autoconnect service
sudo systemctl enable --now nvmf-autoconnect.service

Create LVM Storage:

# Find NVMe device
sudo nvme list
# Example: /dev/nvme0n1

# Create LVM
sudo pvcreate /dev/nvme0n1
sudo vgcreate nvme-storage /dev/nvme0n1
sudo lvcreate -L 500G -n data nvme-storage

# Format and mount (Debian/Ubuntu: ext4 recommended)
sudo mkfs.ext4 /dev/nvme-storage/data
sudo mkdir -p /mnt/nvme-storage
sudo mount /dev/nvme-storage/data /mnt/nvme-storage

# Add to fstab
echo '/dev/nvme-storage/data /mnt/nvme-storage ext4 defaults,_netdev 0 0' | sudo tee -a /etc/fstab

Verify Configuration:

# Check multipath is enabled
cat /sys/module/nvme_core/parameters/multipath  # Should show: Y

# Check all paths are live
sudo nvme list-subsys

# Check IO policy
cat /sys/class/nvme-subsystem/nvme-subsys*/iopolicy  # Should show: queue-depth

# Verify block devices are visible
lsblk | grep nvme

# Verify storage (after LVM/filesystem creation)
df -h | grep nvme

Verify Persistent Connections (recommended):

# Reboot to confirm connections re-establish automatically
sudo reboot

# After reboot, verify all paths reconnected
sudo nvme list-subsys  # All paths should show "live"

Note:

Quick Reference

Command	Description
`sudo nvme list-subsys`	Show all paths and subsystems
`sudo nvme list`	List NVMe devices
`sudo nvme connect -t tcp -a <IP> -s 4420 -n <NQN>`	Connect to subsystem
`sudo nvme disconnect -n <NQN>`	Disconnect from subsystem

For production deployments, see NVMe-TCP Best Practices for:

Network design and VLAN configuration
Performance tuning
Security best practices (AppArmor, firewall options)
Monitoring and troubleshooting
Netplan and interfaces alternatives

Additional Resources: