iSCSI on Debian/Ubuntu - Performance Tuning

Linux
Audience
Public
Product
FlashBlade
FlashArray
Technology Integrations
Linux
Source Type
Documentation
Warning:

This content is for reference only. Always consult official vendor documentation for your distribution. Test thoroughly in a lab environment before production use. In case of conflicts, vendor documentation takes precedence.

Network Performance Optimization

MTU Configuration (Jumbo Frames)

Why use jumbo frames:

  • Reduces CPU overhead by lowering packet count and interrupt rate

  • Improves throughput for large sequential I/O (actual gains vary by workload)

  • Lowers interrupt rate

  • Recommended for high-performance storage (validate with benchmarks)

Configuration: 

# Set MTU 9000 on storage interfaces
ip link set eth0 mtu 9000
ip link set eth1 mtu 9000

# Verify
ip link show eth0 | grep mtu
Important: MTU must be 9000 end-to-end (host -> switch -> storage)

TCP Tuning

Optimize TCP for storage traffic: 

# /etc/sysctl.d/99-iscsi-tuning.conf
# Increase TCP buffer sizes
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864

# Increase connection tracking
net.netfilter.nf_conntrack_max = 1048576

# Optimize for low latency
net.ipv4.tcp_low_latency = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_timestamps = 1

# Apply settings
sysctl -p /etc/sysctl.d/99-iscsi-tuning.conf

iSCSI Session Tuning

Queue Depth

Increase queue depth for better performance: 

# Check current queue depth
cat /sys/block/sda/device/queue_depth

# Increase queue depth (per device)
echo 128 > /sys/block/sda/device/queue_depth

# Make persistent via udev rule (adjust vendor to match your storage)
# /etc/udev/rules.d/99-iscsi-queue-depth.rules
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{device/vendor}=="VENDOR*", ATTR{device/queue_depth}="128"

Recommended values:

  • SSD/Flash storage: 128-256

  • HDD storage: 32-64

Session Parameters

Optimize iSCSI session settings: 

# Increase max receive data segment length
iscsiadm -m node -T <target_iqn> -p <portal_ip> \
    -o update -n node.conn[0].iscsi.MaxRecvDataSegmentLength -v 262144

# Increase first burst length
iscsiadm -m node -T <target_iqn> -p <portal_ip> \
    -o update -n node.session.iscsi.FirstBurstLength -v 262144

# Increase max burst length
iscsiadm -m node -T <target_iqn> -p <portal_ip> \
    -o update -n node.session.iscsi.MaxBurstLength -v 1048576

# Enable immediate data
iscsiadm -m node -T <target_iqn> -p <portal_ip> \
    -o update -n node.conn[0].iscsi.ImmediateData -v Yes

# Increase number of outstanding R2Ts
iscsiadm -m node -T <target_iqn> -p <portal_ip> \
    -o update -n node.session.iscsi.MaxOutstandingR2T -v 1

I/O Scheduler Optimization

For SSD/Flash storage: 

# Use 'none' or 'noop' scheduler for flash storage
echo none > /sys/block/sda/queue/scheduler

# Make persistent via udev rule
# /etc/udev/rules.d/99-iscsi-scheduler.rules
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="none"

For HDD storage: 

# Use 'mq-deadline' for HDD
echo mq-deadline > /sys/block/sda/queue/scheduler

CPU and IRQ Optimization

IRQ Affinity

Distribute interrupts across CPUs: 

# Install irqbalance
# RHEL/Rocky/AlmaLinux:
dnf install -y irqbalance

# Debian/Ubuntu:
apt install -y irqbalance

# Enable and start
systemctl enable --now irqbalance

# Or manually set IRQ affinity
# Find IRQ for network interface
grep eth0 /proc/interrupts

# Set IRQ to specific CPU (example: IRQ 45 to CPU 2)
echo 4 > /proc/irq/45/smp_affinity  # 4 = binary 0100 = CPU 2

CPU Isolation (Advanced)

Dedicate CPUs to storage I/O: 

# Edit kernel boot parameters
# /etc/default/grub
GRUB_CMDLINE_LINUX="isolcpus=2,3,10,11"

# Update grub
# RHEL/Rocky/AlmaLinux:
grub2-mkconfig -o /boot/grub2/grub.cfg

# Debian/Ubuntu:
update-grub

# Reboot required
reboot
Warning:

CPU isolation (isolcpus) is a general system optimization for I/O-intensive workloads. It does not directly affect iSCSI protocol behavior. Measure baseline performance before and after changes to validate impact in your environment.

Read-Ahead Tuning

Optimize read-ahead for workload: 

# Check current read-ahead
blockdev --getra /dev/sda

# For random I/O workloads (databases):
blockdev --setra 256 /dev/sda  # 128 KB

# For sequential I/O workloads (file servers):
blockdev --setra 8192 /dev/sda  # 4 MB

# Make persistent via udev rule (adjust vendor to match your storage)
# /etc/udev/rules.d/99-iscsi-readahead.rules
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{device/vendor}=="VENDOR*", ATTR{bdi/read_ahead_kb}="128"

Monitoring Performance

Key metrics to monitor: 

# I/O statistics
iostat -x 1

# Network statistics
sar -n DEV 1

# iSCSI session statistics
iscsiadm -m session -P 3 | grep -A 20 "iSCSI Session State"

# Multipath I/O statistics
dmsetup status

# System-wide I/O
vmstat 1

Performance indicators:

  • Low latency: < 1ms for flash storage

  • High throughput: Near line speed (10Gbps = ~1.2 GB/s)

  • Low CPU wait: iowait < 5%

  • Balanced paths: Even I/O distribution across all paths

Performance Tuning (iSCSI)

System-Level Optimizations

CPU Isolation for I/O Threads

Isolate CPUs for storage processing: 

# In /etc/default/grub, add to GRUB_CMDLINE_LINUX:
GRUB_CMDLINE_LINUX="isolcpus=2,3 nohz_full=2,3 rcu_nocbs=2,3"
Warning:

CPU isolation (isolcpus) is a general system optimization for I/O-intensive workloads. It does not directly affect iSCSI protocol behavior. Measure baseline performance before and after changes to validate impact in your environment.

Best Practice:

  • Use isolcpus only if CPU contention is observed (high %sys during I/O)

  • Monitor with mpstat -P ALL 1 to identify busy cores

Memory Configuration

Hugepages for large I/O buffers: 

# /etc/sysctl.d/99-storage-performance.conf
vm.nr_hugepages = 1024

# For transparent hugepages (alternative)
echo always > /sys/kernel/mm/transparent_hugepage/enabled

Disable NUMA balancing for predictable latency: 

echo 0 > /proc/sys/kernel/numa_balancing

Network Optimizations

TCP Buffer Tuning

Increase TCP buffer sizes for high throughput: 

# /etc/sysctl.d/99-storage-tcp.conf

# TCP buffer sizes (min, default, max)
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

# Connection queue sizes
net.core.netdev_max_backlog = 30000
net.core.somaxconn = 4096

Network Interface Tuning

Increase ring buffer size: 

# Check current settings
ethtool -g ens1f0

# Increase to maximum
ethtool -G ens1f0 rx 4096 tx 4096

Enable receive-side scaling (RSS): 

# Check RSS configuration
ethtool -l ens1f0

# Set number of channels
ethtool -L ens1f0 combined 8

Interrupt coalescing for throughput: 

# Reduce interrupts for high throughput
ethtool -C ens1f0 rx-usecs 100 tx-usecs 100

# For low latency (more interrupts)
ethtool -C ens1f0 rx-usecs 0 tx-usecs 0

Storage Layer Optimizations

I/O Scheduler Tuning

For SCSI/iSCSI Devices

Recommended: mq-deadline or none 

# Check current scheduler
cat /sys/block/sda/queue/scheduler

# Set to mq-deadline (good for mixed workloads)
echo mq-deadline > /sys/block/sda/queue/scheduler

# Or set to none (lowest latency)
echo none > /sys/block/sda/queue/scheduler

Why: Modern storage arrays handle I/O scheduling internally; kernel scheduler adds latency.

Make persistent: 

# Create udev rule: /etc/udev/rules.d/60-iosched.rules
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/scheduler}="mq-deadline"

Queue Depth Tuning 

# Check current queue depth
cat /sys/block/sda/queue/nr_requests

# Increase to 256 (default is often 64 or 128)
echo 256 > /sys/block/sda/queue/nr_requests

Read-Ahead Configuration 

# Check current read-ahead
blockdev --getra /dev/sda

# Set read-ahead (in 512-byte sectors)
# 16384 = 8MB read-ahead (good for sequential workloads)
blockdev --setra 16384 /dev/sda

iSCSI-Specific Tuning

iSCSI Session Parameters

Optimize session settings in /etc/iscsi/iscsid.conf: 

# Faster failover
node.session.timeo.replacement_timeout = 20

# Larger MaxRecvDataSegmentLength (up to 16MB)
node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144

# Enable immediate data (reduces latency)
node.session.iscsi.ImmediateData = Yes
node.session.iscsi.FirstBurstLength = 262144
node.session.iscsi.MaxBurstLength = 16776192

Multipath Performance

Configure for throughput in /etc/multipath.conf: 

defaults {
    path_grouping_policy    multibus     # Use all paths simultaneously
    path_selector           "round-robin 0"
    failback                immediate
    no_path_retry           queue
}

Monitoring Performance

Real-time I/O monitoring: 

# Per-device statistics
iostat -xz 1

# Watch key metrics:
# - await: Average wait time (ms) - should be <20ms for iSCSI
# - %util: Utilization - sustained >80% may indicate bottleneck
# - r/s, w/s: IOPS
# - rkB/s, wkB/s: Throughput

iSCSI session statistics: 

# Check session parameters
iscsiadm -m session -P 3 | grep -E "Header|Data|Burst"

Debian/Ubuntu-Specific Tuning

Kernel Parameters

Optimize kernel for iSCSI: 

# Create sysctl configuration
sudo tee /etc/sysctl.d/99-iscsi-debian.conf > /dev/null <<'EOF'
# Network performance
net.core.netdev_max_backlog = 5000
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864

# Connection tracking
net.netfilter.nf_conntrack_max = 1048576

# Low latency
net.ipv4.tcp_low_latency = 1

# VM tuning for storage
vm.dirty_ratio = 10
vm.dirty_background_ratio = 5
vm.swappiness = 10

# ARP settings for same-subnet multipath (CRITICAL)
# Prevents ARP responses on wrong interface when multiple NICs share same subnet
# See: Network Concepts documentation for detailed explanation
net.ipv4.conf.all.arp_ignore = 2
net.ipv4.conf.default.arp_ignore = 2
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_announce = 2
# Interface-specific (adjust interface names as needed)
net.ipv4.conf.ens1f0.arp_ignore = 2
net.ipv4.conf.ens1f1.arp_ignore = 2
net.ipv4.conf.ens1f0.arp_announce = 2
net.ipv4.conf.ens1f1.arp_announce = 2
EOF

# Apply settings
sudo sysctl -p /etc/sysctl.d/99-iscsi-debian.conf

I/O Scheduler

Set I/O scheduler for iSCSI devices: 

# Create udev rule
sudo tee /etc/udev/rules.d/99-iscsi-scheduler.rules > /dev/null <<'EOF'
# Set I/O scheduler for iSCSI devices (SSD/Flash)
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="none"

# Set queue depth (adjust vendor to match your storage)
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{device/vendor}=="VENDOR*", ATTR{device/queue_depth}="128"

# Set read-ahead (adjust vendor to match your storage)
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{device/vendor}=="VENDOR*", ATTR{bdi/read_ahead_kb}="128"
EOF

# Reload udev rules
sudo udevadm control --reload-rules
sudo udevadm trigger

IRQ Affinity

Install and configure irqbalance: 

# Install irqbalance
sudo apt install -y irqbalance

# Enable and start
sudo systemctl enable --now irqbalance

# Check IRQ distribution
cat /proc/interrupts | grep -E "eth|ens"