Performance Tuning

Network Performance Optimization

MTU Configuration (Jumbo Frames)

Why use jumbo frames:

Reduces CPU overhead by lowering packet count and interrupt rate
Improves throughput for large sequential I/O (actual gains vary by workload)
Lowers interrupt rate
Recommended for high-performance storage (validate with benchmarks)

Configuration:

# Set MTU 9000 on storage interfaces
ip link set eth0 mtu 9000
ip link set eth1 mtu 9000

# Verify
ip link show eth0 | grep mtu

Important: MTU must be 9000 end-to-end (host → switch → storage).

TCP Tuning

Optimize TCP for storage traffic:

# /etc/sysctl.d/99-iscsi-tuning.conf
# Increase TCP buffer sizes
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864

# Increase connection tracking
net.netfilter.nf_conntrack_max = 1048576

# Optimize for low latency
net.ipv4.tcp_low_latency = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_timestamps = 1

# Apply settings
sysctl -p /etc/sysctl.d/99-iscsi-tuning.conf

iSCSI Session Tuning

Queue Depth

Increase queue depth for better performance:

# Check current queue depth
cat /sys/block/sda/device/queue_depth

# Increase queue depth (per device)
echo 128 > /sys/block/sda/device/queue_depth

# Make persistent via udev rule (adjust vendor to match your storage)
# /etc/udev/rules.d/99-iscsi-queue-depth.rules
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{device/vendor}=="VENDOR*", ATTR{device/queue_depth}="128"

Recommended values:

SSD/Flash storage: 128-256
HDD storage: 32-64

Session Parameters

Optimize iSCSI session settings:

# Increase max receive data segment length
iscsiadm -m node -T <target_iqn> -p <portal_ip> \
    -o update -n node.conn[0].iscsi.MaxRecvDataSegmentLength -v 262144

# Increase first burst length
iscsiadm -m node -T <target_iqn> -p <portal_ip> \
    -o update -n node.session.iscsi.FirstBurstLength -v 262144

# Increase max burst length
iscsiadm -m node -T <target_iqn> -p <portal_ip> \
    -o update -n node.session.iscsi.MaxBurstLength -v 1048576

# Enable immediate data
iscsiadm -m node -T <target_iqn> -p <portal_ip> \
    -o update -n node.conn[0].iscsi.ImmediateData -v Yes

# Increase number of outstanding R2Ts
iscsiadm -m node -T <target_iqn> -p <portal_ip> \
    -o update -n node.session.iscsi.MaxOutstandingR2T -v 1

I/O Scheduler Optimization

For SSD/Flash storage:

# Use 'none' or 'noop' scheduler for flash storage
echo none > /sys/block/sda/queue/scheduler

# Make persistent via udev rule
# /etc/udev/rules.d/99-iscsi-scheduler.rules
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="none"

For HDD storage:

# Use 'mq-deadline' for HDD
echo mq-deadline > /sys/block/sda/queue/scheduler

CPU and IRQ Optimization

IRQ Affinity

Distribute interrupts across CPUs:

# Install irqbalance
# RHEL/Rocky/AlmaLinux:
dnf install -y irqbalance

# Debian/Ubuntu:
apt install -y irqbalance

# Enable and start
systemctl enable --now irqbalance

# Or manually set IRQ affinity
# Find IRQ for network interface
grep eth0 /proc/interrupts

# Set IRQ to specific CPU (example: IRQ 45 to CPU 2)
echo 4 > /proc/irq/45/smp_affinity  # 4 = binary 0100 = CPU 2

CPU Isolation (Advanced)

Dedicate CPUs to storage I/O:

# Edit kernel boot parameters
# /etc/default/grub
GRUB_CMDLINE_LINUX="isolcpus=2,3,10,11"

# Update grub
# RHEL/Rocky/AlmaLinux:
grub2-mkconfig -o /boot/grub2/grub.cfg

# Debian/Ubuntu:
update-grub

# Reboot required
reboot

Warning:

Note: CPU isolation (isolcpus) is a general system optimization for I/O-intensive workloads. It does not directly affect iSCSI protocol behavior. Measure baseline performance before and after changes to validate impact in your environment.

Read-Ahead Tuning

Optimize read-ahead for workload:

# Check current read-ahead
blockdev --getra /dev/sda

# For random I/O workloads (databases):
blockdev --setra 256 /dev/sda  # 128 KB

# For sequential I/O workloads (file servers):
blockdev --setra 8192 /dev/sda  # 4 MB

# Make persistent via udev rule (adjust vendor to match your storage)
# /etc/udev/rules.d/99-iscsi-readahead.rules
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{device/vendor}=="VENDOR*", ATTR{bdi/read_ahead_kb}="128"

Monitoring Performance

Key metrics to monitor:

# I/O statistics
iostat -x 1

# Network statistics
sar -n DEV 1

# iSCSI session statistics
iscsiadm -m session -P 3 | grep -A 20 "iSCSI Session State"

# Multipath I/O statistics
dmsetup status

# System-wide I/O
vmstat 1

Performance indicators:

Low latency: < 1ms for flash storage
High throughput: Near line speed (10Gbps = ~1.2 GB/s)
Low CPU wait: iowait < 5%
Balanced paths: Even I/O distribution across all paths

System-Level Optimizations

CPU Isolation for I/O Threads

Isolate CPUs for storage processing:

# In /etc/default/grub, add to GRUB_CMDLINE_LINUX:
GRUB_CMDLINE_LINUX="isolcpus=2,3 nohz_full=2,3 rcu_nocbs=2,3"

Warning:

Note: CPU isolation (isolcpus) is a general system optimization for I/O-intensive workloads. It does not directly affect iSCSI protocol behavior. Measure baseline performance before and after changes to validate impact in your environment.

Best Practice:

Use isolcpus only if CPU contention is observed (high %sys during I/O)
Monitor with mpstat -P ALL 1 to identify busy cores

Memory Configuration

Hugepages for large I/O buffers:

# /etc/sysctl.d/99-storage-performance.conf
vm.nr_hugepages = 1024

# For transparent hugepages (alternative)
echo always > /sys/kernel/mm/transparent_hugepage/enabled

Disable NUMA balancing for predictable latency:

echo 0 > /proc/sys/kernel/numa_balancing

Network Optimizations

TCP Buffer Tuning

Increase TCP buffer sizes for high throughput:

# /etc/sysctl.d/99-storage-tcp.conf

# TCP buffer sizes (min, default, max)
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

# Connection queue sizes
net.core.netdev_max_backlog = 30000
net.core.somaxconn = 4096

Network Interface Tuning

Increase ring buffer size:

# Check current settings
ethtool -g ens1f0

# Increase to maximum
ethtool -G ens1f0 rx 4096 tx 4096

Enable receive-side scaling (RSS):

# Check RSS configuration
ethtool -l ens1f0

# Set number of channels
ethtool -L ens1f0 combined 8

Interrupt coalescing for throughput:

# Reduce interrupts for high throughput
ethtool -C ens1f0 rx-usecs 100 tx-usecs 100

# For low latency (more interrupts)
ethtool -C ens1f0 rx-usecs 0 tx-usecs 0

Storage Layer Optimizations

I/O Scheduler Tuning

For SCSI/iSCSI Devices

Recommended: mq-deadline or none

# Check current scheduler
cat /sys/block/sda/queue/scheduler

# Set to mq-deadline (good for mixed workloads)
echo mq-deadline > /sys/block/sda/queue/scheduler

# Or set to none (lowest latency)
echo none > /sys/block/sda/queue/scheduler

Why: Modern storage arrays handle I/O scheduling internally; kernel scheduler adds latency.

Make persistent:

# Create udev rule: /etc/udev/rules.d/60-iosched.rules
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/scheduler}="mq-deadline"

Queue Depth Tuning

# Check current queue depth
cat /sys/block/sda/queue/nr_requests

# Increase to 256 (default is often 64 or 128)
echo 256 > /sys/block/sda/queue/nr_requests

Read-Ahead Configuration

# Check current read-ahead
blockdev --getra /dev/sda

# Set read-ahead (in 512-byte sectors)
# 16384 = 8MB read-ahead (good for sequential workloads)
blockdev --setra 16384 /dev/sda

iSCSI-Specific Tuning

iSCSI Session Parameters

Optimize session settings in /etc/iscsi/iscsid.conf:

# Faster failover
node.session.timeo.replacement_timeout = 20

# Larger MaxRecvDataSegmentLength (up to 16MB)
node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144

# Enable immediate data (reduces latency)
node.session.iscsi.ImmediateData = Yes
node.session.iscsi.FirstBurstLength = 262144
node.session.iscsi.MaxBurstLength = 16776192

Multipath Performance

Configure for throughput in /etc/multipath.conf:

defaults {
    path_grouping_policy    multibus     # Use all paths simultaneously
    path_selector           "round-robin 0"
    failback                immediate
    no_path_retry           queue
}

Monitoring Performance

Real-time I/O monitoring:

# Per-device statistics
iostat -xz 1

# Watch key metrics:
# - await: Average wait time (ms) - should be <20ms for iSCSI
# - %util: Utilization - sustained >80% may indicate bottleneck
# - r/s, w/s: IOPS
# - rkB/s, wkB/s: Throughput

iSCSI session statistics:

# Check session parameters
iscsiadm -m session -P 3 | grep -E "Header|Data|Burst"

RHEL-Specific Tuning

Use tuned for automated tuning:

# Install tuned
sudo dnf install -y tuned tuned-utils

# Enable tuned
sudo systemctl enable --now tuned

# List available profiles
sudo tuned-adm list

# Apply throughput-performance profile
sudo tuned-adm profile throughput-performance

# Or network-latency for low-latency workloads
sudo tuned-adm profile network-latency

# Verify active profile
sudo tuned-adm active

Create custom tuned profile for iSCSI:

# Create custom profile directory
sudo mkdir -p /etc/tuned/iscsi-storage

# Create tuned.conf
sudo tee /etc/tuned/iscsi-storage/tuned.conf > /dev/null <<'EOF'
[main]
summary=Optimized for iSCSI storage workloads
include=throughput-performance

[sysctl]
# Network tuning
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864
net.ipv4.tcp_low_latency = 1

# iSCSI tuning
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1

[disk]
# I/O scheduler for iSCSI devices
devices_udev_regex=^sd[a-z]+$
elevator=none

[script]
script=${i:PROFILE_DIR}/script.sh
EOF

# Create script
sudo tee /etc/tuned/iscsi-storage/script.sh > /dev/null <<'EOF'
#!/bin/bash
. /usr/lib/tuned/functions

start() {
    # Set queue depth for iSCSI devices
    for dev in /sys/block/sd*/device/queue_depth; do
        if [ -f "$dev" ]; then
            echo 128 > "$dev"
        fi
    done
    return 0
}

stop() {
    return 0
}

process $@
EOF

# Make script executable
sudo chmod +x /etc/tuned/iscsi-storage/script.sh

# Apply custom profile
sudo tuned-adm profile iscsi-storage

# Verify
sudo tuned-adm active

Warning:

The values in this custom tuned profile are starting points for testing. Actual optimal values depend on:

Driver/firmware limitations: Check NIC and storage driver documentation for supported buffer sizes and queue depths.
Hardware capabilities: Use ethtool -g <interface> to verify ring buffer limits.
Workload characteristics: Sequential vs. random I/O, block sizes, concurrency

Always validate with performance monitoring (iostat -x 1, sar -n DEV 1, perf, vendor telemetry) before deploying to production. Measure baseline performance first, then test changes incrementally.

Optimize kernel for iSCSI:

# Create sysctl configuration
sudo tee /etc/sysctl.d/99-iscsi-rhel.conf > /dev/null <<'EOF'
# Network performance
net.core.netdev_max_backlog = 5000
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864

# Connection tracking
net.netfilter.nf_conntrack_max = 1048576

# Low latency
net.ipv4.tcp_low_latency = 1

# VM tuning for storage
vm.dirty_ratio = 10
vm.dirty_background_ratio = 5
vm.swappiness = 10

# ARP settings for same-subnet multipath (CRITICAL)
# Prevents ARP responses on wrong interface when multiple NICs share same subnet
# See: Network Concepts documentation for detailed explanation
net.ipv4.conf.all.arp_ignore = 2
net.ipv4.conf.default.arp_ignore = 2
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_announce = 2
# Interface-specific (adjust interface names as needed)
net.ipv4.conf.ens1f0.arp_ignore = 2
net.ipv4.conf.ens1f1.arp_ignore = 2
net.ipv4.conf.ens1f0.arp_announce = 2
net.ipv4.conf.ens1f1.arp_announce = 2
EOF

# Apply settings
sudo sysctl -p /etc/sysctl.d/99-iscsi-rhel.conf