Performance Tuning

CPU Isolation for Storage I/O

Dedicate CPU cores to storage I/O processing to reduce latency and improve consistency.

Configuration:

# Edit /etc/default/grub
GRUB_CMDLINE_LINUX="isolcpus=2,3,10,11"

# Update grub
update-grub
reboot

Warning:

CPU isolation (isolcpus) is a general system optimization for I/O-intensive workloads. It does not directly affect storage protocol behavior (iSCSI/NVMe-TCP). Measure baseline performance before and after changes to validate impact in your environment.

Best Practice:

Isolate 2-4 cores per storage NIC
Use cores on the same NUMA node as the NIC
Leave enough cores for VM workloads

IRQ Affinity

Pin NIC interrupts to specific CPUs to reduce cache thrashing and improve performance.

Find NIC IRQs:

# Find IRQ numbers for your storage NIC
grep <interface_name> /proc/interrupts

Set IRQ affinity:

# Pin IRQ to specific CPU (CPU 2 = bitmask 0x4)
echo 4 > /proc/irq/<IRQ_NUMBER>/smp_affinity

Automated script:

#!/bin/bash
# set-irq-affinity.sh
INTERFACE="ens1f0"
CPU_START=2

IRQ_LIST=$(grep $INTERFACE /proc/interrupts | awk '{print $1}' | sed 's/://')
CPU=$CPU_START

for IRQ in $IRQ_LIST; do
    MASK=$(printf "%x" $((1 << $CPU)))
    echo $MASK > /proc/irq/$IRQ/smp_affinity
    echo "IRQ $IRQ -> CPU $CPU (mask: $MASK)"
    CPU=$((CPU + 1))
done

Memory and I/O Tuning

Kernel Parameters

Warning:

The values below are suggested starting points for testing. Actual optimal values depend on your specific hardware, workload, and environment. Always validate with performance telemetry (iostat, sar, perf, etc.) before deploying to production.

Edit /etc/sysctl.conf or create /etc/sysctl.d/99-storage-tuning.conf:

# Network buffer sizes
# Note: Maximum values depend on available system RAM
# These values (128MB max) assume 64GB+ RAM; reduce proportionally for smaller systems
# Example: For 16GB RAM, consider rmem_max/wmem_max = 33554432 (32MB)
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216

# TCP buffer sizes (min, default, max)
# Note: Max values limited by rmem_max/wmem_max above
# Adjust based on workload: higher for throughput, lower for latency-sensitive
net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864

# Connection queue sizes
# Note: Higher values require more kernel memory; scale based on workload
net.core.netdev_max_backlog = 30000
net.core.somaxconn = 4096

# TCP optimizations
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 0    # Trade-off: disables RTT measurement
net.ipv4.tcp_sack = 1

# ARP cache sizes
# Note: Scale based on number of storage targets and network size
# These values suit environments with 50-200 storage paths
net.ipv4.neigh.default.gc_thresh1 = 4096
net.ipv4.neigh.default.gc_thresh2 = 8192
net.ipv4.neigh.default.gc_thresh3 = 16384

# VM dirty page settings
# Note: These affect write caching behavior and depend on RAM and workload
# - dirty_ratio: % of RAM that can hold dirty pages before blocking writes
# - dirty_background_ratio: % of RAM before background writeback starts
# Lower values = more consistent latency; higher = better burst throughput
# For 64GB+ RAM with mixed workloads, start with these values
# For latency-sensitive workloads, consider dirty_ratio=5, dirty_background_ratio=2
vm.dirty_ratio = 10
vm.dirty_background_ratio = 5
vm.swappiness = 10

Apply changes:

sysctl -p /etc/sysctl.d/99-storage-tuning.conf

Why these settings:

Buffer sizes: Accommodate high-bandwidth storage traffic (scale with available RAM)
Backlog: Handle burst traffic without drops
Window scaling: Enable large TCP windows for high-bandwidthdelay networks
Timestamps: Reduce per-packet overhead (trade-off: no RTT measurement)
ARP cache: Prevent ARP table overflow in large storage networks
VM dirty ratios: Prevent large amounts of dirty data in cache, improving write consistency

Tuning considerations:

RAM sizing: Buffer sizes should scale with system memory. The values above assume 64GB+ RAM.
NIC/driver constraints: Some NICs have hardware buffer limits; check ethtool -g <interface> for maximums.
Workload profiling: Use iostat -x 1, sar -n DEV 1, and ss -tnp to measure actual utilization before and after changes.
Iterative testing: Change one parameter at a time and measure impact with consistent benchmarks.

I/O Scheduler Tuning

Recommended: none for NVMe devices (no scheduler): NVMe devices have their own internal scheduling; kernel scheduler adds unnecessary overhead.

# Check current scheduler
cat /sys/block/nvme0n1/queue/scheduler

# Set to none
echo none > /sys/block/nvme0n1/queue/scheduler

Make persistent:

# Create udev rule: /etc/udev/rules.d/60-iosched.rules
ACTION=="add|change", KERNEL=="nvme[0-9]n[0-9]", ATTR{queue/scheduler}="none"

Increase queue depth for better parallelism: Higher queue depth allows more concurrent I/O operations, improving throughput.

# Check current queue depth
cat /sys/block/nvme0n1/queue/nr_requests

# Increase to 1024 (default is often 256)
echo 1024 > /sys/block/nvme0n1/queue/nr_requests

NIC Tuning

Offload Features

Enable hardware offloads to reduce CPU usage: Reduces CPU overhead for packet processing, freeing CPU for other tasks.

# Enable all offloads
ethtool -K <interface> tso on gso on gro on lro on

# Verify
ethtool -k <interface>

Offload types:

TSO (TCP Segmentation Offload): NIC handles TCP segmentation
GSO (Generic Segmentation Offload): Kernel-level segmentation
GRO (Generic Receive Offload): Aggregate received packets
LRO (Large Receive Offload): Hardware packet aggregation

Ring Buffers

Increase ring buffer sizes: Larger buffers reduce packet drops during traffic bursts.

# Check current settings
ethtool -g <interface>

# Set to maximum
ethtool -G <interface> rx 4096 tx 4096

Monitoring Performance

Real-time I/O monitoring:

# Monitor disk I/O
iostat -x 1

# Monitor network I/O
iftop -i <interface>

# Monitor multipath I/O distribution
watch -n 1 'dmsetup status'

Latency monitoring:

# Monitor NVMe latency
nvme smart-log /dev/nvme0n1 | grep -i latency

# Monitor I/O latency with ioping
ioping -c 10 /dev/mapper/<device>

Throughput testing:

# Test sequential read
fio --name=seqread --rw=read --bs=1M --size=10G --filename=/dev/mapper/<device>

# Test sequential write
fio --name=seqwrite --rw=write --bs=1M --size=10G --filename=/dev/mapper/<device>

# Test random read IOPS
fio --name=randread --rw=randread --bs=4k --size=10G --filename=/dev/mapper/<device>

Oracle-Specific Tuning Profiles

Oracle Linux includes tuned profiles optimized for different workloads:

# Install tuned
sudo dnf install -y tuned

# List available profiles
tuned-adm list

# Recommended profiles for storage:
# - throughput-performance (general high-throughput)
# - latency-performance (low-latency workloads)
# - network-latency (network-intensive)

# Set profile
sudo tuned-adm profile throughput-performance

# Verify
tuned-adm active

Custom Tuned Profile for NVMe-TCP

Create a custom profile combining best practices:

# Create custom profile directory
sudo mkdir -p /etc/tuned/nvme-tcp-storage

# Create tuned.conf
sudo tee /etc/tuned/nvme-tcp-storage/tuned.conf > /dev/null <<'EOF'
[main]
summary=Optimized for NVMe-TCP storage workloads on UEK
include=throughput-performance

[cpu]
governor=performance
energy_perf_bias=performance
min_perf_pct=100

[sysctl]
# Network buffers
net.core.rmem_max=134217728
net.core.wmem_max=134217728
net.core.rmem_default=16777216
net.core.wmem_default=16777216
net.ipv4.tcp_rmem=4096 87380 67108864
net.ipv4.tcp_wmem=4096 65536 67108864

# Network stack
net.core.netdev_max_backlog=30000
net.core.somaxconn=4096
net.ipv4.tcp_window_scaling=1
net.ipv4.tcp_timestamps=0
net.ipv4.tcp_sack=1

# VM tuning
vm.dirty_ratio=10
vm.dirty_background_ratio=5
vm.swappiness=10

# ARP cache
net.ipv4.neigh.default.gc_thresh1=4096
net.ipv4.neigh.default.gc_thresh2=8192
net.ipv4.neigh.default.gc_thresh3=16384

# ARP settings for same-subnet multipath (CRITICAL)
# Prevents ARP responses on wrong interface when multiple NICs share same subnet
# See: Network Concepts documentation for detailed explanation
net.ipv4.conf.all.arp_ignore=2
net.ipv4.conf.default.arp_ignore=2
net.ipv4.conf.all.arp_announce=2
net.ipv4.conf.default.arp_announce=2
# Interface-specific (adjust interface names as needed)
net.ipv4.conf.ens1f0.arp_ignore=2
net.ipv4.conf.ens1f1.arp_ignore=2
net.ipv4.conf.ens1f0.arp_announce=2
net.ipv4.conf.ens1f1.arp_announce=2

[disk]
# NVMe devices use 'none' scheduler
elevator=none

[script]
script=${i_path}/script.sh
EOF

# Create script for additional tuning
sudo tee /etc/tuned/nvme-tcp-storage/script.sh > /dev/null <<'EOF'
#!/bin/bash

. /usr/lib/tuned/functions

start() {
    # Set NVMe IO policy to numa
    for ctrl in /sys/class/nvme-subsystem/nvme-subsys*/iopolicy; do
        [ -f "$ctrl" ] && echo "numa" > "$ctrl"
    done

    # Tune network interfaces
    for iface in ens1f0 ens1f1; do
        if [ -d "/sys/class/net/$iface" ]; then
            ethtool -G $iface rx 4096 tx 4096 2>/dev/null || true
            ethtool -C $iface rx-usecs 50 tx-usecs 50 2>/dev/null || true
            ethtool -K $iface tso on gso on gro on 2>/dev/null || true
        fi
    done

    return 0
}

stop() {
    return 0
}

process $@
EOF

sudo chmod +x /etc/tuned/nvme-tcp-storage/script.sh

# Activate custom profile
sudo tuned-adm profile nvme-tcp-storage

# Verify
tuned-adm active
tuned-adm verify

Important:

The values in this custom tuned profile are starting points for testing. Actual optimal values depend on:

Driver/firmware limitations: Check NIC and UEK driver documentation for supported buffer sizes and queue depths
Hardware capabilities: Use ethtool -g <interface> to verify ring buffer limits
Workload characteristics: Sequential vs. random I/O, block sizes, concurrency

Always validate with performance monitoring (iostat -x 1, sar -n DEV 1, perf, Oracle OSWatcher) before deploying to production. Measure baseline performance first, then test changes incrementally.

IRQ Affinity for UEK

UEK has better IRQ handling, but you can optimize further:

# Install irqbalance
sudo dnf install -y irqbalance

# Configure irqbalance
sudo tee /etc/sysconfig/irqbalance > /dev/null <<'EOF'
# Ban CPU 0 from handling IRQs (reserve for system)
IRQBALANCE_BANNED_CPUS=00000001

# Enable for NUMA systems
IRQBALANCE_ARGS="--policyscript=/usr/local/bin/irq-policy.sh"
EOF

# Create policy script for storage NICs
sudo tee /usr/local/bin/irq-policy.sh > /dev/null <<'EOF'
#!/bin/bash
# Pin storage NIC IRQs to specific CPUs
case "$1" in
    ens1f0|ens1f1)
        echo "storage"
        ;;
    *)
        echo "default"
        ;;
esac
EOF

sudo chmod +x /usr/local/bin/irq-policy.sh

# Enable and start
sudo systemctl enable --now irqbalance