Monitoring and Maintenance

Regular Health Checks

Connection Monitoring

Connection Status

iSCSI sessions:

iscsiadm -m session
iscsiadm -m session -P 3 | grep -E "State|iSCSI Connection State"

I/O Statistics

# Real-time I/O stats
iostat -xz 1

# Per-device statistics
iostat -xz -d sda sdb 1

# Key metrics to watch:
# - await: Average I/O wait time (should be <20ms for iSCSI)
# - %util: Device utilization (sustained >80% may indicate bottleneck)

Device Health

Multipath Status

# Show multipath topology
multipath -ll

# Show multipath status
multipathd show status

# Check individual path states
multipathd show paths

Storage Array Health

Monitor iSCSI sessions:

# Check session parameters
iscsiadm -m session -P 3

# Key items to check:
# - Connection state: should be "LOGGED_IN"
# - Internal iscsid Session State: should be "NO CHANGE"

Log Monitoring

Error Detection

Check system logs for storage errors:

# iSCSI errors
journalctl -u iscsid -p err --since "7 days ago"

# Multipath errors
journalctl -u multipathd -p err --since "7 days ago"

# Kernel errors related to storage
dmesg -T | grep -i "iscsi\|dm-\|multipath" | grep -i "error\|fail"

Automated Monitoring Script

#!/bin/bash
# storage-monitor.sh - Quick storage health check

{
    echo "=== Storage Health Check $(date) ==="
    echo "=== iSCSI Sessions ==="
    iscsiadm -m session
    echo "=== Multipath Devices ==="
    multipath -ll
    echo "=== I/O Statistics ==="
    iostat -xz 1 1
    echo "=== Network Statistics ==="
    ip -s link show
} > /var/log/storage-snapshot-$(date +%Y%m%d).log

Maintenance Procedures

Graceful Storage Disconnection

Before maintenance:

# 1. Stop applications using storage
systemctl stop your-application

# 2. Unmount filesystems
umount /dev/mapper/mpatha*

# 3. Flush multipath devices
multipath -F

# 4. Logout iSCSI sessions
iscsiadm -m node -U all

# 5. Stop iSCSI service
systemctl stop iscsid

Configuration Backup

Regular backup of storage configuration:

# Multipath configuration
cp -a /etc/multipath.conf /backup/multipath.conf-$(date +%Y%m%d)

# iSCSI configuration
cp -a /etc/iscsi/ /backup/iscsi-$(date +%Y%m%d)

Automated backup script:

#!/bin/bash
# storage-config-backup.sh

BACKUP_DIR="/backup/storage-config"
DATE=$(date +%Y%m%d)

mkdir -p $BACKUP_DIR/$DATE

cp -a /etc/network/interfaces $BACKUP_DIR/$DATE/
cp -a /etc/multipath.conf $BACKUP_DIR/$DATE/
cp -a /etc/iscsi/ $BACKUP_DIR/$DATE/

# Create tarball
tar -czf $BACKUP_DIR/storage-config-$DATE.tar.gz $BACKUP_DIR/$DATE/

# Keep last 30 days
find $BACKUP_DIR -name "*.tar.gz" -mtime +30 -delete

echo "Backup completed: $BACKUP_DIR/storage-config-$DATE.tar.gz"

Firmware/Driver Updates

Planning checklist: 1. Schedule maintenance window 2. Backup current configuration 3. Migrate VMs if applicable 4. Update drivers/firmware 5. Verify connectivity 6. Test failover

Verification after updates:

# 1. Check iSCSI sessions
iscsiadm -m session

# 2. Verify multipath
multipath -ll

# 3. Check I/O performance
fio --name=test --rw=randread --size=100M --runtime=10

# 4. Test failover (disconnect one path)
# Verify I/O continues without interruption

RHEL-Specific Monitoring

Monitor iSCSI services:

# View iSCSI logs
sudo journalctl -u iscsid -f

# View multipath logs
sudo journalctl -u multipathd -f

# View all storage-related logs
sudo journalctl -u iscsid -u multipathd -u lvm2-* --since "1 hour ago"

Install and configure sysstat:

# Install
sudo dnf install -y sysstat

# Enable data collection
sudo systemctl enable --now sysstat

# View I/O statistics
sar -d 1 10

# View network statistics
sar -n DEV 1 10

# Generate daily report
sar -A

Create monitoring script:

sudo tee /usr/local/bin/iscsi-health-check.sh > /dev/null <<'EOF'
#!/bin/bash
# iSCSI Health Check Script for RHEL

echo "=== iSCSI Health Check ==="
echo "Date: $(date)"
echo

echo "--- iSCSI Sessions ---"
iscsiadm -m session

echo
echo "--- Multipath Status ---"
multipath -ll | head -50

echo
echo "--- Failed Paths ---"
multipath -ll | grep -i "failed\|faulty" || echo "No failed paths"

echo
echo "--- Disk I/O ---"
iostat -x 1 2 | tail -20

echo
echo "--- Network Interfaces ---"
ip -s link show | grep -A 3 "ens1f"

echo
echo "=== End Health Check ==="
EOF

sudo chmod +x /usr/local/bin/iscsi-health-check.sh

# Run health check
sudo /usr/local/bin/iscsi-health-check.sh

Schedule with cron:

# Add to crontab
echo "0 */6 * * * /usr/local/bin/iscsi-health-check.sh >> /var/log/iscsi-health.log 2>&1" | sudo crontab -