Troubleshooting Runbook

FC, iSCSI, and NVMe/TCP Path Failures

Transport path failures are among the most important storage incidents to detect early. A workload may continue running even after redundancy is lost, which means the problem may not become obvious until a second fault or a maintenance action occurs.

Identifying storage path failures

Common indicators include:

• one storage-serving node behaves differently from others

• VM disk operations are slower or inconsistent

• PVC attachment or mount behavior becomes abnormal

• Portworx health appears normal at cluster level but abnormal on one node

• backend access issues affect only a subset of workloads

Isolating host, network, zoning, or array-side causes

Operators should determine whether the issue is most likely related to:

• the host node itself

• FC zoning or fabric state

• iSCSI network reachability or interface state

• NVMe/TCP network or subsystem connectivity

• FlashArray backend access behavior

• a broader cluster-side storage issue

Portworx Pool Full

A full or nearly full Portworx pool can quickly affect provisioning, expansion, restore, and VM operations. Capacity exhaustion is one of the most common storage-side incidents and one of the easiest to detect early if dashboards and routine reviews are in place.

Symptoms of capacity exhaustion

Common symptoms include:

• PVC provisioning failures

• PVC expansion failures

• snapshot or clone operations failing

• VM disk workflows becoming blocked

• storage alerts showing critically low

Immediate containment actions

Operators should first:

• confirm which pool or node is affected

• determine whether the issue is localized or systemic

• pause non-essential storage-consuming operations

• identify whether any workload can be cleaned up safely

• determine whether automated growth policies are available and appropriate

Recovery and cleanup workflow

A safe recovery path usually involves:

• Validating current pool utilization.

• Identifying inactive or removable storage consumers.

• Confirming whether expansion is possible.

• Using planned policy-driven growth where appropriate.

• Rechecking the health of impacted PVCs or VMs after recovery.

VM Migration Failures

VM migration failures can be caused by compute, network, storage, or workload-specific issues. Because live migration is often used during maintenance windows, a failed migration can quickly block planned work or extend risk exposure.

Common causes of failed live migrations

Migration failure is commonly linked to:

• source or target node health problems

• storage attachment or accessibility issues

• transport path instability

• unsupported workload conditions

• concurrency or timing conflicts with other lifecycle operations

Validation checks across compute, network, and storage

When a migration fails, review:

• source node state

• target node readiness

• VM health before migration

• Portworx and volume health

• backend transport state

• whether the failure affects only one VM or many

PVC Stuck Pending

A PVC that remains pending is one of the most visible storage symptoms seen by platform administrators and workload owners. In a policy-driven environment, this usually indicates a mismatch between the requested storage behavior and the current platform or backend state.

Common reasons PVC provisioning stalls

Typical causes include:

• incorrect or unavailable storage class

• backend capacity exhaustion

• Portworx health degradation

• missing or mismatched configuration

• backend reachability or transport issues

StorageClass, backend, and capacity checks

Operators should confirm:

• the correct storage class was selected

• the class is still valid and operational

• the backend has sufficient capacity

• Portworx is healthy

• no transport-level issue is preventing storage presentation

The goal is to determine whether the issue is policy-related, capacity-related, or transport-related.

Resolution workflow

1. Confirm the PVC request is correct.

2. Confirm the storage class is correct.

3. Confirm Portworx health.

4. Confirm backend path health.

5. Confirm capacity and placement viability.

6. Retry only after the blocking condition is understood

Multipath Degraded

Multipath degradation is a critical early warning sign in FC and iSCSI environments. It usually means workloads are still functioning, but the backend storage path is no longer fully redundant. That makes the environment more fragile during any later failure or maintenance event.

Detecting degraded multipath state

Operators should suspect degraded multipath when:

• one storage-serving node behaves differently from others

• transport path alerts appear

• storage operations are slower than normal

• a recent SAN, host, or network change has occurred

• storage health appears stable at high level but node behavior has diverged

Operational impact on VMs and containers

Degraded multipath may not cause an immediate outage, but it can:

• reduce resilience during later node or path failure

• increase sensitivity to maintenance actions

• create inconsistent behavior across similar workloads

• affect both VM disks and container PVCs

Validation and remediation workflow

Operators should:

1. Identify which nodes are affected.

2. Determine whether the issue is local or widespread.

3. Check whether workload symptoms are already visible.

4. Coordinate with storage and network teams as needed.

5. Defer non-essential maintenance until full path redundancy is restored.