How to migrate Portworx CSI 3.X to Portworx CSI 25.X?

KB0021321

Pure Product

Portworx

Question

Answer

INMPORTANT: Best practices for migration

Always wait until the previous node re-joins the cluster and KVDB shows 3 healthy members before continuing to the next node.
Never start the migration process on a second node if the first is not fully healthy and re-integrated.
Ensure no unexpected restarts (reboots/crashes) occur during migration.
Ensure that there are no portworx volumes (volumes that were created on the local PX storage pools). If such volumes exist, they need to be backed up and deleted before the migration. Only Direct Access Volumes can be migrated to the new Portworx CSI.

The below procedure describes migration steps from Portworx CSI for FA/FB to new Portworx CSI 25.1.0 or later.

Migration procedure

1. Update StorageCluster spec

apiVersion: core.libopenstorage.org/v1

kind: StorageCluster

metadata:

annotations:

portworx.io/misc-args: '--oem esse' #change to '--oem px-csi'

spec:

image: portworx/oci-monitor:3.2.1.1 #change/update image version portworx/oci-monitor:25.X.X

stork:

enabled: true #disable/change to false

autopilot:

enabled: true -> #disable/change to false

updateStrategy:

type: RollingUpdate #change to OnDelete

version: 3.2.1.1 #change to 25.X.X

2. Drain node to relocate workloads

kubectl cordon <NODE_NAME>

3. Put the node in maintenance mode

pxctl service maintenance -e

4. SSH to a different, healthy/online node in PX Cluster, then remove the node that was previously targeted with the decommission in steps 2-3 (node should be in maintenance)

pxctl status # get the node id of the node that will be deleted

pxctl cluster delete <PX_NODE_ID> --force (--force will be required if the targeted node was running KVDB).

5. Once the node is deleted from the PX Cluster , label the node to remove portworx service from the node

kubectl label node <NODE_NAME> px/enabled=remove

6. SSH back to the node that is being deleted. Confirm that Portworx was removed from the node by making sure /etc/systemd/system/portworx.service no longer exists. The uninstall process may take up to 5 minutes after applying the above label.

ls -l /etc/systemd/system/portworx.service

ls: cannot access '/etc/systemd/system/portworx.service': No such file or directory

systemctl status portworx

Unit portworx.service could not be found.

If the service did not get removed, please contact Portworx support.

7. Wipe PX configuration data on this node:

pxctl service node-wipe --all

8. Confirm that the portworx kvdb and storage/data drives have been formatted and/or removed

lsblk -f | egrep “pxpool|kvdbvol”

The above command should return nothing

9. (For FlashArray only) Login to FlashArray user interface and disconnect and destroy the above volumes. The drives will be connected to this host/node and contain pxclouddrive-xxxxxxxxxxx as part of the name

10. Confirm that /etc/pwx/config.json file no longer exists.

ls -l /etc/pwx/config.json

ls: cannot access '/etc/pwx/config.json': No such file or directory

If it exists, remove it manually with the following command:

rm -rf /etc/pwx

11. Uncordon the node that was decommissioned/deleted from the PX Cluster, (re-enable scheduling on the node)

kubectl uncordon <NODE_NAME>

12. Make sure the px-cluster pod is no longer on the node that was decommissioned, this is the pod that runs the container named "portworx". If the pod is still running, delete it.

kubectl get pods -A -o wide -l name=portworx | grep <NODE_NAME>

13. Label the decommissioned node to re-add it to the PX Cluster

kubectl label nodes <NODE_NAME> px/service=restart --overwrite

kubectl label nodes <NODE_NAME> px/enabled-

14. Wait approximately 5 minutes and confirm that the node has been re-added to the PX Cluster and that KVDB is healthy (3 members are online)

pxctl status

pxctl service kvdb members

15. Proceed to the next node by following the steps 2 -15

16. Once all of the nodes have been migrated

Remove px-storage pod disruption budget

kubectl delete pdb px-storage

Clean up storagecluster spec (remove ENV variables that are no longer needed)
Restore update strategy back to RollingUpdate

updateStrategy:

type: OnDelete #change to RollingUpdate