Anthos Baremetal - How to run PX on the nodes which are marked for maintenance

KB0022599

Pure Product

Portworx

Environment

PX Operator Version Higher than 25.3.0
Anthos Baremetal Nodes

Symptoms

When a bare-metal node is placed into maintenance mode in Anthos, an additional taint (baremetal.cluster.gke.io/maintenance:NoSchedule) is automatically applied to the node, as documented below.

https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/how-to/maintenance-mode#put_a_node_into_maintenance_mode

If, for any reason, maintenance does not proceed sequentially or there is a delay in removing a node from maintenance mode, Portworx will fail to start on that node due to the additional taint. The solution described here involves configuring the Portworx cluster and API pods to tolerate the specified taint.

Resolution

Creating the following ComponentK8sConfig custom resource will add a toleration for the Anthos taint.

apiVersion: core.libopenstorage.org/v1
kind: ComponentK8sConfig
metadata:
  name: px-placement-config
  namespace: portworx
spec:
  components:
    - componentNames:
       - Portworx API
       - Storage
      workloadConfigs:
        - workloadNames:
          - storage
          - portworx-api
          placement:
            tolerations:
              - key: baremetal.cluster.gke.io/maintenance
                operator: Exists
                effect: NoSchedule

Additional Information

ComponentK8sConfig creation is a onetime configuration and there will be a rolling restart of portworx cluster and api pods.
This configuration is only applicable for the customers who want to run PX while keeping anthos node under maintenance