sharedv4 volumes fail to mount and clone on OCP Bare Metal (OCP Virtualization)

PX 3.3.x in OCP 4.17 (with OVN Kubernetes CNI) having the Data Interface different from Management Interface

PX-RWX Sharedv4 Service volume mount fails in OCP 4.17 (with OVN Kubernetes CNI) if the Data Interface and Management Interface are different

• PX-RWX Sharedv4 (service) volume uses NFS export over k8s clusterIP service
• OC 4.17 uses OVN Kubernetes  and so, NFS server side the incoming interface would be management interface br-ex and the outgoing interface would be data interfaceon the routing table and  kernel drops the packet due to reverse path filtering mechanism (Ref: Constraints of OVN kubernetes - https://access.redhat.com/solutions/6969174) 

PX-RWX volumes created with "Sharedv4 service" failed to get mounted with error "MountVolume.SetUp failed for volume "pvc-<xyz>" : rpc error: code = Unavailable desc = Failed to mount volume 981923402153368557: mountpoint /var/lib/kubelet/pods/<pod-id>/volumes/kubernetes.io~csi/pvc-<xyz>/mount does not exist"

• Temporary workaround would be using sharedv4 instead of sharedv4 service volumes. (Ref: https://docs.portworx.com/portworx-enterprise/concepts/shared-volumes ) like

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: portworx-rwx-rep2
provisioner: pxd.portworx.com
parameters:
    repl: "2"
    sharedv4: "true"
    sharedv4_svc_type: "" ##instead of ClusterIP
reclaimPolicy: Retain
allowVolumeExpansion: true

• Or convert the sharedv4(service) volume to "sharedv4" volume using pxctl CLI

pxctl volume update --sharedv4_service_type="" <volume>

• Because of routing table, on the NFS server side the incoming interface would be management interface br-ex and the outgoing interface would be data interface and kernel drops the packet due to reverse path filtering mechanism (Ref: https://access.redhat.com/solutions/6969174)  So tweaking kernel parameter rp_filter to off or loose mode (value 0 or 2 instead of 1 which is strict mode) would fix the issue. However this is not a secure option. 

• Category : Software
• Subcategory : PX Enterprise