It is important to remember: DO NOT manually edit or remove tags in the puresra namespace unless you understand fully what you are doing. If some rogue admin or typo'ed script deletes the tags during a test recovery it will be necessary to manually intervene.
During the test recovery, all volumes in the source pod get tagged with three pieces of information:
- The operation (like testFailover)
- Timestamp of the test
- UUID of the pod they belong to.
The tags look like this:
If someone deletes one of the tags, the cleanup process will fail. So in this case, my 3rd volume (activeDR-SRM3-RDM) was untagged:
Failed to delete snapshots of replica devices. SRA command 'testFailoverStop' failed. More than one type of devices found in the pod. Please verify that all the volumes are either untagged or tagged with same value in the "puresra" namespace.
If you log into the target array CLI, you can query for the tags of the volumes in the pod:
If you add the target pod name with an asterisk at the end, you will only see volumes in that pod.
Compare that list to the list of volumes in the SRM protection group:
If there is a volume in the protection group that is missing in the tagged volume list (and that volume is in fact in the target pod), you know the tag was for some reason deleted.
If they are the same list of volumes, then it means someone added a volume to the target pod during the test. Look at the above section for handling that.
If a volume is missing the tag, you can either add the tag or re-run the test recovery with the force option and manually demote the pod. This will erase all of the tags on the target volumes. Generally the latter is the preferred option as tagging can be error prone.
To add the tag:
Look at the tags in the pod, copy the key and the value. Identify the volume that needs to be tagged and tag it as follows:
purevol tag SRM-podB::activeDR-SRM3-RDM --key puresra-testFailover_1597691959294 --value e184134e-5a94-9f76-5c53-ae33e6df93d5 --namespace puresra
The above shows listing the tags, adding the tag, then listing the tags again. You must use the puresra namespace--otherwise the tag will not work.
Once done, re-run the test recovery cleanup.
The alternative (let's assume the same scenario as above) is to force the test recovery cleanup and then manually demote the pod.
Let the cleanup complete, then login to the target FlashArray and identify the target pod. Click on the vertical ellipsis and choose Demote. CONFIRM that this is indeed the target pod, not the source pod. The replication direction should be coming towards this pod like in the screenshot below:
Click Demote.
The tags will be removed:
Once you confirm that no issues were caused by the demote, you can eradicate the undo pod (if you need to run another test) or if not, let the eradication timer remove it in due time.