Test Recovery Cleanup Fails: Undo Pod already exists on target site

Troubleshooting for VMware Solutions
Audience
Public
Source Type
Documentation

If a target pod was manually promoted or the SRA was unable to eradicate the undo pod (for instance if safe mode was enabled) a test recovery cleanup will fail as the pod will not be able to be demoted due to the presence of an undo pod.

Failed to delete snapshots of replica devices. SRA command 'testFailoverStop' failed. Array operation failed: "PureRestException: HttpStatusCode = 'BadRequest', RestErrorCode = 'InternalError', Details = '["context":"SRM-podB","message":"Cannot demote pod because the associated undo-demote pod has not been eradicated.","context":"PATCH","message":"https://flasharray-m20-2.purecloud.c..._quiesce=False"]', InnerException = ''". Array operation failed. Please see logs at /srm/sra/log/testFailoverStop_2020-08-17-16-35-16-3662578-45c6e077-144a-4f5b-8c8d-f46c61a40ecb.log.

To verify, login to the target FlashArray and look under Destroyed and Undo Pods and see if there is a destroyed "undo" pod there for your target pod. If this exists, the test recovery cleanup will not be possible until it is fully eradicated.

There are three reasons that the undo pod might still exist:

  1. Someone manually promoted and then demoted the target pod but did not eradicate the undo pod.
  2. The credentials in the array manager within SRM are at a storage admin permission level. Storage admin level permissions cannot eradicate a pod--this requires the highest level of FlashArray permissions (array admin). So either request that an admin eradicates it, wait for timed eradication, or update the SRM array manager credentials for that ActiveDR pair to array admin.
  3. Safemode is enabled on the FlashArray and therefore no object can be manually eradicated.

    Manually eradicate it, or wait for the eradication timer to automatically remove the pod according to the schedule and then try the test recovery.

    Note if you have enabled "SafeMode" on the FlashArray, manual eradication will not be possible. This will also mean that only one SRM test recovery process can be run per pod pair per each eradication window.

    You can then either eradicate the pod and rerun the cleanup:

    Or re-run the cleanup with Force Cleanup selected:

    Then demote the pod when needed or through the automatic eradication window. Note that a subsequent test recovery or recovery will not be possible until the pod is demoted.