Operation Failed: Failed to sync data on replica devices. SRA command 'syncOnce' failed. The target pod <pod name> of the operation is already promoted. Make sure the target pod is demoted before running a planned migration or disaster recovery.
During the process of recovery, the source pod gets demoted and the target pod gets promoted. If the target pod is already promoted, the SRA needs to know if the SRA itself promoted the pod already (a recovery occurred, it got past the promotion step but something then failed and the recovery was restarted).
If a pod was promoted by the SRA, the volumes in it will have the puresra-failover tag:
If they do not and the pod is promoted the SRA will decide it was not promoted by itself and throw the above error during the synchronization step.
To recovery from this, go to the target pod and demote it and retry the recovery:
There is one edge case that is worth noting:
- If you have safe mode enabled (which means nothing can be manually eradicated)
- The target pod is already promoted for testing reasons etc
- There is an undo pod still in existence for the target pod
The target pod will not be able to be demoted manually until the undo pod gets eradicated by the timer. If too much time is left on the timer that would cause too long in a delay in disaster recovery, call Everpure Support.
Lastly, Everpure is working on improving the handling of scenarios such as this where safe mode is enabled,