In the case of an actual failure being the reason for a recovery operation within Site Recovery Manager (as opposed to disaster avoidance or a migration which implies things are currently, at least, fully functional), there may be steps that cannot be executed that are in-plan for a recovery. Any and all steps that involve making changes at the source site can fail and SRM can still recover the workload.
If errors are run into during a recovery, it is generally recommended to try to fix those errors and then re-run the recovery plan in planned migration mode. If that is not possible (due to an extended outage or irrecoverable loss of equipment), the recovery plan can be run (or re-run as the case may be) in Disaster Recovery mode.
If the source SRM server is offline, SRM will give no choice but to run it in that mode. Running the recovery workflow in disaster recovery mode does not change how the SRA behaves-the operations on the target side do not change whether or not the source site is down.
The SRA operations that might fail are:
- Synchronizations
- Demoting the source pod
- Tagging the source volumes
- Disconnecting the source volumes from hosts
Run the recovery in disaster recovery mode to get the workload started in the target site. Once the source site is back online, re-run the workflow in planned migration mode. Though it might not be fully recoverable, if not, manual reconfiguration is necessary.
If it is, a few scenarios might be encountered:
Volumes were never disconnected
If the VMs were never powered off, these will be brought down, the datastores unmounted, and the volumes disconnected.
Source pod is still promoted
In the case that the source pod is still promoted, the SRA will see that and check if the target pod is already promoted AND the volumes in the target pod have the puresra-failover tags. If so, it knows that the SRA previously promoted the target pod and that it is running production now. Therefore it will demote the source pod WITHOUT the quiesce option. This means that any changes build up since on the source side are discarded (or rather not sent to the target side). Once the demotion is complete, the target side becomes the source and it will begin replicating the changes back to the original source side (which is now the target). This demotion reverses the replication direction.
Remember that the demotion will create an undo pod of the state of the original source pod at the point of demotion. So any changes, volumes, configurations will be protected until the undo pod is eradicated manually or eradicated with time as dictated by the eradication timer (defaults to 24 hours).
Note that the source volumes are not tagged with the puresra-demoted tag unlike with a planned migration. Why? The demotion without quiesce will cause the tags to never be sent to the target. A valid question is: why not quiesce? Well because the target is promoted already. Source volume changes, like tags, only get reflected on target volumes when:
- A demoted target pod periodically refreshes itself with the latest replicated data
- A demoted target pod gets promoted
Since the target pod is already promoted, it will never pick up new tags. Creating them is pointless, and replicating them is pointless. But they are needed! This is how we determine that it is safe to run a reprotect. So if we detect upon the recovery side procedures that there is a source pod that has been demoted, and we see a target pod with only puresra-failover tags, not puresra-demoted tags, we then tag the target volumes directly on the target side with puresra-demoted and the original source pod UUID. This allows the tags to be in the exact state we need to run a reprotect. Source is demoted. Target is promoted. Target volumes have the puresra-failover tag with the value of the target pod UUID and the puresra-demoted tag with the value of the source pod UUID.
Source pod is demoted
If the SRA finds the source pod already demoted, it will check to see if target pod is promoted and has the puresra-failover tags. If so, it will know that the SRA has failed over the source side and will move on. If the volumes in the target pod do not have the puresra-demoted tags (meaning they were never applied on the source and therefore the target volumes did not inherit them upon promotion), they will be applied to the target volumes.