A recovery is very similar to a test recovery in process. There are some differences however that are worth noting. ActiveDR was specifically designed to make the process of disaster recovery easier (read: Active Disaster Recovery). The processes for a test recovery and full recovery (planned or otherwise) were thought through to find the most common areas of trouble; how can we make DR safer? How can the FlashArray make it simpler? Because of this attention to detail with the design of ActiveDR, it is extremely simple for a user to manage and for the SRA to automate.
Prior to a recovery, a source pod is in the promoted state and the target pod is in the demoted state. If the target pod is in the promoted state, it must be manually demoted prior to an attempted recovery. To initiate a recovery:
-
The appropriate host or host groups are created on the target FlashArray
To initiate a recovery, navigate within SRM to the recovery plan and click Run.
The recovery wizard will ask you to accept the risk (this takes production offline before restored on the recovery site).
Furthermore, choose either a Planned Migration or a Disaster Recovery.
There is no significant different in the process with either of these options; the main difference is how they respond to certain events. A Planned Migration is what it sounds like: a movement of the workload from site A to site B and that movement is expected, anticipated, and well planned. In this case, any failures encountered in the recovery process will halt the execution of the workflow. The failure must be resolved and the recovery must be restarted. If the Disaster Recovery option is chosen, the workflow will tolerate almost any failure on the source site up to the entire failure of the source environment; this includes but is not limited to the source network, vCenter, SRM server, storage, compute, and like the entire building that it all sits in. It will not, however, tolerate failures on the target side-so a disaster recovery operation might fail if the target side is misconfigured in some way. If failures are encountered there, the process will halt. Fix the issue and re-run the process and the recovery will re-attempt the steps and continue on if the problem has been resolved.
The first operation is a synchronization.
Next the source site will shutdown, VM will be unregistered and the datastores will be unmounted.
During the Change storage to read-only step, the SRA will:
-
Demote the source pod. The demotion will use the "quiesce" flag, meaning that the pod will not finish the demotion until all of the pods' data has been replicated to the remote FlashArray
The volumes in the source pod will each be tagged with a tag with the key of puresra-demoted and the value of the UUID of the source pod.
Right after the tags are created the pod will then be demoted.
There will be a final synchronization at that point.
When logging into to the target array and running a tag query on the volumes in the pod, the demoted tags appear on the target side volumes too:
This is because tags (along with everything else) get replicated to the target pod. So not only the data, but the tags, QoS settings, etc get copied to the target volumes. The final synchronization ensures these tags are replicated over.
The target side FlashArray operations occur during the Change recovery site storage to writable step.
First the pod is promoted.
Then the volumes are tagged with a tag that has a key of puresra-failover and the value of the UUID of the target pod.
If you query for tags without a filter (the above query looks for tags with the key of puresra-failover): you will see all of the tags.
Why does the SRA tag things at all? Well SRM requires operations to be idempotent-which means if you run something, or re-run it, it should not do something differently. A better way to explain this is that SRM needs to be able to pickup where it left off-without being supplied new information. The SRA needs to be able to figure out what it did and what it didn't do on the the last attempt. Tagging the source volumes provides the array a way to say "yes we demoted the source pod directly, it is not an error". The first time a failover is run, it is expected that the source pod is NOT demoted. But if it is run once, the array will demote it, something fails and then it is run again, the SRA shouldn't fail because it is demoted. Tagging it with this gives us a way to know, "yes it is demoted, but it is demoted because we demoted it on purpose, so move forward". Tags get added during transition states and remove during static states (temporary would be between a recovery and a reprotect or between a test recovery start and a test recovery cleanup).
When querying on the target side for tags, if the puresra-demoted tags are not visible it means that the source site was down during the recovery and could not be tagged. Ensure that the recovery is re-run to complete the process (assuming of course the source site is back online).
The last step is the volumes will be connected to the appropriate hosts and host groups. Note that, before any connections are made, the SRA will disconnect all of the volumes in the recovery plan from any hosts or host groups first (in case they were pre-connected) and then connect them as needed. This ensures that the volumes are only connected to the hosts dictated by SRM.
The volumes will be connected, and VMware will rescan the ESXi environment, resignature the datastores and mount them. VMs will then be powered-on according to the recovery plan.
Note that the resignature process adds the snap-XXXXXXX prefix to the datastore names. SRM has the ability to automatically remove this by enabling a non-default option SRM User Guide: Site Recovery Manager Advanced Options.
At the end of the recovery process (a reprotect has NOT been run yet) the target side is automatically already replicating back from the previously target pod. In the example above, the original configuration was the pod called srm-podA replicating to srm-podB. During the recovery, srm-podA was demoted and then srm-podB was promoted. When Purity sees the original source pod demoted and the target pod is promoted-it automatically swaps the replication link. The original target pod (srm-podB) is now a source pod, replicating back to the original source pod, which is now a target pod (srm-podA).
If this is state of the pods BEFORE recovery.
This is the state AFTER recovery and BEFORE reprotect.
Notice that the replication direction is switched.
What if this turns out to be a mistake? What if the data is needed that was in the original source pod (srm-podA) after replication has reversed?
This is why the undo pod exists.
The data (and configuration) point-in-time of the original source pod (srm-podA) is stored in a "snapshot" of the pod that is created upon demotion of the source pod. That pod remains around for the eradication window (default 24 hours) or until manually eradicated-whichever comes first.
This can be seen under Destroyed and Undo Pods on the original source FlashArray.
The process to restore from and undo pod is not built into SRM and will be a manual operation. See the ActiveDR with VMware User Guide for details.
Note:While the SRA attempts to eradicate undo pods created by demotions in a test recovery cleanup, it does not eradicate undo pods created by the demotion during an actual recovery.