vReplicator - VM Failover Application Consistency Issue

Information

Title: 

vReplicator - VM Failover Application Consistency Issue

Author(s): 

Xtravirt (Paul Buckle)

Target Audience: 

Technical - Intermediate

First Published: 

3 February 2009

Products: 

Vizioncore vReplicator

UID: 

XD10011

This hot tip is based on an issue originally identified and discussed in the Vizioncore Support Forum thread at http://tinyurl.com/bhfdwc and is known to apply to vReplicator 2.5.1 (7) and 2.5.2 (4), where there is a consistency issue with vReplicator VM failover

1.0 Overview

vReplicator enables you to clone a VMware VI3 source VM to a destination VM on a remote ESX Server and to keep it synchronised with the source via a user-defined replication schedule. The destination VM can then be manually failed over in the event of a total loss of the source VM, minimising unplanned downtime.

2.0 Issue

Sooner or later, the need will arise to failover from to a destination VM from a source VM that has been kept in sync according to the vReplicator job. However, if you have implemented a configuration that results in the maintaining of an application-consistent destination VM, eg: using Microsoft VSS for a VM hosting a SQL database; you should be aware of a behaviour in the vReplicator failover process that can result in the degradation of the destination VM to merely crash-consistent.

Failover to a destination VM is initiated from the vReplicator Client (after disabling the corresponding replication job) and results in the following prompt:

Vizioncore state that vReplicator takes the following sequential actions for each of the available options:

Yes

• Perform a “Guest Shutdown” on the source VM. *
• Perform a final synchronisation from the source VM to the destination VM.
• Power on the destination VM.

No

• Perform a “Power Off” on the source VM. *
• Power on the destination VM.

Cancel

• Cancel the failover action.

Note: If the source VM is powered on and vReplicator can submit instructions to it via the VMware API.

While the “No” and “Cancel” options operate as described, it is important to note that the “Yes” option does not. Rather than performing a clean “Guest Shutdown” of the source VM, it actually performs a “Power Off” before initiating the final synchronisation (as evidenced by a "Power Off VM" instruction appearing in the Recent Tasks pane of the VMware Virtual Infrastructure Client).

Consequently, the Guest OS in the source VM is not shutdown cleanly and this state is replicated to the destination VM, resulting in the application-consistent destination VM being over written with a crash-consistent one and the potential for data loss or corruption. Note that the “Power Off” instruction is issued to the source VM regardless of whether it is able to successfully action to a “Shutdown Guest” instruction or not.

The result of raising this issue with Vizioncore Support was the confirmation that vReplicator issues a “Power Off” instruction because the developers “found too many use cases where a soft, or even a try-soft, power down never actually powered down the VM”. So, for the time being at least, it appears unlikely that this feature of vReplicator is going to operate as Vizioncore originally intended.

3.0 Workaround

Fortunately there is a simple workaround which entails gracefully shutting down the source VM Guest OS manually before initiating a failover and electing to perform a final synchronisation.

References

References: 

1. Vizioncore Support Forums: http://tinyurl.com/bhfdwc

Links: 

1. Vizioncore Support Forums: http://tinyurl.com/bhfdwc