Loon(y) Juice

Remus-fu

Remus is the Xen Hypervisor HA system, available from Xen 4.x & kernel 2.6.31.6 & higher.

This post documents how to install & use Remus & Xen live migration to deploy a HVM domU (guest OS/VM/VPS) on one hypervisor, use Remus to keep a copy of its disk & memory use updated on a 2nd hypervisor, then Failover (kill the original domU), then Migrate back to the original domU & re-start Remus to be able to do it all again, all without the domU/guest OS loosing any disk or memory!

The noticable item of this is that this doesn’t require any Clustering or Distributed Filesystems, each hypervisor has its own copy of the domU’s “disk”! So there’s no need for DRBD, GFS, CLVM etc etc in this Xen HA implementation!

See http://www.loonjuice.com/2010/03/07/debian-5-04-better-than-centos/ about how to install Remus & Xen on Debian and do so on 2 x HVM capable Hypervisors.

Then, create a HVM domU guest on what we’ll call “hypervisor1″. An example HVM guest domU cfg is:

import os, re
arch = os.uname()[4]
if re.search(’64′, arch):
arch_libdir = ‘lib64′
else:
arch_libdir = ‘lib’

kernel = “/usr/lib/xen/boot/hvmloader”
builder=’hvm’

memory = 256
name = “hvmdomuname”
vif = [ '' ]
disk = ['tap:remus:hypervisor2-IP-addr:9000|aio:/var/lib/xen/images/hvmguest.x86-64.img,hda,w']

device_model = ‘/usr/’ + arch_libdir + ‘/xen/bin/qemu-dm’

boot=”c”
sdl=0
vnc=1
vfb = [ 'type=vnc,vncpasswd=passw,vnclisten=0.0.0.0,vncdisplay=2' ]

serial=’pty’

Note that the IP address of hypervisor2 is used in the disk= line in the HVM guest’s .cfg listed above. This seems far more reliable than hostname use/lookups!

Copy the disk data/image file (/var/lib/xen/images/hvmguest.x86-64.img) to hypervisor2 before starting the domU on hypervisor1.

Once the data data is copied, start the HVM guest: xm create /etc/xen/hvmdomu.cfg

Once running, you’ll see a ‘tapdisk2′ process running on hypervisor1.

eg:

/usr/sbin/tapdisk2 -n remus:hypervisor2-IP-addr:9000|aio:/var/lib/xen/images/hvmguest.x86-64.img

The domU now running on hypervisor1 will eventually ‘pause’, waiting for the remus command to be run on hypervisor1 to start copying buffered & changed memory & disk items of the HVM guest. The guest will un-pause once the remus connection is running between the 2 hypervisors.

So start running remus on hypervisor1, providing the IP addr of hypervisor2. ie:

/usr/bin/remus –no-net hvmdomuname hypervisor2-IP-addr

This will fire up a domU on hypervisor2 which will be paused & the remus command on hypervisor1 will stay running & log its ongoing copying activity once the initial domU setup (memory transferred) is done.

Output of running remus command looks like:

SUSPEND shinfo ffffffffffffffff
delta 6ms, dom0 0%, target 0%, sent 0Mb/s, dirtied 180Mb/s 33 pages
6175: sent 0, skipped 0, delta 1ms, dom0 0%, target 0%, sent 0Mb/s, dirtied 0Mb/s 0 pages
Total pages sent= 178523 (0.17x)
(of which 0 were fixups)
All memory is saved
PROF: resumed at 1275809318.525883
resuming QEMU
Sending 6527 bytes of QEMU state
PROF: flushed memory at 1275809318.527712

And that just keeps going, copying memory every 100ms (default time in remus).

Then, if hypervisor1or the guest domU ‘disappears’ (eg: xm destroy hvmdomuname), then hypervisor2 will un-pause the domU & let it run. Any login or processes running on the domU when it was hosted on hypervisor1 should still be running when the domU is started on hypervisor2. ie: a successful Remus HA Migration!

Note that there will now be a tapdisk2 process running on hypervisor2, but pointing to hypervisor2′s IP address, so you can’t run the remus command on hypervisor2 pointing back to hypervisor1. The way to get remus-based HA & constant syncronisation is to live migrate the domU back to hypervisor1 with the ‘xm migrate –live domuname hypervisor1-IP-addr’ command on hypervisor2!

Once the domU is manually migrated back from hypervisor2 to hypervisor1, then the remus command can be re-run on hypervisor1 to start the constant replication of the domU’s memory & disk!

With that “The Cyle is Complete” – being start a domU on a hypervisor, use Remus to constantly mirror the memory & disk updates of the guest domU OS to another hypervisor, failover the domU to the 2nd hypervisor & then migrate it back to the original/’source’ hypervisor when its back online or available to host the domU again, then run Remus again on the original hypervisor to have the constant live updating of the domU’s memory & disk for HA again!