But I found that the overhead & workload of managing a Redhat Cluster between the 2 x Xen Hypervisors to be excessive & the failure/recovery modes & actions required whenever a Hypervisor host went offline, to be unreliable & requiring manual intervention (reset button) to hypervisors under some failure modes.

I’ve now moved to using OCFS2 on dual-primary DRBD between the Hypervisors & so far this has performed far better in coping with OCFS Peer loss & restart.

Actions req’d to install & setup OCFS2 on the Debian based Xen Remus Hypervisors was:

(on both hypervisor systems, after Xen Remus install)

apt-get install ocfs2-tools ocfs2console

Setup /etc/ocfs2/cluster.conf the same on both Hypervisors, eg:

node:
ip_port = 7777
ip_address = 192.168.X.Y
number = 0
name = debremus2
cluster = ocfs2

node:
ip_port = 7777
ip_address = 192.168.X.Z
number = 1
name = debremus1
cluster = ocfs2

cluster:
node_count = 2
name = xenocfs2cluster

Then initialise OCFS2 on both hosts with:

dpkg-reconfigure o2cb
/etc/init.d/o2cb restart

/etc/init.d/ocfs2 restart

Ensure that the same /etc/ocfs2/cluster.conf file is on both Hypervisors.

Ensure that DRBD is primary/primary on both Hypervisors.

Check the status (after initialisation) of OCFS2 on both hypervisors with:

/etc/init.d/o2cb status

Assuming you see something like the following, then proceed to make the OCFS2 filesystem!

Driver for “configfs”: Loaded
Filesystem “configfs”: Mounted
Stack glue driver: Loaded
Stack plugin “o2cb”: Loaded
Driver for “ocfs2_dlmfs”: Loaded
Filesystem “ocfs2_dlmfs”: Mounted
Checking O2CB cluster xenocfs2cluster: Online
Heartbeat dead threshold = 31
Network idle timeout: 30000
Network keepalive delay: 2000
Network reconnect delay: 2000
Checking O2CB heartbeat: Active

mkfs -t ocfs2 -N 2 -L ocfs2_drbd0 /dev/drbd0

Then you can mount that filesystem on both Hypervisors with:

mount -t ocfs2 /dev/drbd0 /usr/ocfs2

Now, finally, you can start your Remus enabled guests, using a ‘disk=’ line like:

disk = ['tap:remus:192.168.X.Y:9000|aio:/usr/ocfs2/vm1-hvm.img,hda,w']

Where 192.168.X.Y is the IP address of the 2nd/failover-to Hypervisor host!

Then, change the disk of the VM to use an image location being on a GFS2 mounted, DRBD distributed, clustered file system! (eg used below is /usr/gfs0 )

Current Xen 4.1-unstable includes Remus & uses/patches the 2.6.31.13 Linux kernel.

Build & install the latest DRBD sources, from http://www.drbd.org/docs/install/

ie:

download the relevant drbd-*.tar.gz, extract to /usr/local/src

run ./configure –with-km; make KDIR=/lib/modules/2.6.31.13/build ; make KDIR=/lib/modules/2.6.31.13/build install

You then need to configure DRBD using the same /usr/local/etc/drbd.conf on both hosts, eg: using /dev/loop0 as the backing disk device!!

global {
minor-count 10;
}

resource drbdr1 {
protocol C;
meta-disk    internal;
device        /dev/drbd0;
##disk        /dev/mapper/VG-LV_name;
disk        /dev/loop0;

startup {
wfc-timeout 30;
degr-wfc-timeout 30;
become-primary-on both;
}

disk {
on-io-error detach;
}

syncer {
rate 90M; # Note: ‘M’ is MegaBytes, not MegaBits
}

net {
allow-two-primaries;
after-sb-0pri discard-younger-primary;
after-sb-1pri discard-secondary;
after-sb-2pri call-pri-lost-after-sb;
}

on remus1 {
##device        /dev/drbd0;
##disk        /dev/loop0;
##disk        /dev/mapper/VG-LV_name;
address        192.168.X.Y1:7789;
##meta-disk    internal;
}

on remus2 {
##device        /dev/drbd0;
##disk        /dev/mapper/VG-LV_name;
address        192.168.X.Z:7789;
##meta-disk    internal;
}

}

Ensure that the same entries for hosts ‘remus1′ & ‘remus2′ are in each hosts /etc/hosts file! (V.IMPORTANT)

Now instantiate/initially-start DRBD with the following command set:

drbdadm create-md drbdr1

drbdadm attach drbdr1

drbdadm connect drbdr1

watch cat /proc/drbd

Then finally set Dual Primary mode with: (as long as you have dual-primary enabled in your active drbd.conf as perhttp://www.drbd.org/users-guide-emb/s-enable-dual-primary.html )

drbdadm primary drbdr1

With both hosts running in DRBD Primary modes (cat /proc/drbd |grep “Primary/Primary”) you can now get the GFS2 filesystem built on /dev/drbd0

mkfs.gfs2 -t clusternamefromcluster.conf:gfs1 -p lock_dlm -j 2 /dev/drbd0

Then you can mount the GFS2 filesystem on each host:

mkdir /usr/gfs0

mount /dev/drbd0 /usr/gfs0

(on each host)

Check status of Cluster, GFS & DRBD:

cat /proc/drbd

cman_tool status

cman_tool nodes

gfs2_tool df

More to be added, but you can install & run up a domU as per the original Remus-fu post using an image on /usr/gfs0 !

Install Debian 5.04 amd64 via DVD 1 & enable network interface & Network Repos!

Debian 5.04 amd64 ISO is: ftp://mirror.internode.on.net/debian-cd/5.0.4/amd64/iso-dvd/debian-504-amd64-DVD-1.iso

/etc/apt/sources.list setup after install & used throughout Xen Unstable install is:

##deb cdrom:[Debian GNU/Linux 5.0.4 _Lenny_ - Official amd64 DVD Binary-1 20100131-22:09]/ lenny contrib main

deb http://ftp.au.debian.org/debian/ lenny main
deb-src http://ftp.au.debian.org/debian/ lenny main

deb http://security.debian.org/ lenny/updates main contrib
deb-src http://security.debian.org/ lenny/updates main contrib

deb http://volatile.debian.org/debian-volatile lenny/volatile main contrib
deb-src http://volatile.debian.org/debian-volatile lenny/volatile main contrib

Pre-Req’s:

apt-get update

apt-get install’s done:

apt-get install build-essential
apt-get install zlib1g-dev python-dev libncurses5-dev libssl-dev xorg-dev
apt-get install mercurial gawk bcc bin86
apt-get install openssh-server
apt-get install git-core
apt-get install libsdl1.2-dev
apt-get install uuid-dev
apt-get install gettext
apt-get install iasl
apt-get install bridge-utils
apt-get install flex bison libpci-dev

Get to it!:

cd /usr/local/src

hg clone http://xenbits.xen.org/xen-unstable.hg

cd xen-unstable.hg

make world

When “make world” stops to configure kernel, CTRL-C the make menuconfig within the kernel’s build dirs!

Edit ./linux-2.6-pvops.git/arch/x86/configs/i386_defconfig AND ./linux-2.6-pvops.git/arch/x86/configs/x86_64_defconfig to enable whatever config entries you want.

Re-start the xen-unstable.hg/ make dist process & work through the prompts for the kernel config stage & all going well, it should configure & build the kernel & Xen!

make install

mkinitramfs -k -o ./initramfs-2.6.31.6 2.6.31.6

cp /initramfs-2.6.31.6 /boot/initrd.img-2.6.31.6

Set GRUB menu.lst to boot on this Xen install:

title           Xen Unstable + kernel 2.6.31.6
root            (hd0,0)
kernel          /boot/xen-4.0.gz
module          /boot/vmlinuz-2.6.31.6 root=/dev/sda1 ro quiet
module          /boot/initrd.img-2.6.31.6

Boot on that!

First you’ll need to manually start xend for the 1st time!

/etc/init.d/xend start

Check networking, xm list output, etc..

Set xend to start at boot via:

updated-rc.d xend defaults

updated-rc.d xendomains defaults

Update xend-config.sxp to support Live Migration via Replication:

(xend-relocation-server yes)
(xend-relocation-port 8002)
(xend-relocation-ssl no)
(xend-relocation-address ”)
(xend-relocation-hosts-allow ”)

Load those xend-config.sxp changes with:

/etc/init.d/xend restart

See that your Xen is sitting on TCP port 8002 with:

lsof -i tcp:8002

Do all this on > 1 physical host/server to get other Hypervisors available to Replicate to!