But I found that the overhead & workload of managing a Redhat Cluster between the 2 x Xen Hypervisors to be excessive & the failure/recovery modes & actions required whenever a Hypervisor host went offline, to be unreliable & requiring manual intervention (reset button) to hypervisors under some failure modes.

I’ve now moved to using OCFS2 on dual-primary DRBD between the Hypervisors & so far this has performed far better in coping with OCFS Peer loss & restart.

Actions req’d to install & setup OCFS2 on the Debian based Xen Remus Hypervisors was:

(on both hypervisor systems, after Xen Remus install)

apt-get install ocfs2-tools ocfs2console

Setup /etc/ocfs2/cluster.conf the same on both Hypervisors, eg:

node:
ip_port = 7777
ip_address = 192.168.X.Y
number = 0
name = debremus2
cluster = ocfs2

node:
ip_port = 7777
ip_address = 192.168.X.Z
number = 1
name = debremus1
cluster = ocfs2

cluster:
node_count = 2
name = xenocfs2cluster

Then initialise OCFS2 on both hosts with:

dpkg-reconfigure o2cb
/etc/init.d/o2cb restart

/etc/init.d/ocfs2 restart

Ensure that the same /etc/ocfs2/cluster.conf file is on both Hypervisors.

Ensure that DRBD is primary/primary on both Hypervisors.

Check the status (after initialisation) of OCFS2 on both hypervisors with:

/etc/init.d/o2cb status

Assuming you see something like the following, then proceed to make the OCFS2 filesystem!

Driver for “configfs”: Loaded
Filesystem “configfs”: Mounted
Stack glue driver: Loaded
Stack plugin “o2cb”: Loaded
Driver for “ocfs2_dlmfs”: Loaded
Filesystem “ocfs2_dlmfs”: Mounted
Checking O2CB cluster xenocfs2cluster: Online
Heartbeat dead threshold = 31
Network idle timeout: 30000
Network keepalive delay: 2000
Network reconnect delay: 2000
Checking O2CB heartbeat: Active

mkfs -t ocfs2 -N 2 -L ocfs2_drbd0 /dev/drbd0

Then you can mount that filesystem on both Hypervisors with:

mount -t ocfs2 /dev/drbd0 /usr/ocfs2

Now, finally, you can start your Remus enabled guests, using a ‘disk=’ line like:

disk = ['tap:remus:192.168.X.Y:9000|aio:/usr/ocfs2/vm1-hvm.img,hda,w']

Where 192.168.X.Y is the IP address of the 2nd/failover-to Hypervisor host!

Then, change the disk of the VM to use an image location being on a GFS2 mounted, DRBD distributed, clustered file system! (eg used below is /usr/gfs0 )

Current Xen 4.1-unstable includes Remus & uses/patches the 2.6.31.13 Linux kernel.

Build & install the latest DRBD sources, from http://www.drbd.org/docs/install/

ie:

download the relevant drbd-*.tar.gz, extract to /usr/local/src

run ./configure –with-km; make KDIR=/lib/modules/2.6.31.13/build ; make KDIR=/lib/modules/2.6.31.13/build install

You then need to configure DRBD using the same /usr/local/etc/drbd.conf on both hosts, eg: using /dev/loop0 as the backing disk device!!

global {
minor-count 10;
}

resource drbdr1 {
protocol C;
meta-disk    internal;
device        /dev/drbd0;
##disk        /dev/mapper/VG-LV_name;
disk        /dev/loop0;

startup {
wfc-timeout 30;
degr-wfc-timeout 30;
become-primary-on both;
}

disk {
on-io-error detach;
}

syncer {
rate 90M; # Note: ‘M’ is MegaBytes, not MegaBits
}

net {
allow-two-primaries;
after-sb-0pri discard-younger-primary;
after-sb-1pri discard-secondary;
after-sb-2pri call-pri-lost-after-sb;
}

on remus1 {
##device        /dev/drbd0;
##disk        /dev/loop0;
##disk        /dev/mapper/VG-LV_name;
address        192.168.X.Y1:7789;
##meta-disk    internal;
}

on remus2 {
##device        /dev/drbd0;
##disk        /dev/mapper/VG-LV_name;
address        192.168.X.Z:7789;
##meta-disk    internal;
}

}

Ensure that the same entries for hosts ‘remus1′ & ‘remus2′ are in each hosts /etc/hosts file! (V.IMPORTANT)

Now instantiate/initially-start DRBD with the following command set:

drbdadm create-md drbdr1

drbdadm attach drbdr1

drbdadm connect drbdr1

watch cat /proc/drbd

Then finally set Dual Primary mode with: (as long as you have dual-primary enabled in your active drbd.conf as perhttp://www.drbd.org/users-guide-emb/s-enable-dual-primary.html )

drbdadm primary drbdr1

With both hosts running in DRBD Primary modes (cat /proc/drbd |grep “Primary/Primary”) you can now get the GFS2 filesystem built on /dev/drbd0

mkfs.gfs2 -t clusternamefromcluster.conf:gfs1 -p lock_dlm -j 2 /dev/drbd0

Then you can mount the GFS2 filesystem on each host:

mkdir /usr/gfs0

mount /dev/drbd0 /usr/gfs0

(on each host)

Check status of Cluster, GFS & DRBD:

cat /proc/drbd

cman_tool status

cman_tool nodes

gfs2_tool df

More to be added, but you can install & run up a domU as per the original Remus-fu post using an image on /usr/gfs0 !

Download from drbd.org the 8.3.7 via http://oss.linbit.com/drbd/8.3/drbd-8.3.7.tar.gz

Follow their Doco at http://www.drbd.org/docs/about/, which results in the following commands:

tar xzvf drbd-8.3.7.tar.gz
ls -altr /lib/modules/`uname -r`/build
cd drbd-8.3.7
./configure –with-km
make KDIR=/lib/modules/`uname -r`/build
make KDIR=/lib/modules/`uname -r`/build install

modprobe drbd

lsmod

cat /proc/drbd

Then you need to edit /usr/local/etc/drbd.conf

global {
minor-count 10;
}

resource drbdr1 {
protocol C;
meta-disk       internal;

startup {
wfc-timeout 30;
degr-wfc-timeout 30;
become-primary-on both;
}

disk {
on-io-error detach;
}

syncer {
rate 90M; # Note: ‘M’ is MegaBytes, not MegaBits
}

net {
allow-two-primaries;
##data-integrity-alg;
after-sb-0pri discard-younger-primary;
after-sb-1pri discard-secondary;
after-sb-2pri call-pri-lost-after-sb;
}

on serv11 {
device          /dev/drbd0;
disk            /dev/mapper/guests-drbd1;
address         113.212.97.230:7789;
##meta-disk     internal;
}

on serv12 {
device          /dev/drbd0;
disk            /dev/mapper/guests-drbd1;
address         113.212.97.240:7789;
##meta-disk     internal;
}

}

Commands then to run up DRBD:

drbdadm create-md drbdr1
drbdadm syncer drbdr1
drbdadm connect drbdr1
drbdadm up drbdr1
drbdadm — –overwrite-data-of-peer primary drbdr1