http://lists.ausnog.net/pipermail/ausnog/2010-November/009024.html

——

From: Bevan Slattery
Sent: Thursday, 18 November 2010 10:04 AM
To: [removed]
Subject: The Death of NBN 2.0

Guys,

I am going to make a big call and suggest you mark down the 18th November, 2010 in your calendar.  It will be the day that NBN 2.0 became terminal.  Either produce the Red Book and business case, that shows there is no business case and massive risk to the tax-payer, or don’t produce it and the momentum of anger will grow as fast as they “burn the goodwill” of those that support it.  Either way – it dies

——-

Right now I tend to agree. I think it may be too early to make this call, but I fear/suspect that Conroy’s cooked the NBN Golden Goose!

Oh dear.

But I found that the overhead & workload of managing a Redhat Cluster between the 2 x Xen Hypervisors to be excessive & the failure/recovery modes & actions required whenever a Hypervisor host went offline, to be unreliable & requiring manual intervention (reset button) to hypervisors under some failure modes.

I’ve now moved to using OCFS2 on dual-primary DRBD between the Hypervisors & so far this has performed far better in coping with OCFS Peer loss & restart.

Actions req’d to install & setup OCFS2 on the Debian based Xen Remus Hypervisors was:

(on both hypervisor systems, after Xen Remus install)

apt-get install ocfs2-tools ocfs2console

Setup /etc/ocfs2/cluster.conf the same on both Hypervisors, eg:

node:
ip_port = 7777
ip_address = 192.168.X.Y
number = 0
name = debremus2
cluster = ocfs2

node:
ip_port = 7777
ip_address = 192.168.X.Z
number = 1
name = debremus1
cluster = ocfs2

cluster:
node_count = 2
name = xenocfs2cluster

Then initialise OCFS2 on both hosts with:

dpkg-reconfigure o2cb
/etc/init.d/o2cb restart

/etc/init.d/ocfs2 restart

Ensure that the same /etc/ocfs2/cluster.conf file is on both Hypervisors.

Ensure that DRBD is primary/primary on both Hypervisors.

Check the status (after initialisation) of OCFS2 on both hypervisors with:

/etc/init.d/o2cb status

Assuming you see something like the following, then proceed to make the OCFS2 filesystem!

Driver for “configfs”: Loaded
Filesystem “configfs”: Mounted
Stack glue driver: Loaded
Stack plugin “o2cb”: Loaded
Driver for “ocfs2_dlmfs”: Loaded
Filesystem “ocfs2_dlmfs”: Mounted
Checking O2CB cluster xenocfs2cluster: Online
Heartbeat dead threshold = 31
Network idle timeout: 30000
Network keepalive delay: 2000
Network reconnect delay: 2000
Checking O2CB heartbeat: Active

mkfs -t ocfs2 -N 2 -L ocfs2_drbd0 /dev/drbd0

Then you can mount that filesystem on both Hypervisors with:

mount -t ocfs2 /dev/drbd0 /usr/ocfs2

Now, finally, you can start your Remus enabled guests, using a ‘disk=’ line like:

disk = ['tap:remus:192.168.X.Y:9000|aio:/usr/ocfs2/vm1-hvm.img,hda,w']

Where 192.168.X.Y is the IP address of the 2nd/failover-to Hypervisor host!

Then, change the disk of the VM to use an image location being on a GFS2 mounted, DRBD distributed, clustered file system! (eg used below is /usr/gfs0 )

Current Xen 4.1-unstable includes Remus & uses/patches the 2.6.31.13 Linux kernel.

Build & install the latest DRBD sources, from http://www.drbd.org/docs/install/

ie:

download the relevant drbd-*.tar.gz, extract to /usr/local/src

run ./configure –with-km; make KDIR=/lib/modules/2.6.31.13/build ; make KDIR=/lib/modules/2.6.31.13/build install

You then need to configure DRBD using the same /usr/local/etc/drbd.conf on both hosts, eg: using /dev/loop0 as the backing disk device!!

global {
minor-count 10;
}

resource drbdr1 {
protocol C;
meta-disk    internal;
device        /dev/drbd0;
##disk        /dev/mapper/VG-LV_name;
disk        /dev/loop0;

startup {
wfc-timeout 30;
degr-wfc-timeout 30;
become-primary-on both;
}

disk {
on-io-error detach;
}

syncer {
rate 90M; # Note: ‘M’ is MegaBytes, not MegaBits
}

net {
allow-two-primaries;
after-sb-0pri discard-younger-primary;
after-sb-1pri discard-secondary;
after-sb-2pri call-pri-lost-after-sb;
}

on remus1 {
##device        /dev/drbd0;
##disk        /dev/loop0;
##disk        /dev/mapper/VG-LV_name;
address        192.168.X.Y1:7789;
##meta-disk    internal;
}

on remus2 {
##device        /dev/drbd0;
##disk        /dev/mapper/VG-LV_name;
address        192.168.X.Z:7789;
##meta-disk    internal;
}

}

Ensure that the same entries for hosts ‘remus1′ & ‘remus2′ are in each hosts /etc/hosts file! (V.IMPORTANT)

Now instantiate/initially-start DRBD with the following command set:

drbdadm create-md drbdr1

drbdadm attach drbdr1

drbdadm connect drbdr1

watch cat /proc/drbd

Then finally set Dual Primary mode with: (as long as you have dual-primary enabled in your active drbd.conf as perhttp://www.drbd.org/users-guide-emb/s-enable-dual-primary.html )

drbdadm primary drbdr1

With both hosts running in DRBD Primary modes (cat /proc/drbd |grep “Primary/Primary”) you can now get the GFS2 filesystem built on /dev/drbd0

mkfs.gfs2 -t clusternamefromcluster.conf:gfs1 -p lock_dlm -j 2 /dev/drbd0

Then you can mount the GFS2 filesystem on each host:

mkdir /usr/gfs0

mount /dev/drbd0 /usr/gfs0

(on each host)

Check status of Cluster, GFS & DRBD:

cat /proc/drbd

cman_tool status

cman_tool nodes

gfs2_tool df

More to be added, but you can install & run up a domU as per the original Remus-fu post using an image on /usr/gfs0 !

@scalzi

Postsingular author = Rudy Rucker!

@charliestross (nic?)

Download from drbd.org the 8.3.7 via http://oss.linbit.com/drbd/8.3/drbd-8.3.7.tar.gz

Follow their Doco at http://www.drbd.org/docs/about/, which results in the following commands:

tar xzvf drbd-8.3.7.tar.gz
ls -altr /lib/modules/`uname -r`/build
cd drbd-8.3.7
./configure –with-km
make KDIR=/lib/modules/`uname -r`/build
make KDIR=/lib/modules/`uname -r`/build install

modprobe drbd

lsmod

cat /proc/drbd

Then you need to edit /usr/local/etc/drbd.conf

global {
minor-count 10;
}

resource drbdr1 {
protocol C;
meta-disk       internal;

startup {
wfc-timeout 30;
degr-wfc-timeout 30;
become-primary-on both;
}

disk {
on-io-error detach;
}

syncer {
rate 90M; # Note: ‘M’ is MegaBytes, not MegaBits
}

net {
allow-two-primaries;
##data-integrity-alg;
after-sb-0pri discard-younger-primary;
after-sb-1pri discard-secondary;
after-sb-2pri call-pri-lost-after-sb;
}

on serv11 {
device          /dev/drbd0;
disk            /dev/mapper/guests-drbd1;
address         113.212.97.230:7789;
##meta-disk     internal;
}

on serv12 {
device          /dev/drbd0;
disk            /dev/mapper/guests-drbd1;
address         113.212.97.240:7789;
##meta-disk     internal;
}

}

Commands then to run up DRBD:

drbdadm create-md drbdr1
drbdadm syncer drbdr1
drbdadm connect drbdr1
drbdadm up drbdr1
drbdadm — –overwrite-data-of-peer primary drbdr1