Loon(y) Juice

3 Comments to “Blogging updates to that Debia…”

1. 

   By Jon Mitchell, May 6, 2010 @ 7:20 am

   Hey,

   Partially followed your post to get xen4.0.1 + 2.6.31.13 pvops setup on Debian 5.0.4.

   I’m curious if you’ve had a chance to try Remus yet?

   Cheers,

   Jon

   Log in to Reply
2. 

   By rich, May 6, 2010 @ 10:47 am

   Hi Jon,

   Yes & Remus works just fine. There’s a few caveats/gotya’s that I found during the process & I’ll document a few here in point-form, so feel free to get back in contact if you want more info..

   a) I ended up using the xen-4.0-testing.hg GIT repo, as xen-unstable.hg is, well, true to name & some things seemed to be more reliable in xen 4.0′s tree. The latests xen-4.0 GIT now uses the 2.6.31.13 kernel like xen-unstable has, which is fine.

   b) I followed & stepped through the Remus Documentation at http://nss.cs.ubc.ca/remus/doc.html methodically & fixed anything that wasn’t working in a preceding step before trying to go further. Xen Live Migration needs particular settings in xend-config.sxp like:

   (xend-http-server yes)
   (xend-unix-server yes)
   (xend-relocation-server yes)
   (xend-relocation-ssl-server no)
   (xend-unix-path /var/lib/xend/xend-socket)
   (xend-port 8000)
   (xend-relocation-port 8002)
   (xend-relocation-ssl no)
   (xend-address ”)
   (xend-relocation-address ”)
   (xend-relocation-hosts-allow ‘^localhost$ ^localhost\\.localdomain$ ^hostname*\\.domainname\\.com$’)

   Firewall down the 8000, 8002 & port 9000 (tcp & udp) & configure your xend-relocation-hosts-allow accordingly, correctly prefixing any ‘.’ in the hostnames with \\.

   c) Ensure that you can ‘xm migrate –live’ your domains manually & first before trying further ‘remus’ migration commands.

   Note that the key to manual –live migrations is to have the same image store file accessible to both hypervisors. eg: a SAN/NAS/iSCSI/DRBD/NFS based storage mounted for where the image file or LV of the domU is accessed by Xen. If you don’t have that exact same file available to each hypervisor, then your guest domain will crash on the destination hypervisor when it completes its migration (which defeats the purpose of HA).

   d) If you can’t maintain a shared image store filesystem between the 2 x Remus-capable hypervisors, then you need to aim for using the ‘remus’ image type in the domU’s .cfg disk = config paramater.
   eg as per the Remus Doco:
   disk = [ 'tap:remus:bkup:9000|aio:/dev/vmdisk/myguest,xvda1,w' ]

   There is one major issue with Remus disk types, which is that pygrub doesn’t work with these image disk types (yet/currently)! So you end up having to run your remus-based HA guests as HVM mode domUs. I’m currently working through a reliable solution to this, but nothing of substance to talk of yet on that!

   When you run your guests with tap:remus disks, you still just run the same ‘remus’ command for that guest as you may have with a standard/pygrub-supported disk setting.

   I’m using Remus now across 2 sites & with a few Remus-managed VMs on each hypervisor & so far it’s worked aok. Of course, like anything #floss or #opensource, you need to write your own scripts to automate remus restarts (remus exits on the hypervisor that’s handing off the guest & needs to be re-run on the destination hypervisor once the guest is unpaused & running on the destination hypervisor).

   Hope all that helps.

   Log in to Reply
3. 

   By rich, May 6, 2010 @ 11:15 pm

   Further update is that I’ve got PV guests migrating live via Remus AOK, as long as their disk= line is a file: image on a GFS2 DRBD-backed filesystem, which is mounted on each hypervisor and each hypervisor is a ‘primary’ on the DRBD device. ie: “dual primary” or ro:Primary/Primary in your /proc/drbd

   This means you need all the Redhat Cluster relevant packages installed (relevant to Debian Lenny in my context here) as well as GFS2 in the kernel.
   re: “GFS2 in the kernel”, some of the .config flags relevant to that are:
   CONFIG_GFS2_FS=y
   CONFIG_GFS2_FS_LOCKING_DLM=y
   CONFIG_CONFIGFS_FS=y

   Perse CLVM is NOT req’d! (I make a GFS2 filesystem directly on /dev/drbd0)

   Packages with their pre-req’s to be installed via ‘apt-get install’ include:
   gfs2-tools
   cman

   DRBD is installed from source via drbd.org & a make command like:
   cd /usr/local/src/drbd-X.Y.Z
   make KDIR=/lib/modules/`uname -r`/build clean
   make KDIR=/lib/modules/`uname -r`/build
   make KDIR=/lib/modules/`uname -r`/build install

   Then ‘modprobe drbd’ or such to load that DRBD module.

   /etc/drbd.conf (or /usr/local/etc/drbd.conf if built from source) looks something like the following & needs to be the same on both DRBD nodes/hosts:
   where remus1 is on IP 192.168.1.1 & remus2 is on 192.168.1.2

   global {
   minor-count 10;
   }

   resource drbdr1 {
   protocol C;
   meta-disk internal;
   device /dev/drbd0;
   disk /dev/mapper/VG-LV_name;

   startup {
   wfc-timeout 30;
   degr-wfc-timeout 30;
   become-primary-on both;
   }

   disk {
   on-io-error detach;
   }

   syncer {
   rate 90M; # Note: ‘M’ is MegaBytes, not MegaBits
   }

   net {
   allow-two-primaries;
   after-sb-0pri discard-younger-primary;
   after-sb-1pri discard-secondary;
   after-sb-2pri call-pri-lost-after-sb;
   }

   on remus1 {
   address 192.168.1.1:7789;
   }

   on remus2 {
   address 192.168.1.2:7789;
   }

   }

   Get that DRBD device to be UpToDate on both primary nodes before trying to get a domU/guest migrated!

   /etc/cluster/cluster.conf needs to be the same on both/all hypervisors.

   A generic cluster.conf is:
   (where the hostnames ‘remus1′ & ‘remus2′ are defined in /etc/hosts on both hosts)
   (note the ‘two_node=”1″‘ for “dual primary” assistance)

   < ?xml version="1.0"?>
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   

   Then ensure /etc/init.d/cman start succeeds on both hosts & ‘cman_tool status’ yields something like:

   …
   Cluster Member: Yes
   Membership state: Cluster-Member
   Nodes: 2
   Expected votes: 1
   Total votes: 2
   Node votes: 1
   Quorum: 1
   …

   With all that you can then make a GFS2 filesystem on /dev/drbd0 with:

   mkfs.gfs2 -t cluster1:drbd0gfs -p lock_dlm -j 2 /dev/drbd0

   Where:
   -t cluster1 is the cluster name from the cluster.conf file
   drbd0gfs is any arbitary text you’d like for this filesystem on the cluster!

   Then you can mount that filesystem on both the ‘remus1′ & ‘remus2′ hosts with:

   mount -t gfs2 /dev/drbd0 /mntpoint

   Put your Xen guest’s img on /mntpoint & run it on one of the hypervisors with a disk=['file:/mntpoint/vm.img,.....] entry & then migrate it manually (xm migrate –live guestname remote-hypervisor-IP-addr) or via Remus with ‘remus guestname remote-hypervisor-IP-addr’

   Solid braindump there, enjoy!

   Log in to Reply

RSS feed for comments on this post. TrackBack URI