Difference between revisions of "DRBD"

Latest revision as of 21:25, 21 October 2011

[edit] A Vserver guest mirrored by DRBD with heartbeat failover

The idea is simple: Install a vserver on a DRBD resource and let it failover to the other node of a cluster.

This configuration uses vserver, DRBD and heartbeat. This page will not describe, how these packages are installed and configured. Only the important topics are mentioned.

Steps of installation:

• compile a vserver kernel
• compile the DRBD module against this kernel
• install the kernel, the module and reboot
• install the user space utilities of DRBD, vserver and heartbeat
• install the same packages at the second node
• configure and activate a DRBD resource
• make the DRBD resource primary on one node
• create a journalled filesystem (ext3, xfs, reiser) on top of the DRBD device
• mount the DRBD device as /var/lib/vservers/<vserver-name>
• install a vserver
• finish configuration
• sync the configuration to the other node

[edit] In detail

[edit] Install the necessary packages

On Debian Sarge we can use ready packages. Your /etc/apt/sources.list should have an entry pointing to unstable.

apt-get update

Maybe some additional packages are necessary:

apt-get install libdb3-dev libncurses-dev docbook-utils 
apt-get install fakeroot dpatch

Get the kernel source and unpack it:

apt-get install linux-source-2.6.12
cd /usr/src
tar xjf /usr/src/linux-source-2.6.12.tar.bz2

Maybe you want to have your old .config

cd linux-source-2.6.12
cp /boot/config-2.6.12-1-386 .config

Now it's time to install patches:

apt-get install util-vserver -t unstable
apt-get install kernel-patch-vserver -t unstable

Compile the Kernel:

make-kpkg --rootcmd fakeroot --revision custom01 --added-patches  vserver--append-to-version +vserver --initrd binary-arch

Now you should find a .deb ready to install:

cd ..
dpkg -i kernel-image-2.6.12+vserver_custom01_i386.deb

The configuration of GRUB should be updated now to boot the new kernel - try a reboot.

Install the DRBD packages:

apt-get install drbd0.7-module-source -t unstable
apt-get install drbd0.7-utils -t unstable

Unpack the module source:

cd /usr/src/
tar xzf drbd0.7.tar.gz

Compile the DRBD module:

cd linux-source-2.6.12
make-kpkg --append-to-version +vserver modules_image

Install the module:

cd ..
dpkg -i /usr/src/drbd0.7-module-2.6.12+vserver_0.7.11-1+custom01_i386.deb
update-modules
modprobe drbd

[edit] Prepare the disk space

You will need some block device like /dev/hdaX, or a better choice is to use LVM.

lvcreate -L3000 -nlvvs03 vg0

A recommended method is, to have DRBD metadata on an extra block device for each DRBD device.

Prepare space for it:

lvcreate -L132 -nlvvs03md vg0

[edit] Configure DRBD

Create an entry in /etc/drbd.conf using your favorite editor:

resource vs03 {
  protocol C;
  incon-degr-cmd "echo '!DRBD! pri on incon-degr' | wall ; sleep 60 ; halt -f";

  startup {
    wfc-timeout  100;
    degr-wfc-timeout 120;    # 2 minutes.
  }

  disk {
    on-io-error   pass_on;
  }

  net {
    # sndbuf-size 512k;
    # timeout       60;    #  6 seconds  (unit = 0.1 seconds)
    # connect-int   10;    # 10 seconds  (unit = 1 second)
    # ping-int      10;    # 10 seconds  (unit = 1 second)
    # max-buffers     2048;
    # max-epoch-size  2048;
    # ko-count 4;
    # on-disconnect reconnect;
  }

  syncer {
    rate 10M;
    group 1;
    al-extents 257;
  }

  on cel600 {
    device     /dev/drbd1;
    disk       /dev/vg0/lvvs03;
    address    192.168.0.22:7789;
    meta-disk  /dev/vg0/lvvs03md[0];
  }

  on xp2400 {
    device    /dev/drbd1;
    disk      /dev/vg0/lvvs03;
    address   192.168.0.21:7789;
    meta-disk /dev/vg0/lvvs03md[0];
  }
}

Start DRBD

/etc/init.d/drbd start

The DRBD resource is in an undefined state. It needs to be forced to be 'primary' on the current node.

drbdsetup /dev/drbd1 primary --do-what-I-say

Obviously a filesystem is needed on top of DRBD:

mke2fs -j /dev/drbd1

Also we need a mount point somewhere:

mkdir /mnt/vs03

As heartbeat is intended to manage the vserver, using the heartbeat script for mounting is a first test:

/etc/ha.d/resource.d/Filesystem /dev/drbd1 /mnt/vs03 ext3 start

[edit] Install the vserver guest

There are many methods how to install a vserver. Here 'debootstrap' is used. The important option is '--rootdir' to force the installation to the DRBD device.

vserver -v vs03 build --force --rootdir /mnt/vs03 \
-m debootstrap --hostname vs03.nack.at \
--netdev eth0 --interface 192.168.0.3/24 --context 3 -- \
-d sarge -m http://ftp.at.debian.org/debian/

We could start the vserver now. But first we try to make the configuration more pretty.

cd /mnt/vs03/vs03/etc/rc0.d
rm K20makedev K25hwclock.sh S30urandom S31umountnfs.sh  S35networking S36ifupdown S40umountfs S90halt K89klogd
cd ../rc6.d
rm K20makedev K25hwclock.sh S30urandom S31umountnfs.sh  S35networking S36ifupdown S40umountfs S90reboot K89klogd
cd ../rcS.d
rm S05keymap.sh S48console-screen.sh S50hwclock.sh S40networking  S45mountnfs.sh S10checkroot.sh S02mountvirtfs
rm S30procps.sh S35mountall.sh S36mountvirtfs S39ifupdown S30checkfs.sh S18ifupdown-clean S18hwclockfirst.sh
cd ../rc2.d
rm S20makedev S11klogd

If we want to have virtual IPs in the host context managed by heartbeat, there can be conflicts (and funny effects), if we do not 'name' the alias of the vserver IP.

echo vs03 > /etc/vservers/vs03/interfaces/0/name

sshd should only listen to the IPs of the host context:

xp2400:/# cat /etc/ssh/sshd_config | grep ListenAddress
ListenAddress 192.168.0.21:22
ListenAddress 192.168.0.20:22

We want to limit the vserver to 1 GB AS/VM (virtual address space) and 128 MB RSS (real memory). Note, that the units are pages.

xp2400:/# mkdir /etc/vservers/vs03/rlimits
xp2400:/usr/src# echo 250000 > /etc/vservers/vs03/rlimits/as
xp2400:/# echo 32000 > /etc/vservers/vs03/rlimits/rss

We want control CPU consumption:

xp2400:/# echo sched_prio > /etc/vservers/vs03/flags
xp2400:/# echo 6 > /etc/vservers/vs03/schedule
xp2400:/# echo 30 >> /etc/vservers/vs03/schedule
xp2400:/# echo 1000 >> /etc/vservers/vs03/schedule
xp2400:/# echo 200 >> /etc/vservers/vs03/schedule
xp2400:/# echo 3000 >> /etc/vservers/vs03/schedule
xp2400:/# echo dummy >> /etc/vservers/vs03/schedule
xp2400:/# cat /etc/vservers/vs03/schedule
6
30
1000
200
3000
dummy

After these checks it's time to start the vserver:

xp2400:/# vserver vs03 start

Check if the interface is nicely named and shown with ifconfig:

xp2400:/#  ip addr show
[...]
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:30:bd:6b:d6:8f brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.21/24 brd 192.168.0.255 scope global eth0
    inet 192.168.0.3/24 brd 192.168.0.255 scope global secondary eth0:vs03
    inet6 fe80::230:bdff:fe6b:d68f/64 scope link
       valid_lft forever preferred_lft forever

xp2400:/# ifconfig
eth0      Link encap:Ethernet  [...]

eth1      [...]

eth0:vs03 Link encap:Ethernet  HWaddr 00:30:BD:6B:D6:8F
          inet addr:192.168.0.3  Bcast:192.168.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:177 Base address:0xd000

Enter the vserver:

xp2400:/# vserver vs03 enter

In the vserver we check the base configuration and install some packages.

vs03:/# base-config
vs03:/# apt-get update
vs03:/# apt-get install locales
vs03:/# apt-get install less
vs03:/# nano /etc/resolv.conf
vs03:/# ping www.linux-vserver.org

We want to install at least one service, sshd.

vs03:/# apt-get install ssh
vs03:/# nano /etc/ssh/sshd_config
vs03:/# cat /etc/ssh/sshd_config | grep ListenAddress
ListenAddress 192.168.0.3:22
vs03:/# /etc/init.d/ssh restart
Restarting OpenBSD Secure Shell server: sshd.
vs03:/# passwd
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
vs03:/# exit
logout

Test the ssh login:

xp2400:/# ssh root@192.168.0.3
The authenticity of host '192.168.0.3 (192.168.0.3)' can't be established.
RSA key fingerprint is e2:19:d1:8d:7f:ac:0a:ce:12:f5:06:b7:ae:41:96:59.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.0.3' (RSA) to the list of known hosts.
Password: <secret>
vs03:~#

As another service Apache2 should be used:

vs03:/# apt-get install apache2
[...]
Starting web server: Apache2(98)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
Unable to open logs
invoke-rc.d: initscript apache2, action "start" failed.

Setting up apache2 (2.0.54-4) ...

To avoid the above error:

vs03:/# nano /etc/apache2/ports.conf
vs03:/# cat /etc/apache2/ports.conf
Listen 192.168.0.3:80
vs03:/# /etc/init.d/apache2 start
Starting web server: Apache2(98)Address already in use: make_sock: could not bind to address 192.168.0.3:80
no listening sockets available, shutting down
Unable to open logs

Do we have an Apache listening on the vserver IP in the host context?

xp2400:/# cat /etc/apache2/ports.conf
Listen 80
xp2400:/# nano /etc/apache2/ports.conf
xp2400:/# cat /etc/apache2/ports.conf
Listen 192.168.0.21:80
Listen 127.0.0.1:80
xp2400:/# /etc/init.d/apache2 restart
Forcing reload of web server: Apache2.

vs03:/# /etc/init.d/apache2 start
Starting web server: Apache2.

xp2400:/# lynx http://192.168.0.3/
  Test Page for Apache Installation

  If you can see this, it means that the installation of the Apache web
  server software on this system was successful. You may now add content
  to this directory and replace this page.
[...]

The configuration on one node is ready.

To be continued ...