Fuse GlusterFS

GlusterFS is a userspace filesystem which can create network mirrors/stripes etc. While I have not fully investigated all the functionality of GlusterFS, this article will get you started fast and fill in the blanks. Doing this process taught me a little something about Linux VServer and its kernel interactions as well. This process is very short in terms of compile but the steps were spread out all over the place.

References
GlusterFS 10 minute Setup Doc | Fuse on OpenVZ | Fuse in VServer

ServerSystem

i686 server
CentOS 5
Kernel 2.6.22.19-vs2.3.0.34.1
internal IP 10.10.10.1
+Nat

Guest1

test01 10.10.10.2 pkgmgmt internalize
test02 10.10.10.3 pkgmgmt internalize

Stated Goal

Serve a gluster filesystem from test01 to test02

Fuse Patched

on server first you need the kernel source matching your kernel

yum install kernel-devel

mkdir /usr/local/src/fuse
cd /usr/local/src/fuse
wget http://ftp.zresearch.com/pub/gluster/glusterfs/fuse/fuse-2.7.3glfs10.tar.gz
rpmbuild -ta fuse-2.7.2glfs9.tar.gz
rpm -Uvh /usr/src/redhat/RPMS/$(uname -m)/fuse-*rpm
rmmod fuse
depmod -a
modprobe fuse

Copy the device into the guests

This is trick one. guest systems share the kernel so you are not going to be able to install any kernel modules inside the guest.

cp -a /dev/fuse /etc/vservers/test01/vdir/dev
cp -a /dev/fuse /etc/vservers/test02/vdir/dev

Create fuse group and ensure proper permissions

This is the GID that the Cent RPM will use for fuse

vserver test01 enter
groupadd -g 102 fuse
chmod g+rw /dev/fuse
chgrp fuse /dev/fuse

Compile fuse WITHOUT kernel module

This is trick two. Being that the guest cant load a kernel module it can not install one either. This is why I am guessing yum list fuse will not list any packages inside the guest. Also I had trouble with the RPM build technique inside the guest, this time i elected to use the ./configure method

mkdir /usr/local/src/fuse
cd /usr/local/src/fuse
wget http://ftp.zresearch.com/pub/gluster/glusterfs/fuse/fuse-2.7.3glfs10.tar.gz
tar -xzf fuse-2.7.3glfs10.tar.gz
cd fuse-2.7.3glfs10
./configure && make && make install
exit

Adding capabilities

The guest has to have special capabilities in order to mount file systems. INSERT TOKEN SECURITY LESSON HERE :)

/etc/vservers/test01/ccapabilities
SECURE_MOUNT
SECURE_REMOUNT
BINARY_MOUNT 

/etc/vservers/test01/bcapabilities
SYS_ADMIN 

Restart both vservers

vserver test01 stop
vserver test01 start

vserver test02 stop
vserver test02 start

Install GlusterFS

mkdir /usr/local/src/gluster
cd /usr/local/src/gluster
wget http://ftp.zresearch.com/pub/gluster/glusterfs/1.3/glusterfs-1.3.9.tar.gz
tar -xzf glusterfs-1.3.8.tar.gz
cd glusterfs-1.3.8
./configure 
make && make install

Now you can configure and start gluster. This is a simple NFS like configuration.

Client machine: [10.10.10.2]

$ vi /etc/glusterfs/glusterfs-server.vol
$ cat /etc/glusterfs/glusterfs-server.vol
volume brick
  type storage/posix
  option directory /tmp/export
end-volume

volume server
  type protocol/server
  subvolumes brick
  option transport-type tcp/server # For TCP/IP transport
  option auth.ip.brick.allow *
end-volume

$ glusterfsd -f /etc/glusterfs/glusterfs-server.vol

Client machine: [10.10.10.3]

$ mkdir /mnt/glusterfs
$ vi /etc/glusterfs/glusterfs-client.vol
$ cat /etc/glusterfs/glusterfs-client.vol
volume client
  type protocol/client
  option transport-type tcp/client
  option remote-host 10.10.10.2
  option remote-subvolume brick
end-volume

$ glusterfs -f /etc/glusterfs/glusterfs-client.vol /mnt/glusterfs

Now you have it network mount between two vserver guests.