Frequently Asked Questions

• memory: Dynamically.
• CPU usage: Dynamically (token bucket)

• using ulimits and rlimits (rlimit is a new feature of kernel 2.6/vs2.0.) per guest, to limit the memory consumption, the number of processes or file-handles, ... : see Resource Limits
• CPU usage : see CPU Scheduler
• disk space usage : see Disk Limits and Quota

If you want a recipe, do this:

1. Check the size of memory pages. On x86 and x86_64 is usually 4 KB per page. (on linux "getconf -a

# cat /sys/block/hdc/queue/scheduler
noop [anticipatory] deadline cfq

The default since 2.6.18 in Sept 2006 is CFQ, described below, and prior to that was anticipatory a.k.a. "AS" ([Wikipedia]). When running several guests on a host you probably want the I/O performance shared in a fair way among the different guests. The kernel comes with a "completely fair queueing" scheduler, CFQ, which can do that. (More on schedulers can be found at http://lwn.net/Articles/114770/) This is how to set the scheduler to "cfq" manually:

root# echo "cfq" > /sys/block/hdc/queue/scheduler
root# cat /sys/block/hdc/queue/scheduler
noop anticipatory deadline [cfq]

Keep in mind that you have to do it on all physical discs. So if you run an md-softraid, do it to all physical /dev/hdXYZ discs! If you run Debian there is a predefined way to set the /sys values at boot-time:

# apt-get install sysfsutils
[...]

# grep cfq /etc/sysfs.conf
block/sda/queue/scheduler = cfq
block/sdc/queue/scheduler = cfq

# /etc/init.d/sysfsutils restart

For non-vserver processes and CFQ you can set by which key the kernel decides about the fairness:

cat /sys/block/hdc/queue/iosched/key_type
pgid [tgid] uid gid

Hint: The 'key_type'-feature has been removed in the mainline kernel recently. Don't look for it any longer :(

The default is tgid, which means to share fairly among process groups. Think every guest is treated like a own process group. It's not possible to set a scheduler strategy within a guest. All processes belonging to the same guest are treated like "noop" within the guest. So: If you run apache and some ftp-server within the _same_ guest, there is no fair scheduling between them, but there is fair scheduling between the whole guest and all other guests.

It's split into three groups, called real-time, best effort and idle. The default is best-effort, but within best-effort, you can have a niceness from 0 to and including 7. You can set this niceness by the tool ionice, which for debian is either in the package util-linux or schedutils. To change the io-niceness you need the CAP_SYS_NICE, and need to have the same uid as the processe you want to ionice.

Note: If you want to use any schedulung other than best-effort you will also need the CAP_SYS_ADMIN-flag. Be warned that this gives quite some capabilities to the vserver, not just for I/O scheduling!

If you want to increase the niceness of an I/O hogging process within a vserver you need to do:

chcontext --xid sponlp1 sudo -u '#2089' ionice -c2 -n5 -p24409

It creates hardlinks to files named after a hash of the content of the file. If you have a recent version of the vserver patch (2.2+), with CONFIG_VSERVER_COWBL enabled, you can even modify the hardlinked files inside the vservers and the links will be broken automatically.

mkdir /etc/vservers/.defaults/apps/vunify/hash /vservers/.hash
ln -s /vservers/.hash /etc/vservers/.defaults/apps/vunify/hash/root

Then, do this one line per vserver:

mkdir /etc/vservers/<vservername>/apps/vunify   # vhashify reuses vunify configuration

To hashify a running vserver, do (possibly from a cronjob):

vserver name-of-guest hashify

The guest needs to be running because vhashify tries to figure out what files not to hashify by calling the package manager of the guest via vserver enter.

In order for the OS cache to benefit from the hardlinking, you'll have to restart the vservers.

To clean up hashified files that are no longer referenced by any vserver, do (possibly from a cronjob):

find /vservers/.hash -type f -links 1 -print0 | xargs -0 rm

Filesystem XID tagging only works on supported filesystem. Those are currently: ext2/3, reiserfs/reiser3, xfs and jfs. To activate the XID tagging you have to mount the filesystem with "-o tag" (former tagxid is outdated since VS2.2). Attention: It's _not_ possible to "-o remount,tag", you have to mount it freshly. The guests will tag their files automatiaclly. If you copy files in from the host, you have to tag them manually like this:

chxid -c xid -R /var/lib/vservers/<guest>

Note: Context 0 and 1 will see all files, guests will only be able to access untagged files and their own XID. They can see other XID files but no information about the file, e.g. no owner, no group, no permissions.

vnamespace --enter <xid> -- /bin/bash

mount /dev/reiserfsdev /vservers -oattrs
setattr --barrier /vservers

Update from IRC (2011-08-22):
<mmouse> quick question: what is the maximum count of IPs (v4) I can have in a single guest?
<daniel_hozac> unlimited.

Here is a little helper-script that adds a list of IPs defined in a text file, one per line.

#!/bin/bash
j=1
for i in `cat myiplist`; do
        j=$(($j+1))
        mkdir $j
        echo $i > $j/ip
        echo "24" > $j/prefix
done

• add the ip on the host, for example

ip addr add 194.169.123.23/24 dev eth0 

• add the ip to the guest's network context (a guests NID is the same as the XID {context ID})

naddress --add --nid <nid> --ip 194.169.123.23/24 

• enter the guest (best via ssh)
• restart the services that need to make use of the new address if required
• update the config in /etc/vserver/<servername>/interfaces to reflect the changes for the next guest restart (if desired)

iptables -t nat -I POSTROUTING -s $VSERVER_NETZ  ! -d $VSERVER_NETZ -j SNAT --to $EXT_IP

See: HowtoPrivateNetworking and

This is the minimal OpenVPN configuration for the Server which will be used to demonstrate how to get it running in a client:

# Networking setup
server 192.168.16.0     255.255.255.0
dev tun16
ifconfig-noexec
comp-lzo
# Certificates
dh ...
ca ...
cert ...
key ...
# Management
persist-key
keepalive 10 60
verb 4

First of all you have to prepare the host with a persistent interface in the right mode and with the right settings. This is easily done by using openvpn and the ip and route tools.

# openvpn --mktun --dev tun16
# ip link set dev tun16 txqueuelen 100
# ifconfig tun16 192.168.16.1 pointopoint 192.168.16.2 mtu 1500
# route add -net 192.168.16.0 netmask 255.255.255.0 gw 192.168.16.2

If you need different settings, openvpn will tell you the ifconfig and route commands it uses to configure the interface when being started on the host with the original config file, but without ifconfig-noexec. Additionally, the guest needs /dev/net/tun to make OpenVPN happy. This can be created with MAKEDEV:

# cd /var/lib/vserver/<myopenvpnserver>/dev/
# ./MAKEDEV tun
  (creates the dev/net/tun device accessible by the guest - even a tap interface needs /dev/net/tun !)

Finally, the guest needs to have the tun device assigned:

# head /etc/vservers/<myopenvpnserver>/interfaces/1/*
==> /etc/vservers/<myopenvpnserver>/interfaces/1/ip <==
192.168.16.1

==> /etc/vservers/<myopenvpnserver>/interfaces/1/nodev <==
tun16

==> /etc/vservers/<myopenvpnserver>/interfaces/1/prefix <==
24
#

The client's conf may look like that:

# Basic setup
client
proto tcp-client
dev tun
remote <ipaddress>
comp-lzo
verb 4

# Certificate
ca ...

 
sin_addr=inet_addr("xx.xx.xx.xx")}, yy) = -1 EINVAL (Invalid argument)

A: The host/guest cannot communicate with another guest on same host.

• check all netmasks on all interfaces (do they overlap) ?
• check policy routing (disable it temporary) ?
• check that lo is up (Networking within a host/guest always uses lo interface)

Real answer from _are_:

When I once needed 'real' seperate MAC-addresses I used TAP-devices and VDE2 ([http://vde.sourceforge.net/ Virtual Distributed Ethernet]).
Basically vServer is an isolation of existing resources, not a virtualization of 'new' devices.
Without extra fuss you can't add a 'new' network interface to a vServer, no matter if it is eth* or tap*, you always add it to the host and give the vServer access to it.
I got the TAP+VDE2 up and running, but I think it is too much trouble for basically the simple adding of IPs to a vServer unless you really need the MAC address separate.

You can also utilize MACVLAN ability from kernel. I.e. create macvlan0 interface with:

ip link add link eth0 address 00:19:d1:29:d2:58 macvlan0 type macvlan

It means somehow "optimized" :D If you don't want this you have 3 possible solutions (quoting Bertl):

 * disable the auto single IP in the kernel
 * assign more than one IP to the guest
 * disable single ip special casing for that guest

The later is done by : echo "~single_ip" >> /etc/vservers/<VSERVER>/nflags

CTX   PROC    VSZ    RSS  userTIME   sysTIME    UPTIME NAME
0       77 965.1M 334.6M  14m14s18   2m28s69   1h33m46 root server
49152    7    14M   5.2M   0m00s40   0m00s30   1h30m15 chiffon

• http://Openvcp.org which is a distributed system (agent!) with a web-interface, with which you can build/remove guests
• http://vsmon.revolutionlinux.com/ is a distributed monitoring-only solution that allows you to search for a particular vserver in your park.

vserver vubuntu build --force -m debootstrap --hostname vubuntu.myvservers.net --netdev eth0 --interface 192.168.0.2/24 \
--context 42 -- -d breezy -m http://de.archive.ubuntu.com/ubuntu

[UPDATE] Currently there are problems in building breezy under unclear circumstances, which seems to have to do with udev. If the above didnt work, try:

vserver vubuntu build --force -m debootstrap --hostname vubuntu.myvservers.net --netdev eth0 --interface 192.168.0.2/24 \
--context 42 -- -d breezy -m http://de.archive.ubuntu.com/ubuntu -- --exclude=udev

In very recent versions of the utils, the problem should not occur anymore (it has to do with the 'secure-mount' if you look in the MLs)

Well, sid's debootstrap knows how to bootstrap Ubuntu linux. Make sure to have a current debootstrap package:

apt-get update
apt-get install debootstrap

Port 22
# Use these options to restrict which interfaces/protocols sshd will bind to
#ListenAddress ::

And now change the setting to

Port 22
# Use these options to restrict which interfaces/protocols sshd will bind to
ListenAddress your.hosts.ip.here  # not the guests IP! 

Then '/etc/init.d/ssh restart' on the host, after that on the guest (if you did apt-get install ssh on the guest already.) Do I have to explain more? If the hosts sshd binds all available IP addresses on port 22 (The hosts 'sees' even all addresses of the guests!). So if the guest starts its sshd, it can't bind to port 22 any more. You need to change that setting only on the host.

pam authentication (also used with openssh) enables "pam_loginuid.so" in the /etc/pam.d/* files. Comment those out as they are not necessary and will not load within a guest anyway. This probably is also necessary on updates later on, if the configs get changed. You therefore may add the following command line to a cronjob file or your software update script:

/bin/sed --in-place -e "s/^session.*required.*pam_loginuid.so/# session\trequired\tpam_loginuid.so/g" /etc/pam.d/*

The easiest way to do this from the host (assuming the guest is running) is:

vnamespace -e <xid> -- chroot /vservers/<name> emerge nagios-plugins -va

ntpdate -u ntp.domain.xy

or you can use command:

chbind --nid 42 --ip 1.2.3.4 -- ntpdate ntp.domain.xy

echo "default" > /etc/vservers/derjohn/apps/init/mark

If you want to start it earlier, please read the init script "/etc/init.d/util-vserver" to find out how to do it. In most cases you don't need to change this. On Debian the vservers are started at "20", so after most other stuff is up (networking etc.).

For legacy Linux-VServer (i.e. 1.2.x), you have to replace /sbin/halt in the guests with vreboot and start rebootmgr in the host. You also need to have a <guest>.conf file in /etc/vservers for each guest. Please have a look at /etc/init.d/rebootmgr.

[*] Kernel support for ELF binaries
<M> Kernel support for MISC binaries
[*] IA32 Emulation <---- without that, the entire 32bit API is not present
<M>   IA32 a.out support  

You can force the guest to behave like a 32 environment like this:

echo linux_32bit > /etc/vservers/$NAME/personality
echo i686 > /etc/vservers/$NAME/uts/machine

(thanks cehteh for the hint!)

But you can force debootstrap to put 32 bit binaries into the guest by 'export ARCH=i386';

export ARCH=i386 ; vserver build .... 

On debian when using the newvserver script "export ARCH=i386" has no effect, just use:

newvserver --arch i386 ...

On debian debootstrap can also be gived the arch option:

vserver myguest \
build -m debootstrap -n myguest \
--hostname myguest.mydomain.com \
-- -d squeeze -- \
--arch=amd64  (or i386 if you want 32bit)

>i was wondering about the real thing that guest privacy does. 
#ifdef  CONFIG_VSERVER_PRIVACY
#define VS_ADMIN_P      (0)
#define VS_WATCH_P      (0)
#else

> > Does it just prevent the spectator context ? 
 
it prevents the spectator context and the admin 
functionality in all cases which are privacy
sensitive, which includes:

 - ptrace
 - devmapper
 - devpts
 - inode tag permissions
 - mountinfo
 - kill/signal
 - netlink dumps
 - tun control
 - iopriority

> > What security do it bring to the system ?

together with the VXF_STATE_ADMIN it can be
used to secure a guest (to some degree) from
unwanted access from the host admin, of course,
as the admin can change the kernel, this is a
voluntary feature which mostly prevents certain
kinds of accidential peeking or guest modification
 

• http://linux-vserver.derjohn.de/ - "my kernels are always 'devel' branch" (derjohn). This repo contain kernels up to 2.6.29 for amd64, 2.6.26 for i386.
• http://repo.psand.net/info/ has Debian Lenny and Squeeze packages for 2.6.31 (i368 and amd64), 2.6.33, 2.6.35, 2.6.36 (amd64). It also contains newer util-vserver builds.
• http://backports.org/ contains 2.6.32 backports for Lenny at time of writing (11th May 2010)
• http://zbla.net/debian/ Unofficial debian vserver packages WARNING : i386 packets are compiled for 64bits ! apt source line: deb http://zbla.net/debian/ ./

# vnamespace -e XID  mount -t tmpfs -o remount,size=256m,mode=1777 none /var/lib/vservers/<guest>/tmp/

(if there's a problem, try expanding the symlinks in the mount path) Be warned that the guest will not recognize the change, as the /etc/mtab file is not updated when you mount like this. To permanently change the mount, edit /etc/vserver/<guest>/fstab on the host.

If you get:

mount: can't find /var/lib/vservers/<guest>/tmp in /etc/fstab or /etc/mtab

then try instead:

vnamespace -e builder chroot /var/lib/vservers/<guest>/ mount -o remount,size=64m,mode=1777 /tmp

Note that this not work for adding a bindmount (-o bind) of a directory outside of a vserver into the vserver. For this, there is no alternative but restarting the vserver.

In order to mount a device present on the host to a directory on the guest use something like:

vnamespace -e <guestname> mount -n /dev/<device> /vservers/<guest>/place/you/want/to/mount/it

This device can be unmount thus:

vnamespace -e <guestname> umount -n /vservers/<guest>/place/you/want/to/mount/it

# vnamespace -e XID mount -n -t tmpfs -o remount,size=32m tmpfs /<vdir>/<guest>/tmp

or something like that. The arguments are needed since mount is not going to be using /etc/fstab for the information and the version of /proc/mounts is best understood by

# vnamespace -e XID cat /proc/mounts.

/dev/drbd0     /       xfs     rw,dev          0 0

You will also need to remove guest's ip, for example with:

$ strace nice su nobody
[...]
setpriority(PRIO_PROCESS, 0, 0)         = -1 EACCES (Permission denied)

You can use 'su nobody -c nice some_cmd' instead.

The problem is caused by the fact that host system is setting limits for guests (when instructed to do so) and the dropped capability dissallows processess on guest systems to change and increase them later. That means no process on a guest can lower nice value above the limit set by host.

If the pam_limits module is activated on a guest system it will first try to reset nice value to 0 even if /etc/security/limits.conf file is empty or even if there are lower priority limits set in it. The pam_limits module does that since its developers decided that it should reset some limits to defaults and start from scratch when applying new restrictions. Unfortunately, already limited guest system won't be able to do it since resetting nice value to 0 means increasing the limit which is forbidden. See Debian Bug report logs - #311058 for more information about that Debian bug.

CRON[xxxxx]: Permission denied

Similar thing may happen with the su when trying to execute a command as root.

It's not the problem in Cron but in the pam_limits module on a guest system (see FAQ:When using nice and su… for more information about the cause).

There are 4 ways to solve or work-around this problem:

1. Allowing guests to reset resource limits:

It's clean solution but you may expect some guest processes increasing their limits because not everything is controlled by PAM. It also breaks centralized resources limiting approach so a guest can do bad things that may cause other guests and host to be overloaded and unresponsive.

To apply it you have to add CAP_SYS_RESOURCE flag to /etc/vservers/<server name>/bcapabilities (2.6 kernels).

See Capabilities and Flags for more information.

2. Disabling pam_limits on a guest systems:

This workaround is easy and works fine when your guest systems aren't really multiuser but rather service boxes. It disables setting of the resource limits by PAM so the limits can only be set globally for the whole guest (using rlimits or cgroups on a host) but never increased inside of the guest system.

To apply it just enter the guest and edit the files listed below, commenting out any line containing pam_limits:

/etc/pam.d/su
/etc/pam.d/sudo
/etc/pam.d/cron

You can also use this one-liner on a guest:

/bin/sed --in-place -e "s/^\s\?session.*pam_limits.so.*/\#\0/g" /etc/pam.d/{su,sudo,cron}

3. Allowing guest's pam_limits to set limits when possible:

This workaround allows you to have working pam_limits inside a guest system and global limits set by a host system. What's the catch? The problematic PAM module won't fully work for the root user on a guest system as expected and there might appear some PAM's warnings in guest's auth.log. Since using pam_limits to limit regular user processes is far more frequent than using it to limit root processes, this solution may be a good compromise. It is about setting a proper limits in pam_limits configuration and about setting this PAM module in a way that its function is optional (instead of required). The last change makes PAM to continue with session even if pam_limits encounters some error during setting limits (it usually applies to superuser sessions).

The not so nice news is that there might be a need of keeping guest's limits configuration up to date according to limits globally set for a guest. The limits set in pam_limits configuration file(s) shouldn't be higher (lower in case of nice value) than global guest's limits.

To apply it enter the guest and edit the files listed below, replacing occurences of required by optional in the lines containing pam_limits:

/etc/pam.d/su
/etc/pam.d/sudo
/etc/pam.d/cron

For example:

# Sets up user limits, please define limits for cron tasks
# through /etc/security/limits.conf
session       optional        pam_limits.so

Then on a guest system create the file /etc/security/limits.d/01-fixpam.conf containing:

*         -        priority        X   # replace X with your guest's nice value 

You can automate this process to happen automagically for any guest by creating the startup script named /etc/vservers/.defaults/scripts/post-start.d/01-pamfix:

#!/bin/sh

# This script tries to fix pam_limits entries
# to make it possible for PAM in a guest system to
# set its own limits.

PAM_DIR="etc/pam.d"
PAM_SERVICES="su sudo cron"
LIMITS_FILE="etc/security/limits.d/01-pamfix.conf"

vname="$2"
[ -z "$vname" ] && exit 0

vcfg=$( /usr/sbin/vserver-info "$vname" CFGDIR )
[ ! -d "$vcfg" ] && exit 0

for s in $PAM_SERVICES
do
        pamfile="${PAM_DIR#\/}/$s"
        [ -f "$pamfile" ] && /bin/sed --in-place -e "s/\(^\s\?session.*\)required\(.*pam_limits.so.*\)/\1optional\2/g" "$pamfile"
done

[ ! -f "${vcfg}/nice" ] && exit 0
nval=$( /usr/bin/head -1 "${vcfg}/nice" )
[ -n "$nval" ] && echo "* - priority $nval # (added by vserver startup script)" > "${LIMITS_FILE#\/}"

exit 0

4. Disabling resource limits for a guest:

1) Mount the NFS share from the host OS and let vserver guest access it as part of it's file system.

mount --bind may also be beneficial in this scenario.

2) Use util-vserver and create a fstab.remote file in the /etc/vserver/<vserver_name> directory. Populate this with the NFS shares and they will be mounted in the context of the vserver guest.

See http://www.nongnu.org/util-vserver/doc/conf/configuration.html

3) Add capabilities to the vserver guest instance to grant sufficient rights to allow NFS mounts.

Add the following to /etc/vserver/<vserver_name>/bcapabilities

SYS_ADMIN

Add the following to /etc/vserver/<vserver_name>/ccapabilities

SECURE_MOUNT
BINARY_MOUNT

See Capabilities_and_Flags for more information about vserver capabilities.

vnamespace -e <guest> cat /proc/mounts

If there is no /usr, you can fix your problem with simply mounting it using the following command:

vnamespace -e <guest> mount /dev/<device> /usr

check your install log and it should tell you something about that your server didn't get installed properly

• use stable distribution of debian as server (debootstrap may be different over the versions)
• deny_mount, deny_caps and deny_pivot should be off if your running grsec.

1. Stop the vserver in question
2. rename the /vservers/<server name> directory
3. rename the /etc/vservers/<server name> directory
4. update link: /etc/vservers/<server name>/run → /var/run/vservers/<server name>
5. update link: /etc/vservers/<server name>/vdir → /etc/vservers/.defaults/vdirbase/<server name>
6. update link: /etc/vservers/<server name>/cache → /etc/vservers/.defaults/cachebase/<server name>
7. update link: /var/run/vservers.rev/<server XID> → /etc/vservers/<server name>
8. Start the vserver in question. It should start properly.

Just do a : cat /etc/vservers/<guest>/context > /var/run/vservers/<guest>

echo 101 > /etc/vservers/myvserv/context

# cat /usr/local/sbin/vserver-convert-schedule-to-scheddir
#/bin/sh
mkdir /etc/vservers/$1/sched
sed -e 1p -n /etc/vservers/$1/schedule > /etc/vservers/$1/sched/fill-rate
sed -e 2p -n /etc/vservers/$1/schedule > /etc/vservers/$1/sched/interval
sed -e 3p -n /etc/vservers/$1/schedule > /etc/vservers/$1/sched/tokens
sed -e 4p -n /etc/vservers/$1/schedule > /etc/vservers/$1/sched/tokens-min
sed -e 5p -n /etc/vservers/$1/schedule > /etc/vservers/$1/sched/tokens-max

mv /etc/vservers/$1/schedule /etc/vservers/$1/schedule.converted.see.scheddir

# see: http://oldwiki.linux-vserver.org/Scheduler+Parameters
# see: http://www.nongnu.org/util-vserver/doc/conf/configuration.html#sched

Here is an example how to do so:

# mkdir /etc/vservers/<vserver>/sysctl/0 -p
# echo kernel.shmall > /etc/vservers/<vserver>/sysctl/0/setting
# echo 134217728 > /etc/vservers/<vserver>/sysctl/0/value
# mkdir /etc/vservers/<vserver>/sysctl/1 -p
# echo kernel.shmmax > /etc/vservers/<vserver>/sysctl/1/setting
# echo 134217728 > /etc/vservers/<vserver>/sysctl/1/value

It's also explained on the geat flower page:

1. see: http://www.nongnu.org/util-vserver/doc/conf/configuration.html -> Look for "sysctl".

After changing those values, restart your guest, enter it and check if the values are set:

# sysctl -a | grep shm
...
kernel.shmall = 134217728
kernel.shmmax = 134217728

To change a value for a running guest, on the host use:

 vspace -e CONTEXTID --ipc sysctl -w kernel.shmall=134217728
 vspace -e CONTEXTID --ipc sysctl -w kernel.shmmax=134217728