Editing
Running runit-supervised services inside a vserver
From Linix VServer
Jump to navigation
Jump to search
Warning:
You are not logged in. Your IP address will be publicly visible if you make any edits. If you
log in
or
create an account
, your edits will be attributed to your username, along with other benefits.
Anti-spam check. Do
not
fill this in!
This page describes a setup where you have a [http://smarden.org/runit runit] installation on the host and use it to directly supervise services running in vserver guests. == Motivation and goals == This is what I wanted to achieve: * Partition a physical server with many responsibilities into vservers that can easily be upgraded individually without breaking any unrelated stuff. * Each service or small set of services should run in its own vserver. * Services should be supervised (started, stopped and managed) by [http://smarden.org/runit runit]. * Service logs should accumulate on the host, not the guests. ** <tt>svlogd</tt> rotates them nicely and can invoke my <tt>multilogcheck</tt> script to alert me of unusual events as a postprocessor, but ** I don't want to install <tt>multilogcheck</tt> inside every vserver because it's messy. ** I don't like syslog. There are many problems with it, but I won't go into that here. *** Therefore, services mostly log to stdout and thus to svlogd. *** For services that absolutely must use syslog, I provide [http://smarden.org/socklog socklog]. *** Again, I don't want a separate <tt>socklog</tt> instance in every vserver. *** Alas, <tt>socklog</tt> is unable to listen on more than one Unix domain socket. **** Luckily, <tt>socat</tt> can be used to relay syslog messages from the vservers to the master <tt>socklog</tt> on the host. (<tt>syslog-ng</tt> would also have done the job nicely.) **** It's also possible to bind mount the <tt>/dev/log</tt> socket in every guest; however, I'm afraid this breaks if you restart socklog after a guest is up. Workaround: make <tt>/dev/log</tt> a symlink to, say, <tt>/var/run/syslog/socket</tt> everywhere and bind mount the host's <tt>/var/run/syslog</tt> directory in the guests; that way the socket will still be available even if socklog unlinks and recreates it. * It should be straightforward and next to transparent to manage the services running in vservers. ** With <tt>runit</tt>, it's easy to delegate management rights of a service to users (chown and chmod some files in the pertinent <tt>supervise</tt> directory). This should continue to work. == Prerequisites == # A fairly recent vserver kernel with support for persistent contexts (I used 2.6.19.2-vs2.2.0-rc8.7). # A recent version of util-vserver that supports persistent contexts correctly (I used [http://people.linux-vserver.org/~dhozac/p/uv/experimental/util-vserver-0.30.213-rc1.tar.bz2 0.30.213-rc1]). # Daniel_Hozac's [http://people.linux-vserver.org/~dhozac/t/signal-relay.c signal-relay] program. == The big picture == Let's see how it all fits together. We'd like to achieve something like this (output of <tt>vps axfu</tt>): <pre> USER PID CONTEXT %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0 MAIN 0.0 0.0 104 16 ? Ss Jan29 0:06 runit root 4188 0 MAIN 0.0 0.0 132 32 ? Ss Jan29 0:10 runsvdir -P /var/service log: ....................................................... root 4250 0 MAIN 0.0 0.0 108 28 ? Ss Jan29 0:00 \_ runsv vserver-logrelay-squid log 4256 0 MAIN 0.0 0.0 128 44 ? S Jan29 0:00 | \_ svlogd -t /var/log/sv/vserver-logrelay-squid logrelay 3106 0 MAIN 0.0 0.1 25428 1944 ? S 00:44 0:00 | \_ socat -d -d -d -D -ls -u UNIX-LISTEN:/etc/vservers/squid/vdir/dev/log,unlink-early,nonblock,mode=666,setuid=logrelay UNIX-CONNECT:/dev/log,type=2,nonblock,forever root 19272 0 MAIN 0.0 0.0 108 28 ? Ss Feb01 0:00 \_ runsv squid log 19273 0 MAIN 0.0 0.0 128 44 ? S Feb01 0:00 | \_ svlogd -t /var/log/sv/squid root 10324 0 MAIN 0.0 0.0 5728 364 ? S Feb01 0:00 | \_ signal-relay vserver squid exec squid -N -D -sYC proxy 10334 2 squid 0.0 0.5 24968 8036 ? Sl Feb01 0:01 | \_ squid -N -D -sYC root 24219 0 MAIN 0.0 0.0 108 32 ? Ss 02:02 0:00 \_ runsv nmbd root 24220 0 MAIN 0.0 0.0 5732 376 ? S 02:02 0:00 | \_ signal-relay vserver samba exec /usr/sbin/nmbd -F root 24230 3 samba 0.0 0.1 30080 2360 ? Ss 02:02 0:00 | \_ /usr/sbin/nmbd -F root 24251 3 samba 0.0 0.0 32172 1496 ? S 02:02 0:00 | \_ /usr/sbin/nmbd -F root 24253 0 MAIN 0.0 0.0 108 28 ? Ss 02:02 0:00 \_ runsv smbd root 24254 0 MAIN 0.0 0.0 5732 376 ? S 02:02 0:00 | \_ signal-relay vserver samba exec /usr/sbin/smbd -F root 24268 3 samba 0.0 0.2 41888 3404 ? Ss 02:02 0:00 | \_ /usr/sbin/smbd -F root 24289 3 samba 0.0 0.0 41888 1232 ? S 02:02 0:00 | \_ /usr/sbin/smbd -F root 29625 0 MAIN 0.0 0.0 104 28 ? Ss 02:22 0:00 \_ runsv cron-squid root 29626 0 MAIN 0.0 0.0 5732 376 ? S 02:22 0:00 \_ signal-relay vserver squid exec cron -f root 29637 2 squid 0.0 0.0 11492 1060 ? S 02:22 0:00 \_ cron -f </pre> For service supervision to work, we must be able to send signals to our services. Specifically, <tt>runsv</tt> must be able to send signals to its children. Alas, it's not prepared to send signals across context boundaries, which is where <tt>signal-relay</tt> comes in. === <tt>signal-relay</tt> === <tt>signal-relay</tt> is a small program not unlike <tt>runit</tt>'s <tt>chpst</tt> that does the following: * it forks a child; ** inside the child, it execs the program specified on its command line; * in the parent, it sets up signal handlers for every signal that relay the signal to the child, even if the child is running in a different context; * if the child exits, <tt>signal-relay</tt> exits. * (It can also put the child into its own process group; use the <tt>-P</tt> switch. Sending signals to process groups in a different context doesn't work yet, though.) === Setting up the vservers === When we start a service for runit, we want the command that starts the service to stay in the foreground until the moment the service dies. <tt>vserver exec</tt> looks just right, but there is a catch: it only works for vservers that have been "started". <tt>vserver start</tt>, however, doesn't fit very well into the <tt>runit</tt> way of doing things. You can set it up as a service (this is discussed in [[util-vserver:InitStyles]]), but it seems superfluous to leave some processes around just to keep a vserver "started" so that we can run services in it. What we need is a way of basically doing "start vserver <guest> if it's not started, then exec program <service> inside it". Normally, a context with no processes running inside it is destroyed by the kernel; thus, just setting <tt>/etc/vservers/guest/apps/init/cmd.start</tt> to <tt>/bin/true</tt> isn't going to work; we would need something that stays around for a while, like a script that calls <tt>sleep 1m &</tt>. This would make <tt>vserver start</tt> happy, so in our service run script, we could do something like <pre> vserver guest status || vserver guest start exec signal-relay vserver guest exec /path/to/service-program </pre> A more elegant solution is to make the guest context ''persistent''. This way it sticks around even if there are no processes left in it. <pre> cd /etc/vservers/guest echo persistent >>flags echo persistent >>nflags echo /bin/true >apps/init/cmd.start </pre> (I ''guess'' <tt>nflags</tt> is for the network context.) Now, <tt>vserver guest start</tt> should be able to "start" our vserver (set up all of its state), and exit without leaving stray processes around. == Running services == Say you have a vserver called <tt>squid</tt>, and you want to run your <tt>squid</tt> proxy inside it to keep it insulated from the other processes on the system, and vice versa. In our current setup, the following run script will do the trick (it borrows a bit from Debian's initscript and uses the Debian defaults file): <pre> #!/bin/sh exec 2>&1 # Set a default for max. filedescriptors SQUID_MAXFD=4096 # Figure out the name of the service we're running as SVNAME=$(basename $(pwd)) # Read the configfile of this service; if it sets VSERVERNAME, we'll assume we have to run inside the specified vserver CONFIG=/etc/default/"$SVNAME" [ -r "$CONFIG" ] && . "$CONFIG" # Source Debian defaults [ -n "$VSERVERNAME" ] && VROOT="/etc/vservers/$VSERVERNAME/vdir" || VROOT=/ [ -r "$VROOT/etc/default/squid" ] && . "$VROOT/etc/default/squid" [ "$SQUID_MAXFD" -gt 4096 ] && SQUID_MAXFD=4096 if test -f /proc/sys/fs/file-max; then global_file_max=$(cat /proc/sys/fs/file-max) minimal_file_max=$(($SQUID_MAXFD + 4096)) [ "$global_file_max" -lt "$minimal_file_max" ] && echo "$minimal_file_max" >/proc/sys/fs/file-max fi if [ ! "$VSERVERNAME" = "" ]; then VSERVERARGS="signal-relay vserver $VSERVERNAME exec" vserver "$VSERVERNAME" status || vserver "$VSERVERNAME" start # start the vserver if necessary fi exec chpst -o"$SQUID_MAXFD" $VSERVERARGS squid -N -D -sYC </pre> This script is generic in the sense that it works with or without a vserver setup. You can run an <tt>svlogd</tt> for this service like you normally would; or you could have all your <tt>svlogd</tt>s run in a different vserver. A run script for <tt>cron</tt> that automatically figures out if it should run in a vserver: <pre> #!/bin/sh exec 2>&1 DEPENDENCIES="" RLIMIT="chpst -t 172800" SVNAME=$(basename $(pwd)) CONFIG=/etc/default/"$SVNAME" # Assume part after dash is name of vserver if [ ! "${SVNAME/*-/}" = "$SVNAME" ]; then VSERVERNAME="${SVNAME/*-/}" fi # By default, we depend on the socklog service if it exists [ -e /service/socklog ] && DEPENDENCIES=socklog # And also on the logrelay service of the pertinent vserver [ -e /service/vserver-logrelay-"$VSERVERNAME" ] && DEPENDENCIES="$DEPENDENCIES vserver-logrelay-$VSERVERNAME" # Can override VSERVERNAME, RLIMIT and DEPENDENCIES here if necessary [ -r "$CONFIG" ] && . "$CONFIG" if [ ! "$VSERVERNAME" = "" ]; then VSERVERARGS="signal-relay vserver $VSERVERNAME exec" vserver "$VSERVERNAME" status || vserver "$VSERVERNAME" start fi if [ -n "$DEPENDENCIES" ]; then sv start $DEPENDENCIES || exit 1 fi exec $RLIMIT $VSERVERARGS cron -f </pre> Now, if you place this run script in a service directory called <tt>cron-squid</tt>, it will run a cron daemon in the <tt>squid</tt> guest. This takes care of rotating the squid logs under <tt>/var/log/squid</tt>, for example; although this could also be done on the host with a few kludges. The vserver-logrelay-template/run script looks like this: <pre> #!/bin/sh exec 2>&1 SVNAME=$(basename $(pwd)) RUNASUSER=logrelay MODE=666 VSERVERNAME=$(echo $SVNAME | sed 's/vserver-logrelay-//') [ -r "/etc/default/$SVNAME" ] && . "/etc/default/$SVNAME" exec socat -d -d -d -D -ls -u \ UNIX-LISTEN:/etc/vservers/$VSERVERNAME/vdir/dev/log,unlink-early,nonblock,mode=$MODE,setuid=$RUNASUSER \ UNIX-CONNECT:/dev/log,type=2,nonblock,forever </pre> This will pass syslog messages from a vserver to the syslog of the host by acting as a syslog server for the <tt>/dev/log</tt> socket of the guest. Thus, you don't need to run a <tt>syslogd</tt> inside the vserver and you don't need to migrate to <tt>syslog-ng</tt> from <tt>socklog</tt> on the host. Symlink this run script into a service directory called <tt>vserver-logrelay-squid</tt>. Unfortunately, <tt>socat</tt> has a largish memory footprint; something more lightweight could, perhaps, be used. Comments and additions welcome. == Disadvantages == The most significant problem with this approach (as opposed to using initstyle plain and a separate runit instance in each vserver) is that it's no longer straightforward to manage the services running in vservers from inside the vservers; package management scripts can't stop them for upgrades, for example (unless you modify the initscripts in quite horrible ways). Initial setup is also slightly more complicated. These have to be weighed against the advantages of: * having fewer superfluous processes (like a separate <tt>runit</tt> and <tt>runsvdir</tt> in each vserver); * being able to manage the services easily from the host; * avoiding the need to run <tt>sshd</tt> inside each guest in order to be able to easily delegate service management privileges to others. Nowadays I tend to think the initstyle plain method is better after all. --[[User:KornAndras|Guy-]] 12:18, 28 September 2008 (CET) [[Category:Software under linux-vserver]]
Summary:
Please note that all contributions to Linix VServer may be edited, altered, or removed by other contributors. If you do not want your writing to be edited mercilessly, then do not submit it here.
You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource (see
Linix VServer:Copyrights
for details).
Do not submit copyrighted work without permission!
Cancel
Editing help
(opens in new window)
Navigation menu
Page actions
Page
Discussion
Read
Edit
History
Page actions
Page
Discussion
More
Tools
Personal tools
Not logged in
Talk
Contributions
Create account
Log in
About
Overview
Paper
News
Developers
Donations
Search
Getting Started
Downloads
FAQs
Documentation
Support
Participate
How to participate
Report a Bug
Communicate
Teams/Projects
Hall of Fame
Resources
Archives
Recent Wiki Changes
Pastebin
Related Projects
VServer Hosting
Happy VServer Users
Tools
What links here
Related changes
Special pages
Page information