Difference between revisions of "Upstart issues"

From Linux-VServer

Jump to: navigation, search
(running an upstart based guest)
 
(18 intermediate revisions by 12 users not shown)
Line 1: Line 1:
== running an upstart based guest ==
+
== Running Ubuntu Lucid ==
  
currently upstart based guests need some manual tweaking to make them work inside a vserver guest.
+
There's an inipost-file (http://capa-support.mikrov.dk/downloads/initpost) that can be used when creating hosts, so there's no need for manual tweaking.
 +
Put it in <code>/usr/lib/util-vserver/distributions/lucid</code> and make it executable. Install the vserver with <code>--initstyle plain</code>.
  
I made a [initpost-file|http://paste.linux-vserver.org/pastebin.php?dl=16020] to be used when creating hosts, so there's no need for manual tweaking. Put it in /usr/lib/util-vserver/distributions/lucid (this is not tested on anything else)
+
You will still need to look over services when installing since we're tweaking Upstart's normal <code>emit</code> directives.
 +
 
 +
You can also have a look at http://wiki.raceme.org/dokuwiki/tof:computer:vserver:upstart_ubuntu_lucid for a manual procedure.
 +
 
 +
== Running an upstart based guest ==
 +
 
 +
Currently upstart based guests need some manual tweaking to make them work inside a vserver host.
  
 
=== Prepare the host ===
 
=== Prepare the host ===
Line 9: Line 16:
 
  # echo plain > /etc/vservers/<vservername>/apps/init/style
 
  # echo plain > /etc/vservers/<vservername>/apps/init/style
  
Or use '--initstyle plain' when creating the host.
+
Or use <code>--initstyle plain</code> when creating the host.
  
 
I'd also strongly recommend to mount <code>/var/run</code> and <code>/var/lock</code> of the guest as tmpfs. So add the following lines to
 
I'd also strongly recommend to mount <code>/var/run</code> and <code>/var/lock</code> of the guest as tmpfs. So add the following lines to
Line 16: Line 23:
 
  none    /var/lock      tmpfs  size=16m,noexec,nosuid,nodev    0 0
 
  none    /var/lock      tmpfs  size=16m,noexec,nosuid,nodev    0 0
  
=== remove low level services inside the guest ===
+
=== Remove low level services inside the guest ===
Then you need to get rid of anything in <code>/etc/init/</code> that tries to fiddle with your Server's Hardware or other low-level Stuff that's already handled by your host's kernel/system. Having one such scriptlet still enabled might cause the whole upstart system to fail (like not beeing able to open <code>/dev/console</code> or failing to start the udev system).
+
Then you need to get rid of anything in <code>/etc/init/</code> that tries to fiddle with your Server's Hardware or other low-level Stuff that's already handled by your host's kernel/system. Having one such scriptlet still enabled might cause the whole upstart system to fail (like not being able to open <code>/dev/console</code> or failing to start the udev system).
 
Just remove these scriptlets, move them out of the way or rename them to something not ending in <code>.conf</code>. Be aware that upstart also parses subfolders of <code>/etc/init/</code> for <code>.conf</code> files.
 
Just remove these scriptlets, move them out of the way or rename them to something not ending in <code>.conf</code>. Be aware that upstart also parses subfolders of <code>/etc/init/</code> for <code>.conf</code> files.
  
Line 40: Line 47:
 
  tty6.conf
 
  tty6.conf
 
  upstart-udev-bridge.conf
 
  upstart-udev-bridge.conf
 +
acpid.conf
  
 
In Ubuntu 10.04 you also want to remove:
 
In Ubuntu 10.04 you also want to remove:
Line 45: Line 53:
  
 
=== Services you want to look over ===
 
=== Services you want to look over ===
There are files in the /etc/init dir that you want to check the startup line of.
+
There are files in the <code>/etc/init</code> dir that you want to check the startup line of.
It could be waiting for something that will never happend since it is in a VServer and thus never start.
+
It could be waiting for something that will never happen in a VServer, and thus never start.
  
for example mysql.conf needs a 'net-device-up'-emit also.
+
For example mysql.conf needs a 'net-device-up'-emit also.
by running
+
vserver <name> exec initctl emit net-device-up
+
  
 +
By running
 +
vserver <name> exec initctl emit net-device-up
 
it will come up like a charm.
 
it will come up like a charm.
 
Either put this in vserver.conf or remove in from mysql.conf.
 
Either put this in vserver.conf or remove in from mysql.conf.
  
Jobs that doesn't start from scratch:
+
Jobs that don't start from scratch:
 
  mysql-server-5.1 (mysql.conf)
 
  mysql-server-5.1 (mysql.conf)
  
Line 65: Line 73:
 
         chown root:utmp /var/run/utmp
 
         chown root:utmp /var/run/utmp
 
         chmod 664 /var/run/utmp
 
         chmod 664 /var/run/utmp
         initctl emit virtual-filesystems
+
         initctl emit --no-wait virtual-filesystems
         initctl emit local-filesystems
+
         initctl emit --no-wait local-filesystems
         initctl emit remote-filesystems
+
         initctl emit --no-wait remote-filesystems
         initctl emit filesystem
+
         initctl emit --no-wait filesystem
 +
        initctl emit --no-wait static-network-up
 +
        initctl emit --no-wait net-device-up
 
  end script
 
  end script
  
The utmp stuff is needed if you mount <code>/var/run</code> as tmpfs and you don't create utmp anywhere else.
+
The <code>utmp</code> stuff is needed if you mount <code>/var/run</code> as <code>tmpfs</code> and you don't create <code>utmp</code> anywhere else.
  
 
=== Remove console outputs ===
 
=== Remove console outputs ===
Line 92: Line 102:
  
 
which will disable all lines with 'console output', and Upstart won't try output to it. Which is a good thing since it doesn't exist.
 
which will disable all lines with 'console output', and Upstart won't try output to it. Which is a good thing since it doesn't exist.
 +
 +
Another reason might be that you removed control-alt-delete.conf from /etc/init/ - which is used for a proper shutdown of the vserver:
 +
20:00 <@Bertl> util-vserver does not execute stuff inside the guest, it simply sends a signal to init (inside the guest)
 +
20:01 <@Bertl> (the singnal is SIGINT, and this equals to the CTRL-ALT-DEL action)
  
 
=== Adapt /etc/init/rc-sysinit.conf inside the guest ===
 
=== Adapt /etc/init/rc-sysinit.conf inside the guest ===
  
This depends on net-device-up IFACE=lo, remove this from the start line so that it reads:<br/>
+
This depends on net-device-up IFACE=lo, remove this from the start line so that it reads:
<code>start on filesystem</code>
+
start on filesystem
 +
 
 +
This scriptlet tries to parse <code>/proc/cmdline</code> which will not work and probably isn't needed inside a vserver.
  
This scriptlet tries to parse <code>/proc/cmdline</code> which will not work and probably isn't needed inside a vserver.<br/>
 
 
Simply comment out or delete the whole <code>for ARG in $(cat /proc/cmdline) do...done</code> loop.
 
Simply comment out or delete the whole <code>for ARG in $(cat /proc/cmdline) do...done</code> loop.
  
 
=== Notes for debugging ===
 
=== Notes for debugging ===
  
First you need to run Upstart with the --debug flag,
+
First, you need to run Upstart with the <code>--debug</code> flag;
one way of doing it is inserting
+
one way of doing it is inserting <code>--debug</code> in <code>INITCMD_START</code> under the xplain-clause inside <code>/usr/lib/util-vserver/vserver.functions</code> (row 316 here)
"--debug"
+
so it reads
in INITCMD_START under the xplain-clause inside /usr/lib/util-vserver/vserver.functions (row 316 here)
+
INITCMD_START=( /sbin/init --debug )
so it reads INITCMD_START=( /sbin/init "--debug" )
+
  
Second you should enable console output in the rc-files inside /etc/init, if you had commented them out before you could do
+
Second, you should enable console output in the rc-files inside /etc/init. If you had commented them out before, you could do
 
  sed -i 's/#^\(console output\)$/\1/1' /etc/vservers/$name/vdir/etc/init/*
 
  sed -i 's/#^\(console output\)$/\1/1' /etc/vservers/$name/vdir/etc/init/*
  
Line 115: Line 129:
 
  cp -a /dev/tty0 /etc/vservers/$name/vdir/dev/console
 
  cp -a /dev/tty0 /etc/vservers/$name/vdir/dev/console
  
If you do things at a console of the machine it will now print out detailed information. More on how to debug a Upstart job you'll find on http://upstart.ubuntu.com/wiki/Debugging.
+
If you do things at a console of the machine it will now print out detailed information. More on how to debug an Upstart job you'll find on http://upstart.ubuntu.com/wiki/Debugging.
 
Notice that if one fails they all fail.
 
Notice that if one fails they all fail.
  
Line 132: Line 146:
 
then you're probably running the current stable version <code>2.6.22.19-vs2.2.0.7</code>.
 
then you're probably running the current stable version <code>2.6.22.19-vs2.2.0.7</code>.
  
Upstart's <code>/sbin/init</code> implementation requires to be PID 1. It is linked against NPTL (in contrast to sysvinit), which has a bug in it's caching <code>getpid(2)</code> wrapper. With this kernel version <code>getpid(2)</code> returns the cached PID to init, resulting in init replacing itself with telinit. You can fix this by upgrading to a <code>2.6.26+</code> kernel and a newer vserver patch, or by patching upstart:
+
Upstart's <code>/sbin/init</code> implementation requires it to be PID 1. It is linked against NPTL (in contrast to sysvinit), which has a bug in its caching <code>getpid(2)</code> wrapper. With this kernel version <code>getpid(2)</code> returns the cached PID to init, resulting in init replacing itself with telinit. You can fix this by upgrading to a <code>2.6.26+</code> kernel and a newer vserver patch, or by patching upstart:
  
 
  diff --git a/init/main.c b/init/main.c
 
  diff --git a/init/main.c b/init/main.c
Line 158: Line 172:
  
 
=== Init procedure doesn't complete ===
 
=== Init procedure doesn't complete ===
If you've completed all above steps and some init scripts doesn't get invoked at all (most notably old SYSV init scripts) you might have encountered same error that I did.
+
==== SSH ====
In Lucid guest I had to comment out following line in /etc/init/ssh.conf:
+
If you've completed all above steps and some init scripts doesn't get invoked at all (most notably old SYSV init scripts) you might have encountered the same error that I did.
  # replaces SSHD_OOM_ADJUST in /etc/default/ssh
+
In Lucid guest I had to comment out (or remove) the <code>oom never</code> line in <code>/etc/init/ssh.conf</code>:
  oom never
+
to
+
 
   # replaces SSHD_OOM_ADJUST in /etc/default/ssh
 
   # replaces SSHD_OOM_ADJUST in /etc/default/ssh
 
   #oom never
 
   #oom never
 
  
 
After commenting out this line initialization completed successfully.
 
After commenting out this line initialization completed successfully.
 +
 +
==== D-BUS ====
 +
When creating guest as chroot some packages you might expect are missing. One of these is D-BUS. Strangely enough the /etc/init/dbus.conf gets installed which prevents Upstart to finish its starting sequence.
 +
 +
Solution: install dbus package
 +
 +
    sudo aptitude install dbus
 +
 +
[[Category:Documentation]]

Latest revision as of 11:29, 20 April 2015

Contents

[edit] Running Ubuntu Lucid

There's an inipost-file (http://capa-support.mikrov.dk/downloads/initpost) that can be used when creating hosts, so there's no need for manual tweaking. Put it in /usr/lib/util-vserver/distributions/lucid and make it executable. Install the vserver with --initstyle plain.

You will still need to look over services when installing since we're tweaking Upstart's normal emit directives.

You can also have a look at http://wiki.raceme.org/dokuwiki/tof:computer:vserver:upstart_ubuntu_lucid for a manual procedure.

[edit] Running an upstart based guest

Currently upstart based guests need some manual tweaking to make them work inside a vserver host.

[edit] Prepare the host

First of all you need to set the init style of your vserver to plain

# echo plain > /etc/vservers/<vservername>/apps/init/style

Or use --initstyle plain when creating the host.

I'd also strongly recommend to mount /var/run and /var/lock of the guest as tmpfs. So add the following lines to /etc/vservers/<vservername>/fstab

none    /var/run        tmpfs   size=16m,nosuid,mode=0755       0 0
none    /var/lock       tmpfs   size=16m,noexec,nosuid,nodev    0 0

[edit] Remove low level services inside the guest

Then you need to get rid of anything in /etc/init/ that tries to fiddle with your Server's Hardware or other low-level Stuff that's already handled by your host's kernel/system. Having one such scriptlet still enabled might cause the whole upstart system to fail (like not being able to open /dev/console or failing to start the udev system). Just remove these scriptlets, move them out of the way or rename them to something not ending in .conf. Be aware that upstart also parses subfolders of /etc/init/ for .conf files.

This is what I deleted on a basic Ubuntu 9.10 guest:

dmesg.conf
hostname.conf
hwclock-save.conf
hwclock.conf
mountall-net.conf
mountall-reboot.conf
mountall-shell.conf
mountall.conf
network-interface.conf
networking.conf
procps.conf
rsyslog-kmsg.conf
tty1.conf
tty2.conf
tty3.conf
tty4.conf
tty5.conf
tty6.conf
upstart-udev-bridge.conf
acpid.conf

In Ubuntu 10.04 you also want to remove:

console-setup.conf 

[edit] Services you want to look over

There are files in the /etc/init dir that you want to check the startup line of. It could be waiting for something that will never happen in a VServer, and thus never start.

For example mysql.conf needs a 'net-device-up'-emit also.

By running

vserver <name> exec initctl emit net-device-up

it will come up like a charm. Either put this in vserver.conf or remove in from mysql.conf.

Jobs that don't start from scratch:

mysql-server-5.1 (mysql.conf)

[edit] Fake low level events inside the guest

Finally you need to fake emit the events that some of these scripts would do during regular boot. Probably the cleanest approach is to add a scriptlet vserver.conf to /etc/init/ with this content:

start on startup
script
       touch /var/run/utmp
       chown root:utmp /var/run/utmp
       chmod 664 /var/run/utmp
       initctl emit --no-wait virtual-filesystems
       initctl emit --no-wait local-filesystems
       initctl emit --no-wait remote-filesystems
       initctl emit --no-wait filesystem
       initctl emit --no-wait static-network-up
       initctl emit --no-wait net-device-up
end script

The utmp stuff is needed if you mount /var/run as tmpfs and you don't create utmp anywhere else.

[edit] Remove console outputs

If your VServer hangs on the command

vserver <name> stop

and puts out

A timeout occured while waiting for the vserver to finish and it will
be killed by sending a SIGKILL signal. The following process list
might be useful for finding out the reason of this behavior:

----------------------------------------------------------------------
<pid> <context> <name>           ?        Ss     0:00 /sbin/init
<pid> <context> <name>           ?        Ss     0:00 /sbin/init
----------------------------------------------------------------------

you should run

sed -i 's/^\(console output\)$/#\1/1' /etc/vservers/$name/vdir/etc/init/*

which will disable all lines with 'console output', and Upstart won't try output to it. Which is a good thing since it doesn't exist.

Another reason might be that you removed control-alt-delete.conf from /etc/init/ - which is used for a proper shutdown of the vserver: 20:00 <@Bertl> util-vserver does not execute stuff inside the guest, it simply sends a signal to init (inside the guest) 20:01 <@Bertl> (the singnal is SIGINT, and this equals to the CTRL-ALT-DEL action)

[edit] Adapt /etc/init/rc-sysinit.conf inside the guest

This depends on net-device-up IFACE=lo, remove this from the start line so that it reads:

start on filesystem

This scriptlet tries to parse /proc/cmdline which will not work and probably isn't needed inside a vserver.

Simply comment out or delete the whole for ARG in $(cat /proc/cmdline) do...done loop.

[edit] Notes for debugging

First, you need to run Upstart with the --debug flag; one way of doing it is inserting --debug in INITCMD_START under the xplain-clause inside /usr/lib/util-vserver/vserver.functions (row 316 here) so it reads

INITCMD_START=( /sbin/init --debug )

Second, you should enable console output in the rc-files inside /etc/init. If you had commented them out before, you could do

sed -i 's/#^\(console output\)$/\1/1' /etc/vservers/$name/vdir/etc/init/*

Third you need a device to act as console, I used tty0.

cp -a /dev/tty0 /etc/vservers/$name/vdir/dev/console

If you do things at a console of the machine it will now print out detailed information. More on how to debug an Upstart job you'll find on http://upstart.ubuntu.com/wiki/Debugging. Notice that if one fails they all fail.

[edit] Notes for older kernels

If you encounter upstart errors like this:

$ vserver foo start
init: missing runlevel
Try `init --help' for more information.

or

$ vserver foo start
telinit: Failed to connect to socket /com/ubuntu/upstart: Connection refused

then you're probably running the current stable version 2.6.22.19-vs2.2.0.7.

Upstart's /sbin/init implementation requires it to be PID 1. It is linked against NPTL (in contrast to sysvinit), which has a bug in its caching getpid(2) wrapper. With this kernel version getpid(2) returns the cached PID to init, resulting in init replacing itself with telinit. You can fix this by upgrading to a 2.6.26+ kernel and a newer vserver patch, or by patching upstart:

diff --git a/init/main.c b/init/main.c
index bfcd528..c6c9304 100644
--- a/init/main.c
+++ b/init/main.c
@@ -37,6 +37,7 @@
 #include <string.h>
 #include <syslog.h>
 #include <unistd.h>
+#include <syscall.h>
 
 #include <linux/kd.h>
 
@@ -131,7 +132,7 @@ main (int   argc,
 	}
 
 	/* Check we're process #1 */
-	if (getpid () > 1) {
+	if (syscall(SYS_getpid) > 1) {
 		execv (TELINIT, argv);
 		/* Ignore failure, probably just that telinit doesn't exist */

For more information see the man pages of getpid(2) (NOTES) and clone(2) (BUGS).

[edit] Init procedure doesn't complete

[edit] SSH

If you've completed all above steps and some init scripts doesn't get invoked at all (most notably old SYSV init scripts) you might have encountered the same error that I did. In Lucid guest I had to comment out (or remove) the oom never line in /etc/init/ssh.conf:

 # replaces SSHD_OOM_ADJUST in /etc/default/ssh
 #oom never

After commenting out this line initialization completed successfully.

[edit] D-BUS

When creating guest as chroot some packages you might expect are missing. One of these is D-BUS. Strangely enough the /etc/init/dbus.conf gets installed which prevents Upstart to finish its starting sequence.

Solution: install dbus package

   sudo aptitude install dbus
Personal tools