debugging upstart failure

[Michal Suchanek]

On 17 December 2014 at 15:50, Michal Suchanek <hramrach at gmail.com> wrote:
> Hello,
>
> On 15 December 2014 at 00:09, Dimitri John Ledkov <xnox at ubuntu.com> wrote:
>> Hello,
>>
>> On 12 December 2014 at 16:47, Michal Suchanek <hramrach at gmail.com> wrote:
>>> Hello,
>>>
>>> I bootstrapped Debian wheezy chroot, installed upstart, cloned it to
>>> half dozen media, added kernel and bootloader to each, put it to
>>> machines for which these kernels are configured, and on two of them
>>> upstart does not boot the system.
>>>
>>> I can boot with init=/bin/bash
>>>
>>> /var/log/upstart is empty
>>>
>>> verbose upstart logs that mountall-net finished on the console init:
>>> startpar-bridge (mountall-net--stopped) state changed from post-stop
>>> to waiting
>>>
>>> network is brought up but sshd does not run
>>>
>>> getty on console does not run (not sure configuration is correct - had
>>> to hack it myself)
>>>
>>> upgrading to jessie does not solve the problem
>>>
>>> installing sysvinit makes system bootable
>>>
>>> on none of the machines plymouth works. It is somehow half-broken on Debian.
>>>
>>> These two broken machines have very different kernel sources so I am
>>> wondering if I am missing a kernel feature,
>>>
>>> What kernel features are required to run upstart? eg. udev package in
>>> Debian has a list of kernel symbols it checks and I had to add kernel
>>> feature to install jessie udev on the working machines.
>>
>>
>> In terms of kernel config, upstart should be resilient to most things
>> and thus fairly portable.
>>
>> The best way to debug is to change tty1 job to "start on startup" and
>> boot with --verbose flag.
>
> Indeed, changing ttyS0 to start on startup allows me to log in on the console.
> Remounting / readwrite by hand magically makes the boot process finish.
>
> Adding --verbose flag is recommended by the documentation but there is
> no obvious way to  interpret the output:
> http://paste.ubuntu.com/9551122/
>
> On both working and broken system fstab is present but empty.
>
> This is definitely not ideal.
>
> However
>
> 1) it is not the cause as the working system also has empty fstab

So the cause is that the default for mounting root filesystem changed
from readwrite to readonly between kernel versions. Upstart waits for
/ becoming writable so either mounting it rw to start with or
remouting it by hand from console works.

> 2) if it was the cause there should be an error reported so the user
> can diagnose the failure

Waiting indefinitely for / becoming writable should definitely trigger
an error. It may be hard to tell in the upstart job system in general
that nothing is being done for the filesystem to become accessible
since some jobs are probably running and even if those jobs are
permanent services rather than one time jobs there is no saying what
the service is doing in the background.

That said, the default job set has a specific script for mounting the
/ and if mounting / there does not happen it can print a warning.
Unless the user supplied a different job for mounting a root
filesystem boot has just failed.

> 3) sysvinit can also boot with empty fstab - it does not really
> contain any precious information

Indeed, the / filesystem is already mounted so all that is required is
to change it to readwrite. If init can handle that upstart should be
able to do the same.

Thanks

Michal