[Bug 1606472] Re: boot scripts hang in plymouth --ping (aka: Plymouth's bug returns)
Cyrille Bollu
1606472 at bugs.launchpad.net
Tue Oct 25 06:43:33 UTC 2016
So, I installed plymouth-disabler today and rebooted the server.
Though, for some reason (Dell related?) I had to press the power button
to shut my server down, it booted unattended.
So, it seems plymouth-disabler helped in this issue.
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to plymouth in Ubuntu.
https://bugs.launchpad.net/bugs/1606472
Title:
boot scripts hang in plymouth --ping (aka: Plymouth's bug returns)
Status in plymouth package in Ubuntu:
New
Bug description:
Hi,
Summary:
========
My system is not able to boot unattended anymore.
Problem seems similar to
https://bugs.launchpad.net/ubuntu/+source/plymouth/+bug/564360
History:
========
The history of this issue is the following:
1- On 2016.06.14, I "aptitude safe-upgrade" my system. The kernel upgraded from 3.13.0-85-generic to 3.13.0-88-generic. After the upgrade, the system won't boot anymore. After some troubleshooting, suspecting some kernel/low-level issue, I decided to reboot it on kernel 3.13.0-85-generic, and systems boots OK.
2- on 2016.07.26, I "aptitude safe-upgrade" my system. The kernel upgraded from 3.13.0-85-generic to 3.13.0-92-generic. After the upgrade, the system won't finish booting up. Fortunatelly, OpenSSH had started, and I could investigate the issue a little bit further. Hereunder are my findings about this issue.
Findings:
=========
On the console, the last boot message I can see is the following:
* Starting configure virtual network devices [ OK ]
[ 16.192284] bnx2 000:01:00.1 eth1 NIC Copper Link is Down
my /var/log/boot.log file is the following:
root at otrs-prod:/etc/init# cat /var/log/boot.log
* Starting Read required files in advance [ OK ]
* Starting Mount filesystems on boot [ OK ]
* Starting Fix-up sensitive /proc filesystem entries [ OK ]
* Starting Populate /dev filesystem [ OK ]
* Starting Populate and link to /run filesystem [ OK ]
* Stopping Populate /dev filesystem [ OK ]
* Stopping Fix-up sensitive /proc filesystem entries [ OK ]
* Stopping Populate and link to /run filesystem [ OK ]
* Stopping Track if upstart is running in a container [ OK ]
* Starting Signal sysvinit that the rootfs is mounted [ OK ]
* Starting Clean /tmp directory [ OK ]
* Stopping Read required files in advance (for other mountpoints) [ OK ]
* Stopping Read required files in advance (for other mountpoints) [ OK ]
* Stopping Clean /tmp directory [ OK ]
* Starting Initialize or finalize resolvconf [ OK ]
* Starting set console keymap [ OK ]
* Starting Signal sysvinit that virtual filesystems are mounted [ OK ]
* Starting Signal sysvinit that virtual filesystems are mounted [ OK ]
* Starting set sysctls from /etc/sysctl.conf [ OK ]
* Starting Bridge udev events into upstart [ OK ]
* Starting Signal sysvinit that local filesystems are mounted [ OK ]
* Starting Signal sysvinit that remote filesystems are mounted [ OK ]
* Starting device node and kernel event manager [ OK ]
* Starting GNU Screen Cleanup [ OK ]
* Starting Flush boot log to disk [ OK ]
* Starting flush early job output to logs [ OK ]
* Stopping Mount filesystems on boot [ OK ]
* Stopping set sysctls from /etc/sysctl.conf [ OK ]
* Stopping set console keymap [ OK ]
* Stopping Flush boot log to disk [ OK ]
* Stopping flush early job output to logs [ OK ]
* Starting load modules from /etc/modules [ OK ]
* Starting cold plug devices [ OK ]
* Starting log initial device creation [ OK ]
* Stopping GNU Screen Cleanup [ OK ]
* Starting Bridge file events into upstart [ OK ]
* Stopping load modules from /etc/modules [ OK ]
* Starting system logging daemon [ OK ]
* Starting D-Bus system message bus [ OK ]
* Starting SystemD login management service [ OK ]
* Stopping cold plug devices [ OK ]
* Stopping log initial device creation [ OK ]
* Starting configure network device security [ OK ]
* Starting configure network device security [ OK ]
* Starting configure network device security [ OK ]
* Starting configure network device security [ OK ]
* Starting configure network device security [ OK ]
* Starting load fallback graphics devices [ OK ]
* Starting configure network device security [ OK ]
* Stopping load fallback graphics devices [ OK ]
* Starting save udev log and update rules [ OK ]
* Starting set console font [ OK ]
* Stopping save udev log and update rules [ OK ]
* Stopping set console font [ OK ]
* Starting userspace bootsplash [ OK ]
* Starting configure network device [ OK ]
* Starting Send an event to indicate plymouth is up [ OK ]
* Stopping Send an event to indicate plymouth is up [ OK ]
* Starting configure network device security [ OK ]
* Stopping userspace bootsplash [ OK ]
* Starting Mount network filesystems [ OK ]
* Stopping Mount network filesystems [ OK ]
* Starting Mount network filesystems [ OK ]
* Starting configure network device [ OK ]
* Stopping Mount network filesystems [ OK ]
* Starting Mount network filesystems [ OK ]
* Starting configure network device [ OK ]
* Starting System V initialisation compatibility [ OK ]
* Stopping Mount network filesystems [ OK ]
* Starting Uncomplicated firewall [ OK ]
Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd
* Starting configure network device [ OK ]
* Starting Mount network filesystems [ OK ]
* Starting Failsafe Boot Delay [ OK ]
* Stopping Mount network filesystems [ OK ]
* Starting Bridge socket events into upstart [ OK ]
* Starting AppArmor profiles [ OK ]
* Setting up X socket directories... [ OK ]
* Stopping System V initialisation compatibility [ OK ]
* Starting System V runlevel compatibility [ OK ]
* Starting save kernel messages [ OK ]
* Starting regular background program processing daemon [ OK ]
* Starting deferred execution scheduler [ OK ]
* Starting ACPI daemon [ OK ]
* Starting automatic crash report generation [ OK ]
* Stopping save kernel messages [ OK ]
* Starting CPU interrupts balancing daemon [ OK ]
* Starting OpenSSH server [ OK ]
Starting Systems Management Device Drivers:
Starting dell_rbu: *
Starting ipmi driver: * Already started
Starting Systems Management Data Engine:
Starting dsm_sa_datamgrd: * Starting MySQL Server [ OK ]
*
Starting dsm_sa_eventmgrd: *
Starting dsm_sa_snmpd: *
/etc/rc2.d/S20dataeng: 566: /etc/rc2.d/S20dataeng: /opt/dell/srvadmin//lib64/srvadmin-deng/dcfwsnmp.sh: Permission denied
WARNING: The SNMP port appears to be closed by the firewall.
You may not be able to manage this system remotely using SNMP.
See Firewall Configuration in the product user's guide.
* Starting Postfix Mail Transport Agent postfix [ OK ]
* Starting ipmi drivers [ OK ]
* Stopping OpenSSH server [ OK ]
* Starting OpenSSH server [ OK ]
* Starting Mount network filesystems [ OK ]
* Stopping Mount network filesystems [ OK ]
* Starting configure network device [ OK ]
* Stopping OpenSSH server [ OK ]
* Starting OpenSSH server [ OK ]
* Starting Mount network filesystems [ OK ]
* Starting configure network device [ OK ]
* Stopping Mount network filesystems [ OK ]
* Starting configure virtual network devices [ OK ]
root at otrs-prod:/etc/init#
When listing the processes running, I see some plymouth-related
processes which seem hung:
...
root 2944 2715 0 08:34 ? 00:00:00 sshd: support [priv]
support 3021 2944 0 08:34 ? 00:00:00 sshd: support at pts/3
support 3022 3021 0 08:34 pts/3 00:00:00 -bash
otrs 3147 1 0 08:35 ? 00:00:01 /usr/bin/perl /opt/otrs//bin/otrs.Scheduler.pl -w 1
root 3153 1 0 08:35 ? 00:00:00 plymouth --ping
root 3406 2 0 08:39 ? 00:00:00 [kworker/u66:0]
root 3506 3022 0 08:42 pts/3 00:00:00 sudo su
root 3507 3506 0 08:42 pts/3 00:00:00 su
root 3508 3507 0 08:42 pts/3 00:00:00 bash
root 3540 1679 0 08:43 ? 00:00:00 /bin/sh /etc/rc2.d/S70dns-clean start
root 3651 3540 0 08:48 ? 00:00:00 plymouth --ping
postfix 3667 2458 0 08:49 ? 00:00:00 smtpd -n smtp -t inet -u -c -o stress= -s 2
postfix 3668 2458 0 08:49 ? 00:00:00 proxymap -t unix -u
postfix 3669 2458 0 08:49 ? 00:00:00 anvil -l -t unix -u -c
root 3673 1584 0 08:49 ? 00:00:00 /bin/plymouth message --text=Booting system without full network configuration...
root 3681 3508 0 08:49 pts/3 00:00:00 ps -eaf
root at otrs-prod:/home/support#
root at otrs-prod:/home/support# ps -eaf | grep ply
root 367 1 0 08:33 ? 00:00:00 @sbin/plymouthd --mode=boot --attach-to-session
root 368 1 0 08:33 ? 00:00:00 plymouth-upstart-bridge
root 3153 1 0 08:35 ? 00:00:00 plymouth --ping
root 3651 1 0 08:48 ? 00:00:00 plymouth --ping
root 3822 3815 0 08:52 ? 00:00:00 plymouth --ping
root 4042 4022 0 08:56 ? 00:00:00 plymouth --ping
root 4050 3508 0 08:56 pts/3 00:00:00 grep --color=auto ply
root at otrs-prod:/home/support
root at otrs-prod:/home/support# strace -p 367
Process 367 attached
write(12, "Waiting for network configuratio"..., 36
Workaround:
===========
After a few more troubleshooting, failing to understand what the
problem is, I decided to kill a "plymouth --ping" process. Eventually,
it allowed the system to boot a little further (ie: I can see in the
list of processes that initd/systemd/whatever launched a new startup
script. For example, in the processes list here above, the S70dns-
clean startup script would have vanished and a new one, eg. S70ppd-
dns, would have shown up), before it hung again on a "plymouth --ping"
process.
I thus continued to kill those "plymouth --ping" each time allowing
the system to boot a little further, until no "plymouth --ping"
process would show up anymore and the system appeared to be completly
started.
A that time, I also killed "plymouthd" for the sake of completeness.
Expected result:
================
The system should boot without human assistance.
Supportive information:
=======================
My system is a physicall server Dell PowerEdge R710 running Ubuntu
14.04 now running linux kernel 3.13.0-92-generic:
root at otrs-prod:/etc/init# lsb_release -rd
Description: Ubuntu 14.04.4 LTS
Release: 14.04
root at otrs-prod:/etc/init# uname -a
Linux otrs-prod 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
root at otrs-prod:/etc/init#
It has 3 physical ethernet interfaces, with 2 bounded together. Here's
my (edited) interfaces file:
root at otrs-prod:/etc/init# cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto bond0
iface bond0 inet static
address a.b.c.d
netmask 255.255.255.224
gateway a.b.c.1
bond-slaves none
bond_mode 1
bond_miimon 100
iface bond0 inet6 static
address a:b:c:d::e
netmask 64
gateway a:b:c:d::1
# Impletement split access
up ip -f inet6 rule add from a:b:c:d::e table local_v6
up ip -f inet6 route add default via a:b:c:d::1 dev bond0 table local_v6
auto eth0
iface eth0 inet manual
bond-master bond0
bond_mode 1
bond_miimon 100
auto eth1
iface eth1 inet manual
bond-master bond0
bond_mode 1
bond_miimon 100
# The management interface
auto eth2
iface eth2 inet static
address f.g.h.i
netmask 255.255.0.0
network f.g.0.0
broadcast f.g.255.255
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers s.t.u.v w.x.y.z
dns-search fw.belnet.be
up route add -net f.0.0.0/8 gw f.g.0.1 dev eth2
iface eth2 inet6 static
address a:b:c:f::g
netmask 64
up route -A inet6 add a:b:c:h::/56 gw a:b:c:f::1 dev eth2
root at otrs-prod:/etc/init#
Finally, the system runs the OpenManage stuff:
root at otrs-prod:/etc/init# dpkg -l "srvadmin*"
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===============================-====================-====================-===================================================================
ii srvadmin-base 8.3.0 amd64 Meta package for installing the Server Agent
ii srvadmin-deng 8.3.0-1 amd64 Dell OpenManage Data Engine
ii srvadmin-deng-snmp 8.3.0-1 amd64 Dell OpenManage Data Engine SNMP
ii srvadmin-hapi 8.3.0-1 amd64 Dell OpenManage Hardware Abstraction Programming Interface
ii srvadmin-idrac-snmp 8.2.0-1 amd64 CLI utils from the management station to the iDRAC
ii srvadmin-isvc 8.3.0-1 amd64 Dell OpenManage Instrumentation Services
ii srvadmin-isvc-snmp 8.3.0-1 amd64 Dell OpenManage Instrumentation Services SNMP
ii srvadmin-nvme 8.3.0 amd64 This package provides the libaries to manage NVMe devices.
ii srvadmin-omacore 8.3.0-1 amd64 Server Administrator CLI
ii srvadmin-omacs 8.3.0-2 amd64 Dell OpenManage Server Administrator OMACS
ii srvadmin-omcommon 8.3.0-2 amd64 Dell OpenManage Server Administrator Common Framework
ii srvadmin-omilcore 8.3.0 all Dell OpenManage Server Administrator Install Core
ii srvadmin-ominst 8.3.0-1 amd64 OMINST
ii srvadmin-realssd 7.4.0-1 amd64 RealSSD package for storage management
ii srvadmin-rnasoap 7.4.0-1 amd64 Fluid Cache Management
ii srvadmin-server-cli 8.3.0 amd64 Meta package for installing the Server Administrator server cli com
ii srvadmin-server-snmp 8.3.0 amd64 Meta package for installing the Server Administrator snmp component
ii srvadmin-smcommon 8.3.0 amd64 Storage Management common files for GUI and CLI
ii srvadmin-storage 8.3.0 amd64 Storage Management accessors package
ii srvadmin-storage-cli 8.3.0 amd64 Storage Management cli component
un srvadmin-storage-populator <none> <none> (no description available)
ii srvadmin-storage-snmp 8.3.0 amd64 Storage Management SNMP component
ii srvadmin-storageservices 7.4.0 amd64 Meta package for installing the Server Administrator Storage Servic
ii srvadmin-storageservices-cli 7.4.0 amd64 Meta package for storageservices-cli
ii srvadmin-storageservices-snmp 7.4.0 amd64 Meta package for storageservices-snmp
ii srvadmin-storelib 8.3.0 amd64 StoreLib package for storage management
ii srvadmin-storelib-sysfs 8.3.0 amd64 Metapackage for libsysfs2
ii srvadmin-xmlsup 8.3.0-1 amd64 Dell OpenManage XML Support SDK
root at otrs-prod:/etc/init#
root at otrs-prod:/etc/init# cat /etc/apt/sources.list.d/openmanage.ubuntu.com.sources.list
deb http://linux.dell.com/repo/community/ubuntu trusty openmanage
root at otrs-prod:/etc/init#
Plymouth is at latest available version:
root at otrs-prod:/etc/init# dpkg -l "plymouth*"
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===============================-====================-====================-===================================================================
ii plymouth 0.8.8-0ubuntu17.1 amd64 graphical boot animation and logger - main package
un plymouth-theme <none> <none> (no description available)
ii plymouth-theme-ubuntu-text 0.8.8-0ubuntu17.1 amd64 graphical boot animation and logger - ubuntu-logo theme
root at otrs-prod:/etc/init#
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/plymouth/+bug/1606472/+subscriptions
More information about the foundations-bugs
mailing list