[Bug 1285312] Re: Setting up mdadm (3.2.5-5ubuntu3) freezes at Found kernel: when upgrading from Saucy to Trusty

Wed Mar 12 20:23:13 UTC 2014

Problem is still occurring. My server has been sitting in a half-
upgraded unrebooted state since I filed this Bug. I've been killing the
hung mdadm.postint script occasionally and re-running apt-get dist-
upgrade.

Currently, the processes that are hanging are:

root at kami:~# ps auxww | grep dpkg
root     17854  0.0  0.2  17764  5308 pts/3    Ss+  11:10   0:00 /usr/bin/dpkg --status-fd 47 --configure mdadm:amd64 libitm1:amd64 libgomp1:amd64 libasan0:amd64 libatomic1:amd64 libtsan0:amd64 libquadmath0:amd64 cpp-4.8:amd64 libgcc-4.8-dev:amd64 gcc-4.8:amd64 libstdc++-4.8-dev:amd64 g++-4.8:amd64 libpython3.3-minimal:amd64 python3.3-minimal:amd64 libpython3.3-stdlib:amd64 python3.3:amd64 bash-completion:all libsystemd-daemon0:amd64 systemd-shim:amd64 systemd-services:amd64 libpam-systemd:amd64 libpci3:amd64 pciutils:amd64 libsystemd-login0:amd64 imagemagick-common:all libasound2-data:all libasound2:amd64 libcups2:amd64 libcupsfilters1:amd64 libcupsimage2:amd64 libgdk-pixbuf2.0-common:all libgdk-pixbuf2.0-0:amd64 libmagickcore5:amd64 libmagickwand5:amd64 libmagickcore5-extra:amd64 libpolkit-agent-1-0:amd64 libpolkit-backend-1-0:amd64 libpython3.4-minimal:amd64 python3.4-minimal:amd64 libpython3.4-stdlib:amd64 python3.4:amd64 librados2:amd64 librbd1:amd64 libsnmp-base:all libsnmp30:amd64 php5-common:amd64 php5-mysql:amd64 php5-cli:amd64 php5-readline:amd64 libapache2-mod-php5:amd64 php5-gd:amd64 php5-curl:amd64 qemu-system-common:amd64 dh-python:all locales:all upstart:amd64 libisc95:amd64 libdns100:amd64 libisccc90:amd64 libisccfg90:amd64 libbind9-90:amd64 liblwres90:amd64 bind9utils:amd64 bind9:amd64 bind9-host:amd64 dnsutils:amd64 geoip-database:all libglib2.0-data:all lshw:amd64 memtest86+:amd64 openssh-client:amd64 openssh-sftp-server:amd64 openssh-server:amd64 python3-problem-report:all python3-apport:all python3-distupgrade:all ubuntu-release-upgrader-core:all python3-gi:amd64 shared-mime-info:amd64 ufw:all uuid-runtime:amd64 grub-common:amd64 apport:all apport-symptoms:all libdpkg-perl:all dpkg-dev:all imagemagick:amd64 libjs-sphinxdoc:all qemu-keymaps:all qemu-system-x86:amd64 qemu-kvm:amd64 libvirt0:amd64 libvirt-bin:amd64 linux-libc-dev:amd64 php5:all policykit-1:amd64 vim-puppet:all puppet-common:all puppet:all python-cinderclient:all python-gi:amd64 python-libvirt:amd64 python-lxml:amd64 python-novaclient:all python-secretstorage:all unattended-upgrades:all python-software-properties:all python-webtest:all qemu-utils:amd64 snmp:amd64 snmpd:amd64 ssh:all wpasupplicant:amd64 hardening-includes:all python-nova:all nova-common:all python-keystone:all python-swiftclient:all python-vm-builder:all ubuntu-vm-builder:all util-linux-locales:all
root     17855  0.1  0.7  63112 15192 pts/3    S+   11:10   0:00 /usr/bin/perl -w /usr/share/debconf/frontend /var/lib/dpkg/info/mdadm.postinst configure 3.2.5-5ubuntu2
root     17867  0.0  0.0   4444   684 pts/3    S+   11:10   0:00 /bin/sh /var/lib/dpkg/info/mdadm.postinst configure 3.2.5-5ubuntu2
root     18331  0.0  0.0  10464   880 pts/1    S+   11:14   0:00 grep dpkg

And the culprit appears to be update-grub:

root at kami:~# ps auxww | grep grub
root     17854  0.0  0.2  17764  5308 pts/3    Ss+  11:10   0:00 /usr/bin/dpkg --status-fd 47 --configure mdadm:amd64 libitm1:amd64 libgomp1:amd64 libasan0:amd64 libatomic1:amd64 libtsan0:amd64 libquadmath0:amd64 cpp-4.8:amd64 libgcc-4.8-dev:amd64 gcc-4.8:amd64 libstdc++-4.8-dev:amd64 g++-4.8:amd64 libpython3.3-minimal:amd64 python3.3-minimal:amd64 libpython3.3-stdlib:amd64 python3.3:amd64 bash-completion:all libsystemd-daemon0:amd64 systemd-shim:amd64 systemd-services:amd64 libpam-systemd:amd64 libpci3:amd64 pciutils:amd64 libsystemd-login0:amd64 imagemagick-common:all libasound2-data:all libasound2:amd64 libcups2:amd64 libcupsfilters1:amd64 libcupsimage2:amd64 libgdk-pixbuf2.0-common:all libgdk-pixbuf2.0-0:amd64 libmagickcore5:amd64 libmagickwand5:amd64 libmagickcore5-extra:amd64 libpolkit-agent-1-0:amd64 libpolkit-backend-1-0:amd64 libpython3.4-minimal:amd64 python3.4-minimal:amd64 libpython3.4-stdlib:amd64 python3.4:amd64 librados2:amd64 librbd1:amd64 libsnmp-base:all libsnmp30:amd64 php5-common:amd64 php5-mysql:amd64 php5-cli:amd64 php5-readline:amd64 libapache2-mod-php5:amd64 php5-gd:amd64 php5-curl:amd64 qemu-system-common:amd64 dh-python:all locales:all upstart:amd64 libisc95:amd64 libdns100:amd64 libisccc90:amd64 libisccfg90:amd64 libbind9-90:amd64 liblwres90:amd64 bind9utils:amd64 bind9:amd64 bind9-host:amd64 dnsutils:amd64 geoip-database:all libglib2.0-data:all lshw:amd64 memtest86+:amd64 openssh-client:amd64 openssh-sftp-server:amd64 openssh-server:amd64 python3-problem-report:all python3-apport:all python3-distupgrade:all ubuntu-release-upgrader-core:all python3-gi:amd64 shared-mime-info:amd64 ufw:all uuid-runtime:amd64 grub-common:amd64 apport:all apport-symptoms:all libdpkg-perl:all dpkg-dev:all imagemagick:amd64 libjs-sphinxdoc:all qemu-keymaps:all qemu-system-x86:amd64 qemu-kvm:amd64 libvirt0:amd64 libvirt-bin:amd64 linux-libc-dev:amd64 php5:all policykit-1:amd64 vim-puppet:all puppet-common:all puppet:all python-cinderclient:all python-gi:amd64 python-libvirt:amd64 python-lxml:amd64 python-novaclient:all python-secretstorage:all unattended-upgrades:all python-software-properties:all python-webtest:all qemu-utils:amd64 snmp:amd64 snmpd:amd64 ssh:all wpasupplicant:amd64 hardening-includes:all python-nova:all nova-common:all python-keystone:all python-swiftclient:all python-vm-builder:all ubuntu-vm-builder:all util-linux-locales:all
root     17878  0.0  0.1  11736  2156 pts/3    S+   11:10   0:00 /bin/bash /usr/sbin/update-grub
root     18341  0.0  0.0  10464   880 pts/1    S+   11:16   0:00 grep grub

But when I kill mdadm.postint and run update-grub by hand it works
fine...but loops through the kernels twice.

root at kami:~# kill 17855 17867

root at kami:~# update-grub
Searching for GRUB installation directory ... found: /boot/grub
Searching for default file ... found: /boot/grub/default
Testing for an existing GRUB menu.lst file ... found: /boot/grub/menu.lst
Searching for splash image ... none found, skipping ...
Found kernel: /boot/vmlinuz-3.13.0-17-generic
Found kernel: /boot/vmlinuz-3.11.0-15-generic
Found kernel: /boot/memtest86+.bin
Found kernel: /boot/vmlinuz-3.13.0-17-generic
Found kernel: /boot/vmlinuz-3.11.0-15-generic
Found kernel: /boot/memtest86+.bin
Updating /boot/grub/menu.lst ... done

If it run it again, it only loops through once:

root at kami:~# update-grub
Searching for GRUB installation directory ... found: /boot/grub
Searching for default file ... found: /boot/grub/default
Testing for an existing GRUB menu.lst file ... found: /boot/grub/menu.lst
Searching for splash image ... none found, skipping ...
Found kernel: /boot/vmlinuz-3.13.0-17-generic
Found kernel: /boot/vmlinuz-3.11.0-15-generic
Found kernel: /boot/memtest86+.bin
Updating /boot/grub/menu.lst ... done

I finally bit the bullet and tried to reboot (from remote) after getting
a clean apt-get dist-upgrade run and running update-grub manually. The
system is pingable but I can no longer login; I'll have to check console
when I get home.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to mdadm in Ubuntu.
https://bugs.launchpad.net/bugs/1285312

Title:
  Setting up mdadm (3.2.5-5ubuntu3) freezes at Found kernel: when
  upgrading from Saucy to Trusty

Status in GRand Unified Bootloader:
  New
Status in “mdadm” package in Ubuntu:
  New

Bug description:
  I'm unable to upgrade my Dell PowerEdge T105 from Saucy to Trusty
  because madm freezes during the update grub process during setup. This
  system was originally installed in April of 2009 with Hardy 8.04 and
  has successfully upgraded from revision to revision since then. This
  is the first time madm has failed to upgrade.

  Yesterday evening I ran do-release-upgrade -d and it froze at:

  Setting up libdevmapper1.02.1:amd64 (2:1.02.77-6ubuntu2) ...
  Setting up libdevmapper-event1.02.1:amd64 (2:1.02.77-6ubuntu2) ...
  Setting up lvm2 (2.02.98-6ubuntu2) ...
  Installing new version of config file /etc/lvm/lvm.conf ...
  update-initramfs: deferring update (trigger activated)
  Setting up mdadm (3.2.5-5ubuntu3) ...
   Removing any system startup links for /etc/init.d/mdadm-raid ...
  update-initramfs: deferring update (trigger activated)
  update-grub is /usr/sbin/update-grub
  Searching for GRUB installation directory ... found: /boot/grub
  Searching for default file ... found: /boot/grub/default
  Testing for an existing GRUB menu.lst file ... found: /boot/grub/menu.lst
  Searching for splash image ... none found, skipping ...
  Found kernel: /boot/vmlinuz-3.13.0-12-generic
  Found kernel: /boot/vmlinuz-3.11.0-17-generic
  Found kernel: /boot/vmlinuz-3.11.0-15-generic
  Found kernel: /boot/vmlinuz-3.11.0-14-generic
  Found kernel: /boot/vmlinuz-3.11.0-12-generic
  Found kernel: /boot/vmlinuz-3.11.0-7-generic
  Found kernel: /boot/memtest86+.bin
  -- 0:trusty -- time-stamp -- Feb/25/14 17:04:19 --
  -- 0:trusty -- time-stamp -- Feb/25/14 17:11:58 --

  -- 0:trusty -- time-stamp -- Feb/25/14 17:14:00 --
  -- 0:trusty -- time-stamp -- Feb/25/14 17:27:48 --

  It sat there for about half an hour so I canceled the do-release-upgrade and tried to recover with the usual steps after cleaning out some stale kernels:

  root at kami:~# apt-get update
  [...] output removed
  E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem. 
  root at kami:~# apt-get -f install
  E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem. 
  root at kami:~# apt-get dist-upgrade
  E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem. 
  root at kami:~# sudo dpkg --configure -a
  Processing triggers for initramfs-tools (0.103ubuntu3) ...
  update-initramfs: Generating /boot/initrd.img-3.13.0-12-generic
  Setting up mdadm (3.2.5-5ubuntu3) ...
   Removing any system startup links for /etc/init.d/mdadm-raid ...
  update-initramfs: deferring update (trigger activated)
  update-grub is /usr/sbin/update-grub
  Searching for GRUB installation directory ... found: /boot/grub
  Searching for default file ... found: /boot/grub/default
  Testing for an existing GRUB menu.lst file ... found: /boot/grub/menu.lst
  Searching for splash image ... none found, skipping ...
  Found kernel: /boot/vmlinuz-3.13.0-12-generic
  Found kernel: /boot/vmlinuz-3.11.0-17-generic
  Found kernel: /boot/vmlinuz-3.11.0-15-generic
  Found kernel: /boot/memtest86+.bin

  At this point mdadm setup still just sits forever. If I hit Ctrl-C:

  ^Cdpkg: error processing package mdadm (--configure):
   subprocess installed post-installation script was interrupted
  Processing triggers for initramfs-tools (0.103ubuntu3) ...
  update-initramfs: Generating /boot/initrd.img-3.13.0-12-generic
  Errors were encountered while processing:
   mdadm
  root at kami:~# 

  root at kami:~# fdisk -l /dev/sda

  Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
  255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
  Units = sectors of 1 * 512 = 512 bytes
  Sector size (logical/physical): 512 bytes / 512 bytes
  I/O size (minimum/optimal): 512 bytes / 512 bytes
  Disk identifier: 0x000918ae

     Device Boot      Start         End      Blocks   Id  System
  /dev/sda1   *          63    39070079    19535008+  fd  Linux raid autodetect
  /dev/sda2        39070080  1953520064   957224992+   5  Extended
  /dev/sda5        39070143    46877669     3903763+  fd  Linux raid autodetect
  /dev/sda6        46877733  1953520064   953321166   fd  Linux raid autodetect

  root at kami:~# fdisk -l /dev/sdb

  Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
  255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
  Units = sectors of 1 * 512 = 512 bytes
  Sector size (logical/physical): 512 bytes / 512 bytes
  I/O size (minimum/optimal): 512 bytes / 512 bytes
  Disk identifier: 0x000455d6

     Device Boot      Start         End      Blocks   Id  System
  /dev/sdb1   *          63    39070079    19535008+  fd  Linux raid autodetect
  /dev/sdb2        39070080  1953520064   957224992+   5  Extended
  /dev/sdb5        39070143    46877669     3903763+  fd  Linux raid autodetect
  /dev/sdb6        46877733  1953520064   953321166   fd  Linux raid autodetect

  root at kami:~# cat /proc/mdstat 
  Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
  md2 : active raid1 sdb6[1] sda6[0]
        953321088 blocks [2/2] [UU]

  md0 : active raid1 sdb1[1] sda1[0]
        19534912 blocks [2/2] [UU]

  md1 : active raid1 sdb5[1] sda5[0]
        3903680 blocks [2/2] [UU]

  unused devices: <none>

  root at kami:~# vgscan 
    Reading all physical volumes.  This may take a while...
    Found volume group "KamiVG01" using metadata type lvm2

  root at kami:~# lvscan
    ACTIVE            '/dev/KamiVG01/chippoke' [50.00 GiB] inherit
    ACTIVE            '/dev/KamiVG01/nas-office' [850.00 GiB] inherit

  (additional data and logs will be added soon)

To manage notifications about this bug go to:
https://bugs.launchpad.net/grub/+bug/1285312/+subscriptions